Akka HTTP - stable, growing and tons of opportunity #27

Closed
ktoso opened this Issue Aug 19, 2016 · 36 comments

Projects

None yet

10 participants

@ktoso
Member
ktoso commented Aug 19, 2016

Dear hakkers,

After the recent Akka HTTP performance improvement releases, it is time to prepare for the Stable release of Akka HTTP.

We have given this process a lot of thought over the last months, including reaching out personally to many of its significant contributors, as well as considering the long term growth plans of the Akka ecosystem.

In tandem with the upcoming announcement of Akka HTTP becoming a fully stable maintained module, we want to include the following changes.

You, the extended Akka HTTP team

Akka has historically been very strictly owned and driven by the core team, here at Lightbend. With this move, we would like to make an experiment and see if our awesome community is able and interested in taking more ownership and direction-shaping of such an important project as Akka HTTP.

Lightbend and Akka team of course remain heavily invested and interested in Akka HTTP.

We believe it is an important building block for both Akka and reactive applications which like the toolkit (i.e. not-framework) design style that Akka maintains.

Akka HTTP remains supported by Lightbend and we will actively participate in its development. In the short term we promised a few weeks of multiple people working on HTTP/2 as well as final performance improvements. In the long term, we will keep maintaining, reviewing, and working on it, however unlike in the last 2 years we will not exclusively focus only on Akka Streams and HTTP – we have new exciting thing we want to bring to you which you’ll learn about very soon (one of them being the Remoting rewrite, Artery).

Action: This is simply to state, there is a lot of work to be done, and plenty of awesome developers out there that we know are able to pull it off, so we’d like to recognise those efforts by awarding a place for the Akka HTTP team in the github organisation.

Highly Open Participation

As a general rule, we’ll even more strongly than ever before, focus on a highly open and transparent way to participate and shape the future of Akka HTTP.

We would like to nominate a number of people–whom we highly value for their past and current participation in both the Spray as well as Akka HTTP project over the last year–to form the seed of the Akka HTTP team on github. This team is specifically not-only-lightbend employees, and it’s aim is to allow various people with a strong commitment in Akka HTTP to help shape its future and be recognised for it in more than just release notes.

The people whom we’d like to invite to form the initial Akka HTTP team are:

  • @2beaucoup – as recognition of his continued involvement in the project since it’s Spray days,
  • @jypma – for his continued efforts on improving and re-inventing the Directives JavaDSL,
  • @jrudolph – of Spray and Akka HTTP fame, who will be joining us at Lightbend’s Akka team,
  • @sirthias and the existing Akka team.

The team will have merge rights in the project, so will be able to help the community get more pull requests reviewed and merged more quickly.

We initially don’t see the need to split the gitter channel or mailing list, those shall remain gitter.im/akka/dev for developing akka and gitter.im/akka/akka and akka-user for questions about using Akka.

Question: We are also thinking about scheduling a bi-weekly hangout, that would be open to anyone to attend in which we’d talk about the direction and important things to work on next. Let us know if this sounds useful and we’ll figure something out.

It would be possibly a good opportunity to improve the collaboration in the Akka HTTP team by growing our bonds through more than just typed text and tickets.

A new Versioning scheme for Akka HTTP

Binary compatibility is very important to us. We’ve shown one of the best track records in the Scala ecosystem on maintaining such compatibility–including efforts that we spent between 2.3.x and 2.4.x that made these binary compatible, which would have–under the previous versioning scheme–been major releases.

Please note that Akka, since the release of 2.4.0 maintains a MAJOR.MINOR.PATCH versioning, as opposed to the previously used EPOCH.MAJOR.MINOR which used to be Scala versioning scheme inspired.

We can not use the 1.0 version to demarcate this new start as it would be very confusing since that number was already used back in July 2015 while we were working on Akka Streams & HTTP under a separate version scheme than Akka "core", which was at 2.3.x and later on continued to become 2.4.x.

Our goal with the HTTP versioning scheme is to signify that while Akka HTTP may need to bump its major version number, e.g. from 10 to 11, it would be still compatible with Akka 2.x–the stable core which will only break compatibility when a huge change is needed.

Action: After careful consideration of all of the above constraints, we propose that the new versioning scheme be different enough not be confused with Akka "core" versions. Specifically, we think using a version number much higher than 2 or 3 (which would be confusable with Akka versions) is the way to go, and suggest to start at 10.0, simultaneously announcing Akka HTTP to be stable (removing the experimental flag).

To decide: Given troublesome situation of the Directives trait being extended by user code, binary compatibility is impossible to maintain when a new directive is added into it. This will change with Scala 2.12, however with 2.11 it still is the case. We propose to keep source compatibility in minor releases, e.g. a new directive can be added but to a new trait that one mixes-in explicitly, and in a major release we can merge those new traits to the existing Directives structure. A major release may then still happen sooner than an Akka 3.0 (which is very far out), and the project can keep a healthy speed - to be determined by the team.

Split documentation

Akka always was and still is well known for it’s excellent documentation and we intend for it to not only stay this way, but become even better!

Our largest challenge currently with this came with the growth of documentation pages, especially with the addition of Akka HTTP and it’s more than 300 directives (all directives and pages available in both Scala and Java), it is becoming more difficult to read / browse the entire docs together.

Therefore, we would like to split up documentation in different module sub-directories. For example, specifically this means that we’ll move all of Akka HTTP docs to a root-level "http" directory on doc.akka.io.

Additionally we want to move the documentation to Lightbend Paradox a documentation engine. It is much simpler to use than Sphinx and ReStructured Text which Akka is using currently because it is plain Markdown. It will shortly gain fine-tuned Scala documentation features and and can even include executable Akka.js snippets using ScalaFiddle. You can get a glimpse of how documentation with embedded runnable Akka Streams snippets might look like on this example page prepared by Andrea Peruffo (from the Akka.js team).

As you know, we included Algolia powered search in our docs so it’s now much easier to find the right thing in the docs. Often however, you are searching for things in context–i.e. "I’m looking to do X in Akka HTTP" or specifically looking how to do something with Reactive Kafka. Often similar search queries, and confining the search to a given module will yield in a better search experience.

Action: Docs will be hosted under doc.akka.io/ as is currently, however will be moved to their own top-level directory, e.g. doc.akka.io/http/10.0/. We will attempt (and need help doing it, there’s a ton of documentation) to move from Sphinx to Paradox, which should be much simpler to contribute docs with as it’s plain markdown.

Exciting and upcoming: HTTP/2

As briefly mentioned in release notes over the last months: yes, we will work on an HTTP/2 proof-of-concept for Akka HTTP. It is important to realise that unlike for example Play which relies on Netty for implementing the HTTP layer and can simply upgrade netty to get this capability, Akka HTTP is an entire web server, thus the responsibility of implementing all low level bits of the HTTP/2 protocol fall into the hands of akka-http-core.

Action: We will spend a number of weeks developing an HTTP/2 PoC, however are not sure how much features we’ll be able to implement during that time. When we start the HTTP/2 sprint, we’ll invite you, the interested parties in this feature to review, help, and contribute to get this the best possible support. In other words, with the project becoming more and more community led, we would like to encourage a "vote with code" approach–if a feature is important, it should be able to attract contributions to see it shipped.

Summing up

We believe this move is both important and exciting. It will allow to grow the community not only sheer size but also strength and go where no man has gone before. Feel free to comment in this meta thread if you have any thoughts on the topic.

More technical details which we need to decide before calling the module stable will be debated on the github.com/akka/akka-http repository. For example stating and explicit source compatibility requirement or perhaps deprecating the "trait-style" way of using directives to allow us to keep binary compatibility in much more situations.

Again, we’d like to stress that we want to hear and see both feedback and code from you all.

Let’s get hakking!

@ktoso ktoso self-assigned this Aug 19, 2016
@2beaucoup

Some thoughts/questions:

  • How do we deal with tickets? Will they be transfered to the new repo?
  • What areas still need bug-fixing/stabalization efforts? (Client pools come to mind.)
  • Starting versions from 10 seems a bit odd. Why not start from 1.0M1. If we drop "experimental" we get a new artifact id anyway.
  • What modules should be ported over? Is http-core included? Any concrete plans to integrate external modules?
  • Can/should git history be preserved?
  • What compat guarantees should we adhere to? Binary or is source enough?
  • Would this be an opportunity to do some cleanups/remove some ugly bin-compat workarounds?
  • Maintaining the directive docs in their current form will require a lot of effort. Maybe we can enrich the nicely grouped ScalaDoc with code samples, link to it and call it a day?

Looking forward to the move! 🚀

@hseeberger
  • I second that version 10 seems odd
  • Should we include akka-sse?
@drewhk
Member
drewhk commented Aug 25, 2016

Seeming odd does not matter. What matters if a number confuses people and build-tools. I think 10.0 is the least confusing from all the other alternatives and given the experiences with the old streams 1.0-2.0 releases.

@2beaucoup

We can never again with a clear conscience make fun of Windows version schemes. 😢

@drewhk
Member
drewhk commented Aug 25, 2016

For me that is a feature (I'm on Windows 10) ;)

@2beaucoup

😄 Now I know where that 10 came from!

Looks like we're the only ones here. Still a part time Windows 10 user, too.

@drewhk
Member
drewhk commented Aug 25, 2016

In my defense, this did not affect the choice of version number :)

@hseeberger

But why 10 instead of 3?

@patriknw
Member

My answer to two of the questions.

How do we deal with tickets? Will they be transfered to the new repo?

Yes. It's currently 219 tickets. I don't know what is the best way to do that.

Can/should git history be preserved?

That is a must. Not necessarily the exact same commit identifiers, but the authorship must be preserved.

@patriknw
Member

10 avoids the risk of being confused with that it is compatible with Akka 3.0
1.0 has similar problem, soon there will be a 2.0 and 3.0, i.e. matching akka core versions.

@hseeberger

Then maybe we should find a better name, making clear that it's not part of Akka core any more. Maybe ... spray? ;-)

@sirthias

+1 for spray 2.0!
Yipieeh!! :)

@drewhk
Member
drewhk commented Aug 25, 2016

If people prefer, I think it might be fine to find a new codename, but for people who want to upgrade it is easier to find new releases by the artifact name (well, except experimental).

@jypma
jypma commented Aug 25, 2016

We also started a ticket that was supposed to carry the discussion more
openly:

akka/akka-http#3

Maybe move further replies there?

On Thu, Aug 25, 2016 at 3:41 PM, drewhk notifications@github.com wrote:

If people prefer, I think it might be fine to find a new codename, but for
people who want to upgrade it is easier to find new releases by the
artifact name (well, except experimental).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAdgv9ZgnmufWvkM6MlM4OoszCLLCUy7ks5qjZt5gaJpZM4JoZno
.

@jypma
jypma commented Aug 25, 2016

Oh nevermind.

@fsvehla
fsvehla commented Aug 25, 2016

@sirthias I realise that the "Spray 2" remark was meant in a funny way, but Spray does have a good reputation and it would solve the versioning issue.

@drewhk
Member
drewhk commented Aug 25, 2016

Unfortunately it would just confuse things even more.

@viktorklang
Member

And it doesn't google very well..

@ktoso
Member
ktoso commented Aug 26, 2016

How do we deal with tickets? Will they be transfered to the new repo?

Yes, we'll move them. All akka-http tickets should be moved into this repo.
I opened "discuss: where should akka-http-core live?" akka/akka-http#9 in which I propose we move akka-http-core as well into the new repo, in order to make issue tracking managable.

I'll figure out if asking github could make moving the tickets somewhat smarter (perhaps they can redirect from old ones etc)

What areas still need bug-fixing/stabalization efforts? (Client pools come to mind.)

In my mind the trickiest things nowadays are:

  • client behaviours, we have a few bugs around it
  • tripping people up that end up not consuming the entities (even if they're aware of it it's too easy to get wrong), so some form of auto draining or warning is needed

Most other things are new features that we can start adding in the new cycle.

Starting versions from 10 seems a bit odd. Why not start from 1.0M1. If we drop "experimental" we get a new artifact id anyway.

Odd yes, but less dangerous to confuse.

  • We can't use 1.0 because that exists (and is ancient), it would be googling hell "akka http 1.0... yeah but THE OTHER ONE".
  • We really don't want to use the same exact version as Akka, because the purpose is to move somewhat faster than Akka (which really has no reason to break binary compatibility until we hit huge awesome new issues in Akka 3). This rules out using any 2.x or 3.x as it would be confusable with Akka normal version.

Let's discuss versioning in akka/akka-http#3

What modules should be ported over? Is http-core included?

akka-http, akka-http-testkit and all the json and xml things for sure.

akka-http-core is tricky, let's discuss in akka/akka-http#9
Initially I was leaning to keep it in akka/akka however after a few months of thinking I've stopped believing that would help in reality. Expectations still are that bugs are fixed. Truth is also that we at Lightbend will likely pick bugs esp. in akka-http-core, because those are a) hard b) relatively not-rewarding-to-work-on, thus less likely to get community help. Community help in core would be very awesome though of course.

My current thinking is that moving is healthier, because if someone looks for akka http or files a bug, they'd do so in the akka-http repo. I don't want to explain to people who just found a bug that they're reporting in the wrong place.

Any concrete plans to integrate external modules?

Yes, that's part of the plan – @hseeberger already pinged about akka-sse, and I think we could be by far more aggressive in including things and offering their maintainers committer rights in akka/akka-http.

This is an open invitation, if you feel your module makes sense to be included please step up and let us know.
I guess this needs a formal write up so that people know this is an option.

E.g. akka-http session, more json support traits, caching etc are great candidates here.

Can/should git history be preserved?

Yes, as Patrik said this is a must.
We can do this rather easily, https://help.github.com/articles/splitting-a-subfolder-out-into-a-new-repository/

What compat guarantees should we adhere to? Binary or is source enough?

Let's discuss that in the versioning scheme topic as it relates to how we bump versions.
I think we won't get away with not doing binary at all, but let's discuss in: akka/akka-http#3

Would this be an opportunity to do some cleanups/remove some ugly bin-compat workarounds?

Do you have some specific ones in mind? Akka HTTP (the DSLs) were breaking compatibility.
If we decide we need to move out akka-http-core as well (I'm not sure about that, both ways have their upsides).

Maintaining the directive docs in their current form will require a lot of effort. Maybe we can enrich the nicely grouped ScalaDoc with code samples, link to it and call it a day?

I like that idea a lot!

Let's continue on it here: akka/akka-http#10

@ktoso
Member
ktoso commented Aug 26, 2016 edited

+1 for spray 2.0!

Is both funny and true ;-) However also very confusing...
I'm not sure it's a viable option really - unless we have strong proponents of it.

@2beaucoup

Would this be an opportunity to do some cleanups/remove some ugly bin-compat workarounds?

Do you have some specific ones in mind?

E.g. things like akka.http.scaladsl.model.headers.HttpChallenge where we recently made params optional or the static ALL fields in javadsl.

@2beaucoup

HttpEntity.dataBytes: Source[ByteString, Any] would be another offender.

@ktoso
Member
ktoso commented Aug 26, 2016

True, that one's a bit painful hm.
Let's collect those in a ticket and access when to best make those?

@ktoso
Member
ktoso commented Sep 5, 2016

Moving on, starting the move this week, progress here: akka/akka-http#13

This was referenced Sep 8, 2016
@jypma
jypma commented Sep 8, 2016

@ctataryn You're indeed correct about the direction. You're best off migrating from spray to akka 2.4 and akka-http where you can.

@ktoso
Member
ktoso commented Sep 8, 2016

2-3 years ago Spray began the journey to become Akka HTTP.
Akka HTTP is a full rewrite of the internals actually, however much of the feel remains the same and true to Spray.
The team which worked on Akka HTTP was the original Spray team + the core Akka team.
Part of the Spray team has by now joined the Akka team.

The annoucement you're looking for on Spray.io has been basically waiting for the DSL to stabilise in Akka HTTP. The core has been stable since half a year or so already. The moment we announce the DSL to be stable is coming within a few weeks from now, so yes - there will be such note on Spray.io finally :-)

Please note that not only is Akka HTTP "like Spray", it added a ton of features, including websockets, native streaming as well as a full JavaDSL (which you as Scala guy probably don't care about, but it was pretty interesting for many of our users).

@ctataryn
ctataryn commented Sep 8, 2016

@jypma you are talking to a man who is just now upgrading from Slick 2.1 -> 3.x. Am I going to have the same "kicked in the teeth" feeling with this move from spray->akka-http? 😷 😩

@ctataryn
ctataryn commented Sep 8, 2016 edited

@ktoso will there be a decent migration guide available? I really felt the upgrade guide for Slick 2.1->3.x was highly highly lacking. Which is disappointing because I felt their previous upgrade guides always covered the bases. If it weren't for the Essential Slick 3 book, I'd be pretty hosed.

As it appears I'll have to upgrade to Akka-HTTP I'd be happy to contribute to such a guide.

@jypma
jypma commented Sep 8, 2016

@ctataryn Apologies, no kick was intended. I realise we're not all living on the edge, being in an environment myself where many are still making the transition from Java 7 to 8.

However, yes, a migration is probably going to be a bigger job; if you're on Spray you may also be still on scala 2.10, and perhaps akka 2.3 or earlier. There've been quite a few changes since. However, both the spirit and terminology of all involved libraries have mostly been the same. Having migrated a rather large code base from the mentioned versions to akka-http 2.4 myself, I can say that it's perfectly doable, but should be planned for accordingly. There are migration guides available for much of it, e.g. for spray here and of course there's always the akka gitter channel and mailing list.

@ktoso
Member
ktoso commented Sep 8, 2016

The DSL layer is very very similar, nowhere near a Slick 2->3 move. The internals are a full rewrite, the DSL not as much - plenty small improvements but it remains very similar.

A migration guide is available and we do welcome feedback and help upgrading it even better: http://doc.akka.io/docs/akka/2.4/scala/http/migration-from-spray.html

@ctataryn
ctataryn commented Sep 8, 2016

@jypma The project is for my own startup http://crowdscriber.com. Problem is, I bootstrap myself so it's slow and expensive when I realize I have to upgrade. Slick has been especially challenging. It is an awesome framework but the upgrade from 2.1->3.x has been particularly painful (but needed). Sadly I won't be able to take advantage of non-blocking I/0 for some time as my Repository layer wasn't designed to hand back Futures. For now, Await.result(..., Duration.Inf) for me... 😑

Here's my current dependencies FWIW:

scalaVersion  := "2.11.2"
  "io.spray"                %    "spray-routing_2.11"           %    "1.3.1",
  "io.spray"                %    "spray-testkit_2.11"           %    "1.3.1",
  "io.spray"                %    "spray-json_2.11"              %    "1.2.6",
  "io.spray"                %    "spray-can_2.11"               %    "1.3.1",
  "io.spray"                %    "spray-client_2.11"            %    "1.3.1",
  "com.typesafe.akka"       %    "akka-actor_2.11"              %    "2.3.5",
  "com.typesafe.akka"       %    "akka-testkit_2.11"            %    "2.3.5",
  "com.typesafe.akka"       %    "akka-camel_2.11"              %    "2.3.5",
  "com.typesafe.slick"      %    "slick_2.11"                   %    "2.1.0",

If any of you folks will be at JavaOne this year I'm given a talk on the tech-stack we chose.
https://twitter.com/craiger/status/766025292494274560 I just ask that you wait until after the session to call me out over 🍻

This was referenced Sep 8, 2016
@ktoso
Member
ktoso commented Oct 18, 2016

Sorry I missed you on J1 @ctataryn, would have been fun to meet - hope to see you next time (if you're in SF, then I'm there for QCon next month) :-)

Most things in the ticket were addressed, we'll close it once Akka HTTP 3.0.0 hits (RC1 is currently out).
See also:

@ktoso
Member
ktoso commented Oct 18, 2016 edited

If you're interested in participating more actively, we've just set up a call for next week so we can match faces to nicknames :-)
See here: #34

@ctataryn

@ktoso darn, wish I could go to QConSF! I've always wanted to go to that one. I'll definitely hop on the call, thanks for the invite!

@ktoso ktoso closed this Oct 26, 2016
@ktoso
Member
ktoso commented Oct 26, 2016

Closing, the ticket has served it's annoucement purpose :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment