Consider forking twitter's hpack

[nmittler]

In order to be able to more easily incorporate performance improvements going forward, we should consider forking twitter's hpack implementation and just making it part of Netty.

@normanmaurer @louiscryan @Scottmitch WDYT?

[normanmaurer]

@nmittler I'm not sure yet... trying to find out at the moment who is maintaining it @ twitter atm.

[nmittler]

@normanmaurer ok, although the fact that you have to do that suggests it to be a problem :).

The project hasn't had a commit since June (and < 60 commits total) ... not a good sign :/

[buchgr]

afaik none of the guys works at Twitter anymore, and Twitter Open Source got killed recently as well.

[louiscryan]

Jeff is still doing reviews.

e.g. twitter/hpack#28

it may just be that they haven't done anything because no one asked them to (they have no open issues after all) so it's by no means dead. I think what would be most useful would be to list out what we think the primary benefits of having an internal impl might be vs using twitters aside from project health issues:

• Using Buf directly instead of streams during parsing
• Avoidance of intermediate buffer types (above PR mitigates that significantly but not entirely)
• Ability to use alternate 'static tables' for advanced users (could be pushed down into twitters I suppose)
• Ability to avoid the huffman encoding for constant header names and values a la Jetty (http://download.eclipse.org/jetty/9.3.2.v20150730/apidocs/org/eclipse/jetty/http2/hpack/HpackFieldPreEncoder.html) (could be pushed down into twitter too I think)

I don't think buffer pooling is a significant concern here as most headers are small but I could be convinced otherwise...

[louiscryan]

@jpinner Hey Jeff, should we worry about your bus no. ?

[jpinner]

There are no maintainers of the library at Twitter anymore. I still have commit access but would need to ask my contacts at twitter to publish a new release.

In terms of bus factor, there's still @wgallagher but that's about it. The more people familiar with the code the better.

As for number and timing of commits, velocity was higher back when we were designing hpack/http2 and trailed off once the RFC was published.

[Scottmitch]

My vote is to fork. Seems like the project maintenance path is in limbo and we have been kicking around the idea (#3597) of changing the APIs for a while.

[louiscryan]

@jpinner Who are the direct users of the library these days aside from Netty and OkHttp?

[louiscryan]

Actually it looks like okhttp forked too. Given that I'm OK with forking too

[jpinner]

@louiscryan only other thing i know of is https://github.com/twitter/netty-http2 -- i'm fine with you guys forking it btw

[Scottmitch]

/cc @yschimke @madster26 ... looks like they are still contributing to twitter's netty-http2 implementation. Do you guys plan to move toward Netty's codec-http2?

[yschimke]

I think it makes a lot of sense to fork the hpack project and make it part of the broader Netty project.

Regarding Twitter's maintenance of the netty-http2 library, the recent changes I made are a stepping stone to using netty 4.1. When we are on 4.1 then we will be able to adopt Netty's codec-http2. AFAIK no-one in Twitter plans to actively maintain the netty-http2 library considering Netty's official codec-http2.

[normanmaurer]

@yschimke thanks for the info!

[nmittler]

Ok great, thanks everyone! Sounds like we've agreed to move forward.

@buchgr would you have bandwidth and/or interest in taking a crack at this?

[buchgr]

@nmittler yes if it's not high priority i.e. can't guarantee that I ll have it done by christmas. working on my thesis right now and trying to stay on top of classes. so can't guarantee fast delivery, only eventual delivery :).

[nmittler]

@madongfly FYI

[nmittler]

FYI, I've made a PR to import fork hpack under netty: #4440

[nmittler]

Closing via #4440