net/http: HTTP/2 POST via Akamai (maybe) corrupted

[jeffallen]

I am currently chasing down a poor interaction between Go's HTTP/2, github.com/xenolf/lego, github.com/letsencrypt/boulder and Akamai:
letsencrypt/boulder#1279

The server is seeing different bytes in the body of the POST depending on if the connection comes in as HTTP/2 or not.

I do not have any evidence yet that Go's HTTP/2 is implicated in this, but Brad asked me to file this while we are looking for more info.

I am using:
$ go version
go version devel +85dd62d Sat Dec 12 06:46:56 2015 +0000 darwin/amd64

lego from commit bf740fa2cafb7d6deb0911792a13f37ef5995a03

[jeffallen]

With http2VerboseLogs on, here is a repro:

$ ./lego -m "test12@nella.org" -s "https://acme-staging.api.letsencrypt.org/directory" -d nella.org -d blog.nella.org run
2015/12/16 11:47:40 No key found for account test12@nella.org. Generating a 2048 bit key.
2015/12/16 11:47:41 Saved key to /Users/jra/src/github.com/xenolf/lego/.lego/accounts/acme-staging.api.letsencrypt.org/test12@nella.org/keys/test12@nella.org.key
2015/12/16 11:47:41 Unhandled Setting: [MAX_HEADER_LIST_SIZE = 16384]
2015/12/16 11:47:41 Transport received [FrameHeader WINDOW_UPDATE len=4]: &http.http2WindowUpdateFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x8, Flags:0x0, Length:0x4, StreamID:0x0}, Increment:0x1}
2015/12/16 11:47:41 Transport received [FrameHeader SETTINGS flags=ACK len=0]: &http.http2SettingsFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x4, Flags:0x1, Length:0x0, StreamID:0x0}, p:[]uint8{}}
2015/12/16 11:47:42 Transport received [FrameHeader HEADERS flags=END_HEADERS stream=1 len=244]: &http.http2HeadersFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x1, Flags:0x4, Length:0xf4, StreamID:0x1}, Priority:http.http2PriorityParam{StreamDep:0x0, Exclusive:false, Weight:0x0}, headerFragBuf:[]uint8{0x88, 0x1f, 0x27, 0x5, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0x1f, 0x10, 0x10, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x1f, 0xd, 0x3, 0x32, 0x37, 0x39, 0x10, 0xc, 0x72, 0x65, 0x70, 0x6c, 0x61, 0x79, 0x2d, 0x6e, 0x6f, 0x6e, 0x63, 0x65, 0x2b, 0x71, 0x51, 0x58, 0x63, 0x6b, 0x32, 0x78, 0x51, 0x52, 0x7a, 0x70, 0x79, 0x4f, 0x68, 0x67, 0x79, 0x49, 0x32, 0x52, 0x42, 0x55, 0x6b, 0x47, 0x76, 0x58, 0x44, 0x33, 0x32, 0x74, 0x56, 0x56, 0x32, 0x42, 0x5a, 0x79, 0x42, 0x56, 0x49, 0x35, 0x70, 0x58, 0x79, 0x34, 0x10, 0xf, 0x78, 0x2d, 0x66, 0x72, 0x61, 0x6d, 0x65, 0x2d, 0x6f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x4, 0x44, 0x45, 0x4e, 0x59, 0x1f, 0x29, 0xe, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x36, 0x30, 0x34, 0x38, 0x30, 0x30, 0x1f, 0x15, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x33, 0x20, 0x47, 0x4d, 0x54, 0x1f, 0x9, 0x1d, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x30, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x10, 0x6, 0x70, 0x72, 0x61, 0x67, 0x6d, 0x61, 0x8, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x1f, 0x12, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x33, 0x20, 0x47, 0x4d, 0x54}}
2015/12/16 11:47:42 Header field: {Name::status Value:200 Sensitive:false}
2015/12/16 11:47:42 Header field: {Name:server Value:nginx Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:content-type Value:application/json Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:content-length Value:279 Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:replay-nonce Value:qQXck2xQRzpyOhgyI2RBUkGvXD32tVV2BZyBVI5pXy4 Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:x-frame-options Value:DENY Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:strict-transport-security Value:max-age=604800 Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:expires Value:Wed, 16 Dec 2015 05:47:43 GMT Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:cache-control Value:max-age=0, no-cache, no-store Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:pragma Value:no-cache Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:date Value:Wed, 16 Dec 2015 05:47:43 GMT Sensitive:true}
2015/12/16 11:47:42 Transport received [FrameHeader DATA stream=1 len=279]: &http.http2DataFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x0, Flags:0x0, Length:0x117, StreamID:0x1}, data:[]uint8{0x7b, 0x22, 0x6e, 0x65, 0x77, 0x2d, 0x61, 0x75, 0x74, 0x68, 0x7a, 0x22, 0x3a, 0x22, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2d, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x6c, 0x65, 0x74, 0x73, 0x65, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x2e, 0x6f, 0x72, 0x67, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2f, 0x6e, 0x65, 0x77, 0x2d, 0x61, 0x75, 0x74, 0x68, 0x7a, 0x22, 0x2c, 0x22, 0x6e, 0x65, 0x77, 0x2d, 0x63, 0x65, 0x72, 0x74, 0x22, 0x3a, 0x22, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2d, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x6c, 0x65, 0x74, 0x73, 0x65, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x2e, 0x6f, 0x72, 0x67, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2f, 0x6e, 0x65, 0x77, 0x2d, 0x63, 0x65, 0x72, 0x74, 0x22, 0x2c, 0x22, 0x6e, 0x65, 0x77, 0x2d, 0x72, 0x65, 0x67, 0x22, 0x3a, 0x22, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2d, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x6c, 0x65, 0x74, 0x73, 0x65, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x2e, 0x6f, 0x72, 0x67, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2f, 0x6e, 0x65, 0x77, 0x2d, 0x72, 0x65, 0x67, 0x22, 0x2c, 0x22, 0x72, 0x65, 0x76, 0x6f, 0x6b, 0x65, 0x2d, 0x63, 0x65, 0x72, 0x74, 0x22, 0x3a, 0x22, 0x68, 0x74, 0x74, 0x70, 0x73, 0x3a, 0x2f, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2d, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x2e, 0x61, 0x70, 0x69, 0x2e, 0x6c, 0x65, 0x74, 0x73, 0x65, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x2e, 0x6f, 0x72, 0x67, 0x2f, 0x61, 0x63, 0x6d, 0x65, 0x2f, 0x72, 0x65, 0x76, 0x6f, 0x6b, 0x65, 0x2d, 0x63, 0x65, 0x72, 0x74, 0x22, 0x7d}}
2015/12/16 11:47:42 DATA: "{"new-authz":"https://acme-staging.api.letsencrypt.org/acme/new-authz\",\"new-cert\":\"https://acme-staging.api.letsencrypt.org/acme/new-cert\",\"new-reg\":\"https://acme-staging.api.letsencrypt.org/acme/new-reg\",\"revoke-cert\":\"https://acme-staging.api.letsencrypt.org/acme/revoke-cert\"}"
2015/12/16 11:47:42 Transport received [FrameHeader DATA flags=END_STREAM stream=1 len=0]: &http.http2DataFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x0, Flags:0x1, Length:0x0, StreamID:0x1}, data:[]uint8{}}
2015/12/16 11:47:42 directory: acme.directory{NewAuthzURL:"https://acme-staging.api.letsencrypt.org/acme/new-authz", NewCertURL:"https://acme-staging.api.letsencrypt.org/acme/new-cert", NewRegURL:"https://acme-staging.api.letsencrypt.org/acme/new-reg", RevokeCertURL:"https://acme-staging.api.letsencrypt.org/acme/revoke-cert"}
2015/12/16 11:47:42 DATA: ""
2015/12/16 11:47:42 [INFO] acme: Registering account for test12@nella.org
2015/12/16 11:47:42 Transport received [FrameHeader HEADERS flags=END_HEADERS stream=3 len=242]: &http.http2HeadersFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x1, Flags:0x4, Length:0xf2, StreamID:0x3}, Priority:http.http2PriorityParam{StreamDep:0x0, Exclusive:false, Weight:0x0}, headerFragBuf:[]uint8{0x88, 0x1f, 0x27, 0x5, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0x1f, 0x10, 0x10, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x6a, 0x73, 0x6f, 0x6e, 0x10, 0xc, 0x72, 0x65, 0x70, 0x6c, 0x61, 0x79, 0x2d, 0x6e, 0x6f, 0x6e, 0x63, 0x65, 0x2b, 0x51, 0x41, 0x30, 0x76, 0x5f, 0x62, 0x31, 0x2d, 0x6c, 0x75, 0x44, 0x77, 0x51, 0x42, 0x6c, 0x6e, 0x4b, 0x55, 0x6f, 0x42, 0x6b, 0x53, 0x44, 0x4e, 0x6d, 0x6f, 0x75, 0x6b, 0x32, 0x55, 0x45, 0x77, 0x65, 0x77, 0x48, 0x54, 0x43, 0x34, 0x66, 0x56, 0x58, 0x46, 0x59, 0x10, 0xf, 0x78, 0x2d, 0x66, 0x72, 0x61, 0x6d, 0x65, 0x2d, 0x6f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x4, 0x44, 0x45, 0x4e, 0x59, 0x1f, 0x29, 0xe, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x36, 0x30, 0x34, 0x38, 0x30, 0x30, 0x1f, 0xd, 0x1, 0x30, 0x1f, 0x15, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x34, 0x20, 0x47, 0x4d, 0x54, 0x1f, 0x9, 0x1d, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x30, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x10, 0x6, 0x70, 0x72, 0x61, 0x67, 0x6d, 0x61, 0x8, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x1f, 0x12, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x34, 0x20, 0x47, 0x4d, 0x54}}
2015/12/16 11:47:42 Header field: {Name::status Value:200 Sensitive:false}
2015/12/16 11:47:42 Header field: {Name:server Value:nginx Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:content-type Value:application/json Sensitive:true}
2015/12/16 11:47:42 Header field: {Name:replay-nonce Value:QA0v_b1-luDwQBlnKUoBkSDNmouk2UEwewHTC4fVXFY Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:x-frame-options Value:DENY Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:strict-transport-security Value:max-age=604800 Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:content-length Value:0 Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:expires Value:Wed, 16 Dec 2015 05:47:44 GMT Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:cache-control Value:max-age=0, no-cache, no-store Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:pragma Value:no-cache Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:date Value:Wed, 16 Dec 2015 05:47:44 GMT Sensitive:true}
2015/12/16 11:47:43 Transport received [FrameHeader DATA flags=END_STREAM stream=3 len=0]: &http.http2DataFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x0, Flags:0x1, Length:0x0, StreamID:0x3}, data:[]uint8{}}
2015/12/16 11:47:43 DATA: ""
2015/12/16 11:47:43 POST https://acme-staging.api.letsencrypt.org/acme/new-reg
2015/12/16 11:47:43 {"payload":"eyJyZXNvdXJjZSI6Im5ldy1yZWciLCJjb250YWN0IjpbIm1haWx0bzp0ZXN0MTJAbmVsbGEub3JnIl19","protected":"eyJhbGciOiJSUzI1NiIsImp3ayI6eyJrdHkiOiJSU0EiLCJuIjoidmpBYTk5QXpXWTBLTjhrNXhGZlAyNW90WVRMUGJtTWpuZ29fNmduU0t6NDBxU2Q1bl95R0t5OXJsOFRza1RQUWZLN1hFZWJ4Y2ZmeG5VUUNSbUxCNlZBTU96T0VsMThVb05lZmJ4YXc5RjhWZTVyVzBmY29ScFpvUVZfQlFnTUFjQ041SE5iSk1fNE5kTWU5UDMtcjdhaVZjRXd5RHJic0xPOEh6bEdlQlJRT2dMZzdTUlVJbW5LZS1xR3F0SVhyZDd3bjZVM3hOOHNhZGFpcTktdWlYUEw5eG11elJuMUp2Ync2aGs1ZVlNalR5SUpZS1UxbDFXMFgxVDczODVSN2hOMG1CeWpfWUp6VFJ0Uy1LU19XR3RLZnRuQWgzTUFQNlowZkRicmR3bFNqV1RfMG1FeHhLWkh0cUNrZTR5bkZ5cURNZUNlZTBrTnI1cjAzcHFTSmNRIiwiZSI6IkFRQUIifSwibm9uY2UiOiJRQTB2X2IxLWx1RHdRQmxuS1VvQmtTRE5tb3VrMlVFd2V3SFRDNGZWWEZZIn0","signature":"e1wVw0AEDYnxXN6fEhKkMkBpYRmYftkri-PIchejHfwvz_HP8VXEsceXUrwG3BCVDD9KEMbm0ysb5okX7RTqPQimjiew-nMYUG__loSmEtoUPKaLUT5tz-Eh4Uc8Vk74NV0greqMHvf6rSbR-L7cUWgncYwpovx_Eck5M1_qi1xEZWP3oKOD8klQRKJ7U83QcslkKdtBc4Xrd_LJcFdMqrUfFYwgkdXvSh43-9A-pEXMZo5rUNZ_bR5iDs-WiYL4feNpXBEWqIyxCGwwZf99xiAyQ-PEN30PvEaEdinftcFeNClnH9p77Pqb29zC5XRDfeTYmXS7eMuPSd1Na1PA0g"}
2015/12/16 11:47:43 Transport received [FrameHeader HEADERS flags=END_HEADERS stream=5 len=213]: &http.http2HeadersFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x1, Flags:0x4, Length:0xd5, StreamID:0x5}, Priority:http.http2PriorityParam{StreamDep:0x0, Exclusive:false, Weight:0x0}, headerFragBuf:[]uint8{0x8c, 0x1f, 0x27, 0x5, 0x6e, 0x67, 0x69, 0x6e, 0x78, 0x1f, 0x10, 0x18, 0x61, 0x70, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x70, 0x72, 0x6f, 0x62, 0x6c, 0x65, 0x6d, 0x2b, 0x6a, 0x73, 0x6f, 0x6e, 0x1f, 0xd, 0x3, 0x31, 0x31, 0x33, 0x10, 0xc, 0x72, 0x65, 0x70, 0x6c, 0x61, 0x79, 0x2d, 0x6e, 0x6f, 0x6e, 0x63, 0x65, 0x2b, 0x61, 0x76, 0x49, 0x37, 0x58, 0x56, 0x66, 0x48, 0x75, 0x70, 0x55, 0x43, 0x4c, 0x6d, 0x4e, 0x42, 0x45, 0x73, 0x75, 0x48, 0x38, 0x64, 0x31, 0x47, 0x39, 0x6b, 0x5a, 0x51, 0x74, 0x48, 0x34, 0x33, 0x44, 0x42, 0x41, 0x4c, 0x63, 0x6a, 0x6f, 0x56, 0x58, 0x37, 0x41, 0x1f, 0x15, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x34, 0x20, 0x47, 0x4d, 0x54, 0x1f, 0x9, 0x1d, 0x6d, 0x61, 0x78, 0x2d, 0x61, 0x67, 0x65, 0x3d, 0x30, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x2c, 0x20, 0x6e, 0x6f, 0x2d, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x10, 0x6, 0x70, 0x72, 0x61, 0x67, 0x6d, 0x61, 0x8, 0x6e, 0x6f, 0x2d, 0x63, 0x61, 0x63, 0x68, 0x65, 0x1f, 0x12, 0x1d, 0x57, 0x65, 0x64, 0x2c, 0x20, 0x31, 0x36, 0x20, 0x44, 0x65, 0x63, 0x20, 0x32, 0x30, 0x31, 0x35, 0x20, 0x30, 0x35, 0x3a, 0x34, 0x37, 0x3a, 0x34, 0x34, 0x20, 0x47, 0x4d, 0x54}}
2015/12/16 11:47:43 Header field: {Name::status Value:400 Sensitive:false}
2015/12/16 11:47:43 Header field: {Name:server Value:nginx Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:content-type Value:application/problem+json Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:content-length Value:113 Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:replay-nonce Value:avI7XVfHupUCLmNBEsuH8d1G9kZQtH43DBALcjoVX7A Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:expires Value:Wed, 16 Dec 2015 05:47:44 GMT Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:cache-control Value:max-age=0, no-cache, no-store Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:pragma Value:no-cache Sensitive:true}
2015/12/16 11:47:43 Header field: {Name:date Value:Wed, 16 Dec 2015 05:47:44 GMT Sensitive:true}
2015/12/16 11:47:43 Transport received [FrameHeader DATA stream=5 len=113]: &http.http2DataFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x0, Flags:0x0, Length:0x71, StreamID:0x5}, data:[]uint8{0x7b, 0x22, 0x74, 0x79, 0x70, 0x65, 0x22, 0x3a, 0x22, 0x75, 0x72, 0x6e, 0x3a, 0x61, 0x63, 0x6d, 0x65, 0x3a, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x3a, 0x6d, 0x61, 0x6c, 0x66, 0x6f, 0x72, 0x6d, 0x65, 0x64, 0x22, 0x2c, 0x22, 0x64, 0x65, 0x74, 0x61, 0x69, 0x6c, 0x22, 0x3a, 0x22, 0x55, 0x6e, 0x61, 0x62, 0x6c, 0x65, 0x20, 0x74, 0x6f, 0x20, 0x72, 0x65, 0x61, 0x64, 0x2f, 0x76, 0x65, 0x72, 0x69, 0x66, 0x79, 0x20, 0x62, 0x6f, 0x64, 0x79, 0x20, 0x3a, 0x3a, 0x20, 0x50, 0x61, 0x72, 0x73, 0x65, 0x20, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x20, 0x72, 0x65, 0x61, 0x64, 0x69, 0x6e, 0x67, 0x20, 0x4a, 0x57, 0x53, 0x22, 0x2c, 0x22, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x3a, 0x34, 0x30, 0x30, 0x7d}}
2015/12/16 11:47:43 DATA: "{"type":"urn:acme:error:malformed","detail":"Unable to read/verify body :: Parse error reading JWS","status":400}"
2015/12/16 11:47:43 Transport received [FrameHeader DATA flags=END_STREAM stream=5 len=0]: &http.http2DataFrame{http2FrameHeader:http.http2FrameHeader{valid:true, Type:0x0, Flags:0x1, Length:0x0, StreamID:0x5}, data:[]uint8{}}
2015/12/16 11:47:43 DATA: ""
2015/12/16 11:47:43 Could not complete registration
acme: Error 400 - urn:acme:error:malformed - Unable to read/verify body :: Parse error reading JWS

Lines marked with ** above are debugging I added into lego's jws.go in order to know exactly what was passed into http.Post.

You can use the repro case exactly as this. The test11@nella.org account does not exist in staging, so it will always try to make a new one and fail on the POST new-reg.

The problem, as I see it, is that I can't get visibility into what's coming out of Akami's nginix and going towards letencrypt's origin server.

[jeffallen]

Added logging to h2_bundle.go and confirmed that writeRequestBody is making one sole call to cc.fr.WriteData with all the data in the third argument, and no error returned.

[bradfitz]

Rather than editing h2_bundle.go directly (which is kind of ugly), I find it easier to modify golang.org/x/net/http2 instead and then run the bundle command to bring it into net/http:

#!/bin/bash
set -e
go get golang.org/x/tools/cmd/bundle
bundle golang.org/x/net/http2 net/http http2 > /tmp/bund.go
mv /tmp/bund.go $GOROOT/go/src/net/http/h2_bundle.go

The logs above are too verbose. I think we only care about the headers and data the h1 and h2 transports write. (not read)

[bradfitz]

I left details in letsencrypt/boulder#1279

[bradfitz]

Per letsencrypt/boulder#1279 (comment) , this was confirmed to be a problem on Akamai's side which they're fixing. Closing this bug.