New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx 1.13.1 + HTTP/2 Full HPACK encoding patch verification? #72

Closed
centminmod opened this Issue Jun 23, 2017 · 30 comments

Comments

6 participants
@centminmod

centminmod commented Jun 23, 2017

I patched my Nginx 1.13.1 server with your 1.13.1 version of HTTP/2 full HPACK encoding patch but using h2load tests outlined at https://blog.cloudflare.com/hpack-the-silent-killer-feature-of-http-2/, I am not seeing the header savings expected on subsequent h2load test runs ?

The test page is just a single index.html page so not sure if that matters ?

url=https://domain.com

the headers

curl -I $url  
HTTP/2 200 
date: Fri, 23 Jun 2017 11:36:50 GMT
content-type: text/html; charset=utf-8
content-length: 6090
last-modified: Fri, 23 Jun 2017 11:24:24 GMT
vary: Accept-Encoding
etag: "594cfa68-17ca"
server: nginx centminmod
x-powered-by: centminmod
expires: Sat, 24 Jun 2017 11:36:50 GMT
cache-control: max-age=86400
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
accept-ranges: bytes

h2load test runs 1-4

h2load --version
h2load nghttp2/1.24.0-DEV

with cloudflare HTTP/2 HPACK Full encoding patch

for i in $(seq 1 4); do echo "h2load run $i"; h2load $url -n $i | tail -6 | head -1; done
h2load run 1
traffic: 6.33KB (6481) total, 324B (324) headers (space savings 15.62%), 5.95KB (6090) data
h2load run 2
traffic: 12.61KB (12913) total, 648B (648) headers (space savings 15.62%), 11.89KB (12180) data
h2load run 3
traffic: 18.89KB (19345) total, 972B (972) headers (space savings 15.62%), 17.84KB (18270) data
h2load run 4
traffic: 25.17KB (25777) total, 1.27KB (1296) headers (space savings 15.62%), 23.79KB (24360) dat

without cloudflare HTTP/2 HPACK Full encoding patch strange get higher compression levels/header space savings ?

for i in $(seq 1 4); do echo "h2load run $i"; h2load $url -n $i | tail -6 | head -1; done
h2load run 1
traffic: 6.29KB (6442) total, 285B (285) headers (space savings 25.78%), 5.95KB (6090) data
h2load run 2
traffic: 12.53KB (12835) total, 570B (570) headers (space savings 25.78%), 11.89KB (12180) data
h2load run 3
traffic: 18.78KB (19228) total, 855B (855) headers (space savings 25.78%), 17.84KB (18270) data
h2load run 4
traffic: 25.02KB (25621) total, 1.11KB (1140) headers (space savings 25.78%), 23.79KB (24360) data

4 run h2load full results with HPACK full encoding patch

h2load $url -n 4
starting benchmark...
spawning thread #0: 1 total client(s). 4 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 25% done
progress: 50% done
progress: 75% done
progress: 100% done

finished in 2.22ms, 1798.56 req/s, 11.05MB/s
requests: 4 total, 4 started, 4 done, 4 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 4 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 25.17KB (25777) total, 1.27KB (1296) headers (space savings 15.62%), 23.79KB (24360) data
                     min         max         mean         sd        +/- sd
time for request:       59us       142us        88us        32us    75.00%
time for connect:     1.46ms      1.46ms      1.46ms         0us   100.00%
time to 1st byte:     1.61ms      1.61ms      1.61ms         0us   100.00%
req/s           :    2121.60     2121.60     2121.60        0.00   100.00%

4 run h2load full result without HPACK full encoding patch

h2load $url -n 4
starting benchmark...
spawning thread #0: 1 total client(s). 4 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 25% done
progress: 50% done
progress: 75% done
progress: 100% done

finished in 2.19ms, 1825.65 req/s, 11.15MB/s
requests: 4 total, 4 started, 4 done, 4 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 4 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 25.02KB (25621) total, 1.11KB (1140) headers (space savings 25.78%), 23.79KB (24360) data
                     min         max         mean         sd        +/- sd
time for request:       70us       147us        99us        29us    75.00%
time for connect:     1.41ms      1.41ms      1.41ms         0us   100.00%
time to 1st byte:     1.55ms      1.55ms      1.55ms         0us   100.00%
req/s           :    2124.74     2124.74     2124.74        0.00   100.00%

nghttp2 -nv run with HTTP/2 HPACK full encoding patch

nghttp -nv $url 
[  0.053] Connected
The negotiated protocol: h2
[  0.055] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[  0.055] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.055] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.055] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.055] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.055] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.055] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.055] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.055] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.055] send HEADERS frame <length=48, flags=0x25, stream_id=13>
          ; END_STREAM | END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: GET
          :path: /
          :scheme: https
          :authority: domain.com
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.24.0-DEV
[  0.055] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.055] recv (stream_id=13) :status: 200
[  0.055] recv (stream_id=13) date: Fri, 23 Jun 2017 11:47:37 GMT
[  0.055] recv (stream_id=13) content-type: text/html; charset=utf-8
[  0.055] recv (stream_id=13) last-modified: Fri, 23 Jun 2017 11:24:24 GMT
[  0.055] recv (stream_id=13) vary: Accept-Encoding
[  0.055] recv (stream_id=13) etag: W/"594cfa68-17ca"
[  0.055] recv (stream_id=13) server: nginx centminmod
[  0.055] recv (stream_id=13) x-powered-by: centminmod
[  0.055] recv (stream_id=13) expires: Sat, 24 Jun 2017 11:47:37 GMT
[  0.055] recv (stream_id=13) cache-control: max-age=86400
[  0.055] recv (stream_id=13) x-frame-options: SAMEORIGIN
[  0.055] recv (stream_id=13) x-xss-protection: 1; mode=block
[  0.055] recv (stream_id=13) x-content-type-options: nosniff
[  0.055] recv (stream_id=13) content-encoding: gzip
[  0.055] recv HEADERS frame <length=311, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
[  0.055] recv DATA frame <length=1951, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.055] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

now without HTTP/2 HPACK full encoding patch

nghttp -nv $url 
[  0.043] Connected
The negotiated protocol: h2
[  0.044] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[  0.044] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.044] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.044] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.044] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.044] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.044] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.044] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.044] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.044] send HEADERS frame <length=48, flags=0x25, stream_id=13>
          ; END_STREAM | END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: GET
          :path: /
          :scheme: https
          :authority: domain.com
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.24.0-DEV
[  0.045] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.045] recv (stream_id=13) :status: 200
[  0.045] recv (stream_id=13) date: Fri, 23 Jun 2017 11:58:49 GMT
[  0.045] recv (stream_id=13) content-type: text/html; charset=utf-8
[  0.045] recv (stream_id=13) last-modified: Fri, 23 Jun 2017 11:24:24 GMT
[  0.045] recv (stream_id=13) vary: Accept-Encoding
[  0.045] recv (stream_id=13) etag: W/"594cfa68-17ca"
[  0.045] recv (stream_id=13) server: nginx centminmod
[  0.045] recv (stream_id=13) x-powered-by: centminmod
[  0.045] recv (stream_id=13) expires: Sat, 24 Jun 2017 11:58:49 GMT
[  0.045] recv (stream_id=13) cache-control: max-age=86400
[  0.045] recv (stream_id=13) x-frame-options: SAMEORIGIN
[  0.045] recv (stream_id=13) x-xss-protection: 1; mode=block
[  0.045] recv (stream_id=13) x-content-type-options: nosniff
[  0.045] recv (stream_id=13) content-encoding: gzip
[  0.045] recv HEADERS frame <length=282, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
[  0.045] recv DATA frame <length=1951, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.045] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

nginx -V
nginx version: nginx/1.13.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.2l 25 May 2017
TLS SNI support enabled
configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.0 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0 --add-module=../set-misc-nginx-module-0.31 --add-module=../echo-nginx-module-0.60 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-module=../headers-more-nginx-module-0.32 --with-pcre=../pcre-8.40 --with-pcre-jit --with-zlib=../zlib-1.2.11 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.0.2l

grep -C3 -rnw NGX_HTTP_V2_HPACK_ENC /svr-setup/nginx-1.13.1 | grep -v  '.patch'
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-416-        pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(5),
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-417-                                    h2c->max_hpack_table_size);
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-418-        h2c->indicate_resize = 0;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c:419:#if (NGX_HTTP_V2_HPACK_ENC)
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-420-        ngx_http_v2_table_resize(h2c);
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-421-#endif
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_filter_module.c-422-    }
--
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-349-}
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-350-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-351-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c:352:#if (NGX_HTTP_V2_HPACK_ENC)
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-353-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-354-static ngx_int_t
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2_table.c-355-hpack_get_static_index(ngx_http_v2_connection_t *h2c, u_char *val, size_t len);
--
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-117-} ngx_http_v2_hpack_t;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-118-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-119-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h:120:#if (NGX_HTTP_V2_HPACK_ENC)
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-121-typedef struct {
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-122-    uint64_t                         hash_val;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-123-    uint32_t                         index;
--
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-197-    unsigned                         goaway:1;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-198-    unsigned                         indicate_resize:1;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-199-
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h:200:#if (NGX_HTTP_V2_HPACK_ENC)
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-201-    ngx_http_v2_hpack_enc_t          hpack_enc;
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-202-#endif
/svr-setup/nginx-1.13.1/src/http/v2/ngx_http_v2.h-203-};
--
--
--
--
--
--
/svr-setup/nginx-1.13.1/auto/modules-437-    fi
/svr-setup/nginx-1.13.1/auto/modules-438-
/svr-setup/nginx-1.13.1/auto/modules-439-    if [ $HTTP_V2_HPACK_ENC = YES ]; then
/svr-setup/nginx-1.13.1/auto/modules:440:        have=NGX_HTTP_V2_HPACK_ENC . auto/have
/svr-setup/nginx-1.13.1/auto/modules-441-    fi
/svr-setup/nginx-1.13.1/auto/modules-442-
/svr-setup/nginx-1.13.1/auto/modules-443-    if :; then
@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

Did you specify --with-http_v2_hpack_enc in the configure line?

@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

ah that was missing but with --with-http_v2_hpack_enc i get the following error

ccache gcc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -m64 -march=native -DTCP_FASTOPEN=23 -g -O2 -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf -DNDK_SET_VAR -DNDK_UPSTREAM_LIST -I src/core -I src/event -I src/event/modules -I src/os/unix -I ../ngx_devel_kit-0.3.0/objs -I objs/addon/ndk -I ../pcre-8.40 -I ../openssl-1.0.2l/.openssl/include -I ../zlib-1.2.11 -I objs -I src/http -I src/http/modules -I src/http/v2 -I ../ngx_devel_kit-0.3.0/src -I ../ngx_devel_kit-0.3.0/src -I ../ngx_devel_kit-0.3.0/objs -I objs/addon/ndk \
        -o objs/src/http/v2/ngx_http_v2_table.o \
        src/http/v2/ngx_http_v2_table.c
src/http/v2/ngx_http_v2_table.c: In function ‘hpack_get_static_index’:
src/http/v2/ngx_http_v2_table.c:727:9: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]
         if (pref != *(uint64_t *) header->val) {
         ^
src/http/v2/ngx_http_v2_table.c:741:18: error: dereferencing type-punned pointer will break strict-aliasing rules [-Werror=strict-aliasing]
                  != (*(uint64_t *) &val[i]| 0x2020202020202020) )
                  ^
cc1: all warnings being treated as errors
make[1]: *** [objs/src/http/v2/ngx_http_v2_table.o] Error 1
make[1]: Leaving directory `/svr-setup/nginx-1.13.1'
make: *** [build] Error 2
nginx configure options (alternate ordering):
./configure --with-ld-opt="-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib" --with-cc-opt="-m64 -march=native -DTCP_FASTOPEN=23 -g -O2 -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf" --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.0 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0 --add-module=../set-misc-nginx-module-0.31 --add-module=../echo-nginx-module-0.60 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-module=../headers-more-nginx-module-0.32 --with-pcre=../pcre-8.40 --with-pcre-jit --with-zlib=../zlib-1.2.11 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.0.2l
checking for OS
 + Linux 3.10.0-514.21.2.el7.x86_64 x86_64
checking for C compiler ... found
 + using GNU C compiler
 + gcc version: 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) 
checking for gcc -pipe switch ... found
checking for --with-ld-opt="-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib" ... found
checking for -Wl,-E switch ... found
checking for gcc builtin atomic operations ... found
checking for C99 variadic macros ... found
checking for gcc variadic macros ... found
checking for gcc builtin 64 bit byteswap ... found
checking for unistd.h ... found
checking for inttypes.h ... found
checking for limits.h ... found
checking for sys/filio.h ... not found
checking for sys/param.h ... found
checking for sys/mount.h ... found
checking for sys/statvfs.h ... found
checking for crypt.h ... found
checking for Linux specific features
checking for epoll ... found
checking for EPOLLRDHUP ... found
checking for EPOLLEXCLUSIVE ... not found
checking for O_PATH ... found
checking for sendfile() ... found
checking for sendfile64() ... found
checking for sys/prctl.h ... found
checking for prctl(PR_SET_DUMPABLE) ... found
checking for crypt_r() ... found
checking for sys/vfs.h ... found
checking for nobody group ... found
checking for poll() ... found
checking for /dev/poll ... not found
checking for kqueue ... not found
checking for crypt() ... not found
checking for crypt() in libcrypt ... found
checking for F_READAHEAD ... not found
checking for posix_fadvise() ... found
checking for O_DIRECT ... found
checking for F_NOCACHE ... not found
checking for directio() ... not found
checking for statfs() ... found
checking for statvfs() ... found
checking for dlopen() ... not found
checking for dlopen() in libdl ... found
checking for sched_yield() ... found
checking for sched_setaffinity() ... found
checking for SO_SETFIB ... not found
checking for SO_REUSEPORT ... found
checking for SO_ACCEPTFILTER ... not found
checking for SO_BINDANY ... not found
checking for IP_BIND_ADDRESS_NO_PORT ... not found
checking for IP_TRANSPARENT ... found
checking for IP_BINDANY ... not found
checking for IP_RECVDSTADDR ... not found
checking for IP_SENDSRCADDR ... not found
checking for IP_PKTINFO ... found
checking for IPV6_RECVPKTINFO ... found
checking for TCP_DEFER_ACCEPT ... found
checking for TCP_KEEPIDLE ... found
checking for TCP_FASTOPEN ... found
checking for TCP_INFO ... found
checking for accept4() ... found
checking for eventfd() ... found
checking for int size ... 4 bytes
checking for long size ... 8 bytes
checking for long long size ... 8 bytes
checking for void * size ... 8 bytes
checking for uint32_t ... found
checking for uint64_t ... found
checking for sig_atomic_t ... found
checking for sig_atomic_t size ... 4 bytes
checking for socklen_t ... found
checking for in_addr_t ... found
checking for in_port_t ... found
checking for rlim_t ... found
checking for uintptr_t ... uintptr_t found
checking for system byte ordering ... little endian
checking for size_t size ... 8 bytes
checking for off_t size ... 8 bytes
checking for time_t size ... 8 bytes
checking for AF_INET6 ... found
checking for setproctitle() ... not found
checking for pread() ... found
checking for pwrite() ... found
checking for pwritev() ... found
checking for sys_nerr ... found
checking for localtime_r() ... found
checking for posix_memalign() ... found
checking for memalign() ... found
checking for mmap(MAP_ANON|MAP_SHARED) ... found
checking for mmap("/dev/zero", MAP_SHARED) ... found
checking for System V shared memory ... found
checking for POSIX semaphores ... not found
checking for POSIX semaphores in libpthread ... found
checking for struct msghdr.msg_control ... found
checking for ioctl(FIONBIO) ... found
checking for struct tm.tm_gmtoff ... found
checking for struct dirent.d_namlen ... not found
checking for struct dirent.d_type ... found
checking for sysconf(_SC_NPROCESSORS_ONLN) ... found
checking for openat(), fstatat() ... found
checking for getaddrinfo() ... found
configuring additional modules
adding module in ../ngx_cache_purge-2.3
 + ngx_http_cache_purge_module was configured
adding module in ../ngx_devel_kit-0.3.0
 + ngx_devel_kit was configured
adding module in ../set-misc-nginx-module-0.31
found ngx_devel_kit for ngx_set_misc; looks good.
 + ngx_http_set_misc_module was configured
adding module in ../echo-nginx-module-0.60
 + ngx_http_echo_module was configured
adding module in ../redis2-nginx-module-0.14
 + ngx_http_redis2_module was configured
adding module in ../ngx_http_redis-0.3.7
 + ngx_http_redis_module was configured
adding module in ../memc-nginx-module-0.18
 + ngx_http_memc_module was configured
adding module in ../srcache-nginx-module-0.31
 + ngx_http_srcache_filter_module was configured
adding module in ../headers-more-nginx-module-0.32
 + ngx_http_headers_more_filter_module was configured
configuring additional dynamic modules
adding module in ../ngx-fancyindex-0.4.0
 + ngx_http_fancyindex_module was configured
checking for GD library ... found
checking for GD WebP support ... not found
checking for GeoIP library ... found
checking for GeoIP IPv6 support ... found
checking for atomic_ops library ... found
creating objs/Makefile

Configuration summary
  + using threads
  + using PCRE library: ../pcre-8.40
  + using OpenSSL library: ../openssl-1.0.2l
  + using zlib library: ../zlib-1.2.11
  + using system libatomic_ops library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"


Fri Jun 23 12:36:40 UTC 2017
Success: Nginx configure ok
@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

I also get that error. Removing -O3 did the trick (unfortunately). See also #71 (comment).

@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

I setup my script to detect if hpack patch is used and if set -O2 instead of -O3 but do you mean totally remove -O2 or -O3 ?

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

I totally removed -O3, just to be sure, I just finished my first successful build some minutes ago. Now I have this up & running I will experiment with -O2 and with adding back the great -flto step by step, to see if I can squeeze some more performance out of it again.

@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

Ok add missing --with-http_v2_hpack_enc configure option and confirmed if i remove -O2 or -O3 during Nginx compile when HPACK patch detected in my script, then yes Nginx 1.13.1 compiles with full HPACK encoding patch ! Thanks @HansVanEijsden

confirm the improved header savings in h2load tests :)

url=https://domain.com
for i in $(seq 1 4); do echo "h2load run $i"; h2load $url -n $i | tail -6 | head -1; done
h2load run 1
traffic: 6.29KB (6441) total, 284B (284) headers (space savings 26.04%), 5.95KB (6090) data
h2load run 2
traffic: 12.45KB (12752) total, 487B (487) headers (space savings 36.59%), 11.89KB (12180) data
h2load run 3
traffic: 18.62KB (19063) total, 690B (690) headers (space savings 40.10%), 17.84KB (18270) data
h2load run 4
traffic: 24.78KB (25374) total, 893B (893) headers (space savings 41.86%), 23.79KB (24360) data

nginx -V
nginx version: nginx/1.13.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.2l 25 May 2017
TLS SNI support enabled
configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -DTCP_FASTOPEN=23 -g -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.0 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0 --add-module=../set-misc-nginx-module-0.31 --add-module=../echo-nginx-module-0.60 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-module=../headers-more-nginx-module-0.32 --with-pcre=../pcre-8.40 --with-pcre-jit --with-zlib=../zlib-1.2.11 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.0.2l

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

Glad you have it working @centminmod 😃
Adding -O2 gives the error too, so both -O3 and -O2 don't work. But adding back -flto makes a succesful build, fortunately!

nginx version: nginx/1.13.1
built by gcc 4.9.2 (Debian 4.9.2-10)
built with OpenSSL 1.0.2l 25 May 2017
TLS SNI support enabled
configure arguments: --prefix=/opt/nginx --user=www-data --group=www-data --add-module=/usr/local/src/ngx_brotli_module --with-http_v2_module --with-http_ssl_module --with-http_v2_hpack_enc --with-openssl=/usr/local/src/openssl-1.0.2l --with-openssl-opt='enable-ec_nistp_64_gcc_128 -DCFLAGS='-march=native'' --with-pcre-jit --with-file-aio --with-http_flv_module --with-http_geoip_module --with-http_mp4_module --with-http_realip_module --with-http_stub_status_module --with-threads --add-module=/usr/local/src/headers-more-nginx-module --add-module=/usr/local/src/echo-nginx-module --add-module=/usr/local/src/ngx_http_substitutions_filter_module --add-module=/usr/local/src/srcache-nginx-module --add-module=/usr/local/src/redis2-nginx-module --add-module=/usr/local/src/ngx_http_redis-0.3.8 --add-module=/usr/local/src/ngx_devel_kit --add-module=/usr/local/src/set-misc-nginx-module --with-cc-opt='-DTCP_FASTOPEN=23 -march=native -flto -fstack-protector-strong -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --with-ld-opt=-Wl,-z,relro

@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

nice to know, wasn't sure if -flto is of any benefit in nginx ? i saw regressions for performance but I only test with GCC 4.8.5, 5.3.1 and 6.2.1 on CentOS 7.

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

To me, -flto saved 2% - 5% CPU time on controlled Wordpress testing. I have great results with it, especially when I also build openssl with it, by using --with-openssl-opt='enable-ec_nistp_64_gcc_128 -DCFLAGS='-march=native -O3 -flto'' (will be my next step).

$ uname -a
Linux vps 4.9.0-0.bpo.3-amd64 #1 SMP Debian 4.9.25-1~bpo8+1 (2017-05-19) x86_64 GNU/Linux
@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

sweet thanks @HansVanEijsden will retest and see :)

nginx -V
nginx version: nginx/1.13.1
built by gcc 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC)
built with OpenSSL 1.0.2l 25 May 2017
TLS SNI support enabled
configure arguments: --with-ld-opt='-ljemalloc -Wl,-z,relro -Wl,-rpath,/usr/local/lib' --with-cc-opt='-m64 -march=native -DTCP_FASTOPEN=23 -g -fstack-protector-strong -flto -fuse-ld=gold --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream=dynamic --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.0 --add-module=../ngx_cache_purge-2.3 --add-module=../ngx_devel_kit-0.3.0 --add-module=../set-misc-nginx-module-0.31 --add-module=../echo-nginx-module-0.60 --add-module=../redis2-nginx-module-0.14 --add-module=../ngx_http_redis-0.3.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-module=../headers-more-nginx-module-0.32 --with-pcre=../pcre-8.40 --with-pcre-jit --with-zlib=../zlib-1.2.11 --with-http_ssl_module --with-http_v2_module --with-http_v2_hpack_enc --with-openssl=../openssl-1.0.2l

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Jun 23, 2017

BTW, you don't need to use "enable-ec_nistp_64_gcc_128" in openssl anymore.

@centminmod

This comment has been minimized.

centminmod commented Jun 23, 2017

@vkrasnov thanks that applies to both openssl 1.0.2 and 1.1 branches or just openssl 1.1 ?

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Jun 23, 2017

@centminmod both. You can compile and then run : 'openssl speed ecdsap256' to check.

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

@vkrasnov strange, but thanks - it's still in the OpenSSL install instructions: https://github.com/openssl/openssl/blob/master/INSTALL

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Jun 23, 2017

@HansVanEijsden. Yes, but it does no longer apply for x86-64 and armv8, that have faster assembly code instead.

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jun 23, 2017

HAPPY!!

$ h2load https://www.hansvaneijsden.com -n 100  | tail -6 |head -1  
traffic: 5.87MB (6158533) total, 1.74KB (1784) headers (space savings 97.11%), 5.86MB (6148600) data
@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Jun 26, 2017

@centminmod can I close this one?

@centminmod

This comment has been minimized.

centminmod commented Jun 27, 2017

Yup close it

@centminmod centminmod closed this Jun 27, 2017

@Wonderfall

This comment has been minimized.

Wonderfall commented Jul 11, 2017

Are you sure there is no other way than disabling compilation optimizations? Tried to compile nginx (using this script) with this patch but unfortunately the compilation fails like discussed here.

@HansVanEijsden

This comment has been minimized.

HansVanEijsden commented Jul 11, 2017

@Wonderfall there is. See #71 (comment) - use -Wno-error=strict-aliasing.

@Wonderfall

This comment has been minimized.

Wonderfall commented Jul 11, 2017

@HansVanEijsden That was fast! I'll try that, thanks.

EDIT : It works as expected!

@fanendra22

This comment has been minimized.

fanendra22 commented Sep 8, 2017

Hey Guys,

I was trying to get it done however I am not getting any savings. Please find my nginx build details

nginx -V nginx version: nginx/1.13.5 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) built with OpenSSL 1.0.2l 25 May 2017 TLS SNI support enabled configure arguments: --user=nginx --group=nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-select_module --with-poll_module --with-file-aio --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-file-aio --with-ipv6 --with-http_v2_module --with-http_realip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-google_perftools_module --with-cpp_test_module --with-debug --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-http_v2_module --with-threads --with-cpu-opt=CPU --with-pcre --with-pcre-jit --with-sha1-asm --with-zlib-asm=CPU --with-libatomic --with-debug --with-http_geoip_module --with-ld-opt=-Wl,-E --with-openssl=/usr/local/src/openssl-1.0.2l/ --with-http_v2_hpack_enc

I am using nginx 1.13.5 with the patch 1.13.1 and patch was successful. Please find the snap shot for the patch
[root@nprssapp15100 nginx-1.13.5]# patch -p1 < nginx_1.13.1_http2_hpack.patch patching file auto/modules patching file auto/options Hunk #2 succeeded at 223 (offset 1 line). Hunk #3 succeeded at 434 (offset 2 lines). patching file src/core/ngx_murmurhash.c patching file src/core/ngx_murmurhash.h patching file src/http/v2/ngx_http_v2.c Hunk #2 succeeded at 2020 (offset 5 lines). patching file src/http/v2/ngx_http_v2.h Hunk #5 succeeded at 401 (offset 9 lines). patching file src/http/v2/ngx_http_v2_filter_module.c Hunk #2 succeeded at 48 with fuzz 2 (offset 2 lines). Hunk #3 succeeded at 133 (offset 4 lines). Hunk #4 succeeded at 142 (offset 4 lines). Hunk #5 succeeded at 405 (offset 4 lines). Hunk #6 succeeded at 413 (offset 4 lines). Hunk #7 succeeded at 433 (offset 4 lines). Hunk #8 succeeded at 480 (offset 4 lines). Hunk #9 succeeded at 532 with fuzz 1 (offset 4 lines). Hunk #10 succeeded at 674 (offset 116 lines). Hunk #11 succeeded at 700 (offset 116 lines). patching file src/http/v2/ngx_http_v2_table.c
Here is the output for h2load
[root@nbtmapp4285 fanendra.tripathi]# h2load https://nprelease.indiatimes.com/feeds/photo/49859087.cms -n 6 | tail -6 |head -1 traffic: 26.06KB (26688) total, 1.12KB (1146) headers (space savings 0.00%), 24.64KB (25236) data [root@nbtmapp4285 fanendra.tripathi]# h2load https://nprelease.indiatimes.com/feeds/photo/49859087.cms -n 100 | tail -6 |head -1 traffic: 430.03KB (440352) total, 18.65KB (19095) headers (space savings 0.00%), 410.74KB (420600) data
I am naive in nginx and was trying to setup hpack since we are having many headers in our requests/responses (app generates for event capturing). It would be of great help if you can guide me where is the problem or help me to identify the problem.

@kn007

This comment has been minimized.

kn007 commented Sep 9, 2017

@fanendra22 Try using gcc 6.2.1 compile it.

h2load https://kn007.net/ -n 100 | tail -6 |head -1
traffic: 2.14MB (2246317) total, 8.55KB (8759) headers (space savings 76.39%), 2.13MB (2233900) data
@fanendra22

This comment has been minimized.

fanendra22 commented Sep 9, 2017

@kn007 Unfortunately it didn't work

[root@nprssapp15100 nginx-1.13.5]# date Sat Sep 9 09:39:49 IST 2017 [root@nprssapp15100 nginx-1.13.5]# [root@nprssapp15100 nginx-1.13.5]# nginx -V nginx version: nginx/1.13.5 built by gcc 6.2.1 20160916 (Red Hat 6.2.1-3) (GCC) built with OpenSSL 1.0.2l 25 May 2017 TLS SNI support enabled configure arguments: --user=nginx --group=nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-select_module --with-poll_module --with-file-aio --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module --with-http_image_filter_module --with-http_geoip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-file-aio --with-ipv6 --with-http_v2_module --with-http_realip_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module --with-mail --with-mail_ssl_module --with-stream --with-stream_ssl_module --with-google_perftools_module --with-cpp_test_module --with-debug --with-http_degradation_module --with-http_stub_status_module --with-http_perl_module --with-http_v2_module --with-threads --with-cpu-opt=CPU --with-pcre --with-pcre-jit --with-sha1-asm --with-zlib-asm=CPU --with-libatomic --with-debug --with-http_geoip_module --with-ld-opt=-Wl,-E --with-openssl=/usr/local/src/openssl-1.0.2l/ --with-http_v2_hpack_enc [root@nprssapp15100 nginx-1.13.5]# [root@nprssapp15100 nginx-1.13.5]# h2load https://nprelease.indiatimes.com/feeds/photo/49859087.cms -n 10 | tail -6 |head -1 traffic: 43.44KB (44480) total, 1.87KB (1910) headers (space savings 0.00%), 41.07KB (42060) data
Could you please check if I need to include any other module in nginx which is required for hpack to work.

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Sep 9, 2017

It looks like you are using http/1. Are you sure you have http2 enabled in the config file?

@fanendra22

This comment has been minimized.

fanendra22 commented Sep 9, 2017

Hi @vkrasnov, I have verified it and it is on http/2.0. This domain is public so even you can check it is
on http/2.0.
screen shot 2017-09-09 at 9 58 37 am
screen shot 2017-09-09 at 9 58 19 am

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Sep 9, 2017

@fanendra22, I did, and that is what I'm getting:

h2load https://nprelease.indiatimes.com/ -n 2
starting benchmark...
spawning thread #0: 1 total client(s). 2 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES256-SHA384
No protocol negotiated. Fallback behaviour may be activated

Server does not support NPN/ALPN. Falling back to HTTP/1.1.

Application protocol: http/1.1
progress: 50% done
progress: 100% done

finished in 2.38s, 0.84 req/s, 892B/s
requests: 2 total, 2 started, 2 done, 2 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 2 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 2.07KB (2124) total, 528B (528) headers (space savings 0.00%), 1.44KB (1478) data
min max mean sd +/- sd
time for request: 410.07ms 751.16ms 580.61ms 170.54ms 100.00%
time for connect: 1.21s 1.21s 1.21s 0us 100.00%
time to 1st byte: 1.96s 1.96s 1.96s 0us 100.00%
req/s : 0.84 0.84 0.84 0.00 100.00%

It does look like chrome gets http/2, but h2load fails. First time I see something like that.

@fanendra22

This comment has been minimized.

fanendra22 commented Sep 9, 2017

@vkrasnov Yeah. Now I see. I can also see in logs the protocol being used is http/1.1 while in client chrome/firefox I can see it as http/2.0 for the same request

@fanendra22

This comment has been minimized.

fanendra22 commented Sep 9, 2017

@vkrasnov Interestingly I found this while checking ALPN status of the server

[root@nprssapp15100 nginx]# echo | openssl s_client -alpn h2 -connect nprelease.indiatimes.com:443 | grep ALPN depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root verify error:num=19:self signed certificate in certificate chain DONE ALPN protocol: h2

It says the certificate chain contains self signed certificate. Could this be a problem while doing SSL handshake and is leading the protocol to be used as http/1.1 instead of http/2.0

@vkrasnov

This comment has been minimized.

Contributor

vkrasnov commented Sep 9, 2017

I don't think thats the problem. Also it uses openssl, and as you can see openssl works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment