Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HA Proxy worker nodes segfault when using h2 and under load #373

Closed
phaze-debug opened this issue Nov 21, 2019 · 9 comments

Comments

@phaze-debug
Copy link

@phaze-debug phaze-debug commented Nov 21, 2019

Output of haproxy -vv and uname -a

HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1c  28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services : none

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

What's the configuration?

frontend server_process_api_18604
bind *:18604 proto h2
mode http
option http-use-htx
use_backend server_process_api_18604

backend server_process_api_18604
balance roundrobin
mode http
option forwardfor
option http-use-htx
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto http
server xxx_xxx_xx_xx_xxxxx xxx_xxx_xx_xx:xxxxx proto h2
server xxx_xxx_xx_xx_xxxxx xxx_xxx_xx_xx:xxxxx proto h2
server xxx_xxx_xx_xx_xxxxx xxx_xxx_xx_xx:xxxxx proto h2

Steps to reproduce the behavior

Our setup - Two java processes (JDK 8, Tomcat 8.5, HttpClient - OkHTTP) communicating via HA Proxy over Http/2 (h2c). In order to verify if this issue has anything to do with tomcat, we removed the ha proxy layer and didn’t observe any issues with connection/streams(with the client hitting the server endpoint directly). Saw same issue with 2.0.9 as well 11/15 release

Test Flow -

Jmeter -> Test Client (This is http 1.1 and HA Proxy is not involved)
Test Client -> HA Proxy (Test Client is using OkHTTP library for h2c calls, Spring boot process running with embedded tomcat 8.5)
HA Proxy -> Test Server (Server process is also Spring boot and running with embedded tomcat 8.5 with upgraded protocol and supports h2c)

Actual behavior

dockerd-current[2637011]: [ALERT] 287/160651 (96) : Current worker #1 (101) exited with code 139 (Segmentation fault)

Snippet of backtrace

#0 0x000055ba0910ad94 in b_alloc_margin (margin=0, buf=0x55ba09256380 <__compound_literal.7+128>) at include/common/buffer.h:165
160 cached = 0;
161 idx = pool_get_index(pool_head_buffer);
162 if (idx >= 0)
163 cached = pool_cache[tid][idx].count;
164
165 *buf = BUF_WANTED;
166
167 #ifndef CONFIG_HAP_LOCKLESS_POOLS
168 HA_SPIN_LOCK(POOL_LOCK, &pool_head_buffer->lock);
169 #endif
#1 h2_get_buf (h2c=h2c@entry=0x7f8aa403bdb0, bptr=bptr@entry=0x55ba09256380 <__compound_literal.7+128>) at src/mux_h2.c:436
436 unlikely((buf = b_alloc_margin(bptr, 0)) == NULL)) {
431 static inline struct buffer *h2_get_buf(struct h2c *h2c, struct buffer *bptr)
432 {
433 struct buffer buf = NULL;
434
435 if (likely(!LIST_ADDED(&h2c->buf_wait.list)) &&
436 unlikely((buf = b_alloc_margin(bptr, 0)) == NULL)) {
437 h2c->buf_wait.target = h2c;
438 h2c->buf_wait.wakeup_cb = h2_buf_available;
439 HA_SPIN_LOCK(BUF_WQ_LOCK, &buffer_wq_lock);
440 LIST_ADDQ(&buffer_wq, &h2c->buf_wait.list);
#2 0x000055ba0910d0c2 in h2c_decode_headers (h2c=h2c@entry=0x7f8aa403bdb0, rxbuf=rxbuf@entry=0x55ba09256380 <__compound_literal.7+128>, flags=flags@entry=0x55ba0925636c <__compound_literal.7+108>, body_len=body_len@entry=0x55ba09256378 <__compound_literal.7+120>) at src/mux_h2.c:3689
3689 if (!h2_get_buf(h2c, rxbuf)) {
3684
3685 hdrs += 5; // stream dep = 4, weight = 1
3686 flen -= 5;
3687 }
3688
3689 if (!h2_get_buf(h2c, rxbuf)) {
3690 h2c->flags |= H2_CF_DEM_SALLOC;
3691 goto leave;
3692 }
3693
#3 0x000055ba09111d41 in h2c_bck_handle_headers (h2s=, h2c=0x7f8aa403bdb0) at src/mux_h2.c:2145
2145 error = h2c_decode_headers(h2c, &h2s->rxbuf, &h2s->flags, &h2s->body_len);
2140
2141 if (b_data(&h2c->dbuf) < h2c->dfl && !b_full(&h2c->dbuf))
2142 return NULL; // incomplete frame
2143
2144 if (h2s->st != H2_SS_CLOSED) {
2145 error = h2c_decode_headers(h2c, &h2s->rxbuf, &h2s->flags, &h2s->body_len);
2146 }
2147 else {
2148 / the connection was already killed by an RST, let’s consume
2149 * the data and send another RST.

Expected behavior

No segmentation fault

Do you have any idea what may have caused this?

No

Do you have an idea how to solve the issue?

No

@TimWolla

This comment has been minimized.

Copy link
Contributor

@TimWolla TimWolla commented Nov 21, 2019

Note to other people reading this issue (because I also missed it at first):

Saw same issue with 2.0.9 as well 11/15 release

@TimWolla

This comment has been minimized.

Copy link
Contributor

@TimWolla TimWolla commented Nov 21, 2019

@phaze-debug Which version is the stack trace and disassembly from? For me the line numbers match up with neither 2.0.3 nor 2.0.9.

@phaze-debug

This comment has been minimized.

Copy link
Author

@phaze-debug phaze-debug commented Nov 21, 2019

Sorry for the confusion...The stack trace is from 2.0.8 image. To be clear, we tried all versions from 2.0.3-2.0.9, but faced the same issue. The vv output is from the currently deployed 2.0.3 version

@TimWolla

This comment has been minimized.

Copy link
Contributor

@TimWolla TimWolla commented Nov 21, 2019

@phaze-debug If this crash easy for you to trigger within your test setting: Can you give the full non-shortened backtrace? Ideally from a 2.0.9.

Put it into a code block (three backticks ```) to ensure that GitHub is not breaking it by attempting to interpret it as markdown.

@phaze-debug

This comment has been minimized.

Copy link
Author

@phaze-debug phaze-debug commented Nov 21, 2019

This is the full back trace from 2.0.8. Had to replace some of the test data with "data here" because they cannot be shared. We will try to run with 2.0.9 soon and get the full bt

#0  0x000055ba0910ad94 in b_alloc_margin (margin=0, buf=0x55ba09256380 <__compound_literal.7+128>) at include/common/buffer.h:165
        area = <optimized out>
        idx = <optimized out>
        cached = 13
        area = <optimized out>
        idx = <optimized out>
        cached = <optimized out>
#1  h2_get_buf (h2c=h2c@entry=0x7f8aa403bdb0, bptr=bptr@entry=0x55ba09256380 <__compound_literal.7+128>) at src/mux_h2.c:436
        buf = 0x0
#2  0x000055ba0910d0c2 in h2c_decode_headers (h2c=h2c@entry=0x7f8aa403bdb0, rxbuf=rxbuf@entry=0x55ba09256380 <__compound_literal.7+128>, flags=flags@entry=0x55ba0925636c <__compound_literal.7+108>, body_len=body_len@entry=0x55ba09256378 <__compound_literal.7+120>) at src/mux_h2.c:3689
        hdrs = 0x55ba09b6a3a0 "?\341\037\210\305\304\303\302\301\300\277\276\037\022\226\337=\277J\t\245\065\021*\b\001}A\006\340\034\270'\324\305\243\177"
        tmp = 0x7f8aaaeb5590
        list = {{n = {ptr = 0x0, len = 2}, v = {ptr = 0x55ba09277caf "GET", len = 3}}, {n = {ptr = 0x0, len = 3}, v = {ptr = 0x7f8aa4009bc1 "<data here>", len = 85}}, {n = {ptr = 0x7f8aa4009c2c ":authority", len = 10}, v = {ptr = 0x7f8aa4009c17 "<data here>", len = 20}}, {n = {ptr = 0x0, len = 4}, v = {ptr = 0x55ba09268904 "http", len = 4}}, {n = {ptr = 0x7f8aa4009c56 "content-type", len = 12}, v = {ptr = 0x7f8aa4009c37 "<data here>", len = 30}}, {n = {ptr = 0x7f8aa4009c6e "cache-control", len = 13}, v = {ptr = 0x7f8aa4009c63 "<data here>", len = 10}}, {n = {ptr = 0x7f8aa4009ca1 "<data here>", len = 8}, v = {ptr = 0x7f8aa4009c7c "<data here>", len = 36}}, {n = {ptr = 0x7f8aa4009ce5 "accept", len = 6}, v = {ptr = 0x7f8aa4009caa "<data here>", len = 58}}, {n = {ptr = 0x7f8aa4009cf1 "h2", len = 2}, v = {ptr = 0x7f8aa4009cec "true", len = 4}}, {n = {ptr = 0x7f8aa4009cf6 "content-length", len = 14}, v = {ptr = 0x7f8aa4009cf4 "0", len = 1}}, {n = {ptr = 0x7f8aa4009d0a "accept-encoding", len = 15}, v = {ptr = 0x7f8aa4009d05 "gzip", len = 4}}, {n = {ptr = 0x7f8aa4009d28 "user-agent", len = 10}, v = {ptr = 0x7f8aa4009d1a "okhttp/3.14.0", len = 13}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x0, len = 0}} <repeats 74 times>, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x0, len = 140233549747752}}, {n = {ptr = 0x50000 <error: Cannot access memory at address 0x50000>, len = 18446744073709551615}, v = {ptr = 0x0, len = 0}}, {n = {ptr = 0x7f8aaaeb1c10 "\300\"\353\252\212\177", len = 1}, v = {ptr = 0x7f8aaaeb1c20 "\001\200\255\373", len = 1}}, {n = {ptr = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 140233564138260}, v = {ptr = 0x0, len = 94257505040436}}, {n = {ptr = 0x7f8a9c043d02 "<data here>"..., len = 140233433935296}, v = {ptr = 0x7f8aabd8ee65 <dot> ".", len = 0}}, {n = {ptr = 0x7f8aaaeb1d50 "\002", len = 0}, v = {ptr = 0x0, len = 0}}, {n = {ptr = 0x0, len = 94257505041305}, v = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 140233549747840}}, {n = {ptr = 0x6500000000 <error: Cannot access memory at address 0x6500000000>, len = 140230682214497}, v = {ptr = 0x7f8aa405b7cb "\t\272U", len = 94257505314822}}, {n = {ptr = 0x8 <error: Cannot access memory at address 0x8>, len = 140233549747696}, v = {ptr = 0x7f8aaaeb23c0 "", len = 7}}, {n = {ptr = 0x65 <error: Cannot access memory at address 0x65>, len = 140233299869152}, v = {ptr = 0x17d <error: Cannot access memory at address 0x17d>, len = 94257504411514}}, {n = {ptr = 0x7f8a9c063e58 "<data here>"..., len = 12}, v = {ptr = 0x7f8a9c063e64 "<data here>"..., len = 30}}, {n = {ptr = 0x7f8a9c063e82 "<data here>"..., len = 13}, v = {ptr = 0x7f8a9c063e8f "<data here>"..., len = 10}}, {n = {ptr = 0x7f8a9c063e99 "<data here>"..., len = 8}, v = {ptr = 0x7f8a9c063ea1 "<data here>"..., len = 36}}, {n = {ptr = 0x7f8a9c063ec5 "<data here>"..., len = 6}, v = {ptr = 0x7f8a9c063ecb "<data here>"..., len = 58}}, {n = {ptr = 0x7f8a9c063f05 "<data here>"..., len = 2}, v = {ptr = 0x7f8a9c063f07 "<data here>"..., len = 4}}, {n = {ptr = 0x7f8a9c063f0b "<data here>"..., len = 14}, v = {ptr = 0x7f8a9c063f19 "<data here>"..., len = 1}}, {n = {ptr = 0x7f8a9c063f1a "<data here>"..., len = 15}, v = {ptr = 0x7f8a9c063f29 "<data here>"..., len = 4}}, {n = {ptr = 0x7f8a9c063f2d "<data here>"..., len = 10}, v = {ptr = 0x7f8a9c063f37 "<data here>"..., len = 13}}, {n = {ptr = 0x7f8a9c063f44 "<data here>"..., len = 4}, v = {ptr = 0x7f8a9c063f48 "<data here>"..., len = 20}}, {n = {ptr = 0x55ba0926ddee "", len = 0}, v = {ptr = 0x7f8aa405b90c "<data here>"..., len = 1}}, {n = {ptr = 0x7f8aa405b90d "<data here>"..., len = 15}, v = {ptr = 0x7f8aa405b91c "<data here>"..., len = 4}}, {n = {ptr = 0x7f8aa405b920 "<data here>"..., len = 10}, v = {ptr = 0x7f8aa405b92a "<data here>"..., len = 13}}, {n = {ptr = 0x7f8aa405b937 "<data here>"..., len = 4}, v = {ptr = 0x7f8aa405b93b "<data here>"..., len = 20}}, {n = {ptr = 0x55ba0926ddee "", len = 0}, v = {ptr = 0x7f8aac3236e0 <__funlockfile> "H\213\227\210", len = 140233549749648}}, {n = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 0}, v = {ptr = 0x0, len = 0}}, {n = {ptr = 0x0, len = 206158430240}, v = {ptr = 0x7f8aaaeb1fa0 "\360\"\353\252\212\177", len = 140233549749984}}, {n = {ptr = 0xd68 <error: Cannot access memory at address 0xd68>, len = 94257505826205}, v = {ptr = 0xb <error: Cannot access memory at address 0xb>, len = 94257505826200}}, {n = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 94257505826234}, v = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 140233565362272}}, {n = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 0}, v = {ptr = 0x7f8aaaeb1e50 "\001\200\255\373\060", len = 140233565552544}}, {n = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 140233564213932}, v = {ptr = 0x7f8aaaeb1a00 "\240 \353\252\212\177", len = 140233564096702}}, {n = {ptr = 0x3638322e373731 <error: Cannot access memory at address 0x3638322e373731>, len = 54}, v = {ptr = 0x0, len = 18204466196317010944}}, {n = {ptr = 0x7f8aaaeb1f30 "", len = 140233549749840}, v = {ptr = 0x7f8aaaeb1e40 "0K", len = 21}}, {n = {ptr = 0x7f8aaaeb1fc0 "\001\200\255\373\060", len = 94257505826234}, v = {ptr = 0x7f8aabdc3fa0 <_IO_strn_jumps> "", len = 140233564078354}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x1d <error: Cannot access memory at address 0x1d>, len = 0}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x7f8aaaeb1a20 "\360 \353\252\212\177", len = 18446744073709551615}}, {n = {ptr = 0x0, len = 7}, v = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 94257505820142}}, {n = {ptr = 0x0, len = 94257505819904}, v = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 0}}, {n = {ptr = 0x0, len = 140233565550432}, v = {ptr = 0xd68 <error: Cannot access memory at address 0xd68>, len = 94257505826205}}, {n = {ptr = 0xb <error: Cannot access memory at address 0xb>, len = 94257505826200}, v = {ptr = 0x2382 <error: Cannot access memory at address 0x2382>, len = 94257505826234}}, {n = {ptr = 0x0, len = 140230682214400}, v = {ptr = 0x7f8aaaeb1ac0 "\330!\353\252\212\177", len = 18446744073709551615}}, {n = {ptr = 0x55ba0926f5b9 "s", len = 6}, v = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 10}}, {n = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 94257505870003}, v = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 4294967292}}, {n = {ptr = 0x0, len = 206158430232}, v = {ptr = 0x7f8aaaeb20a0 "OW", len = 140233549750208}}, {n = {ptr = 0x7f8aaaeb1fb0 "\320!\353\252\212\177", len = 14}, v = {ptr = 0x7f8aaaeb20f0 " \360\254\t\272U", len = 140233565356275}}, {n = {ptr = 0x7f8aabdc4360 <_IO_str_jumps> "", len = 140233564078354}, v = {ptr = 0x0, len = 140233549750272}}, {n = {ptr = 0x76 <error: Cannot access memory at address 0x76>, len = 0}, v = {ptr = 0x0, len = 140230682214400}}, {n = {ptr = 0x7f8aaaeb1b90 "", len = 18446744073709551615}, v = {ptr = 0x0, len = 140233564264584}}, {n = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 206158430218}, v = {ptr = 0x7f8aaaeb2120 "\225", len = 140233549750368}}, {n = {ptr = 0x7f8aaaeb2050 "PS", len = 147}, v = {ptr = 0x7f8aaaeb21d8 " \211\n\244\212\177", len = 94257505873786}}, {n = {ptr = 0x7f8aabdc3fa0 <_IO_strn_jumps> "", len = 140233564078354}, v = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 140233565356264}}, {n = {ptr = 0x48b3 <error: Cannot access memory at address 0x48b3>, len = 0}, v = {ptr = 0x0, len = 94257352278016}}, {n = {ptr = 0x7f8aaaeb1c30 "\200\206\334\253\212\177", len = 18446744073709551615}, v = {ptr = 0x0, len = 6}}, {n = {ptr = 0x0, len = 10}, v = {ptr = 0x0, len = 94257505870003}}, {n = {ptr = 0x7f8a00000000 <error: Cannot access memory at address 0x7f8a00000000>, len = 4294967292}, v = {ptr = 0x0, len = 140233565550432}}, {n = {ptr = 0xd68 <error: Cannot access memory at address 0xd68>, len = 94257505873765}, v = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 94257505873752}}, {n = {ptr = 0x0, len = 94257505873788}, v = {ptr = 0x0, len = 140233565550432}}, {n = {ptr = 0xd68 <error: Cannot access memory at address 0xd68>, len = 94257505873765}, v = {ptr = 0xb <error: Cannot access memory at address 0xb>, len = 94257505873752}}, {n = {ptr = 0x20 <error: Cannot access memory at address 0x20>, len = 18204466196317010944}, v = {ptr = 0x7f8aaaeb1c20 "\001\200\255\373", len = 140233549749280}}, {n = {ptr = 0x7f8aaaeb1d50 "\002", len = 94257514507723}, v = {ptr = 0xffffffff00000000 <error: Cannot access memory at address 0xffffffff00000000>, len = 206158430232}}, {n = {ptr = 0x7f8aaaeb22c0 "\320\"\353\252\212\177", len = 140233549750768}, v = {ptr = 0xfbad8001 <error: Cannot access memory at address 0xfbad8001>, len = 18204466196317010944}}, {n = {ptr = 0x7f8aabdc8680 <_IO_2_1_stderr_> "\207(\255\373", len = 18204466196317010944}, v = {ptr = 0x7f8aabdc8680 <_IO_2_1_stderr_> "\207(\255\373", len = 140233549750752}}, {n = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 140233549750768}, v = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 140233564077746}}, {n = {ptr = 0x0, len = 18204466196317010944}, v = {ptr = 0x7f8aabdc8680 <_IO_2_1_stderr_> "\207(\255\373", len = 140233549750816}}, {n = {ptr = 0x55ba09b02e00 "\002\001\001\002", len = 140233549750832}, v = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 140233564077746}}, {n = {ptr = 0x3ff <error: Cannot access memory at address 0x3ff>, len = 94257505826200}, v = {ptr = 0x7f8aaaeb1e50 "\001\200\255\373\060", len = 18204466196317010944}}, {n = {ptr = 0x7f8aabdc8680 <_IO_2_1_stderr_> "\207(\255\373", len = 140233549750896}, v = {ptr = 0x7f8aa40a8920 "\"\"M", len = 140233549750912}}, {n = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 18204466196317010944}, v = {ptr = 0x7f8aabdc8680 <_IO_2_1_stderr_> "\207(\255\373", len = 140233549750944}}, {n = {ptr = 0x7f8aa40a8430 "", len = 140233549750960}, v = {ptr = 0x2 <error: Cannot access memory at address 0x2>, len = 140233564077746}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x0, len = 18204466196317010944}}, {n = {ptr = 0x7f8a00000002 <error: Cannot access memory at address 0x7f8a00000002>, len = 94257505870432}, v = {ptr = 0x7f8aaaeb1ec0 "\020", len = 94257514507723}}, {n = {ptr = 0xffffffff00000000 <error: Cannot access memory at address 0xffffffff00000000>, len = 1571951189}, v = {ptr = 0x7f8aabd8e87f "%hu%n:%hu%n:%hu%n", len = 140233564218349}}, {n = {ptr = 0x7f8afbad8001 <error: Cannot access memory at address 0x7f8afbad8001>, len = 94257514507724}, v = {ptr = 0x55ba00000002 <error: Cannot access memory at address 0x55ba00000002>, len = 94257505870104}}, {n = {ptr = 0x55ba00000002 <error: Cannot access memory at address 0x55ba00000002>, len = 94257505870920}, v = {ptr = 0x55ba09ab6dcb "0", len = 94257514507723}}, {n = {ptr = 0x55ba00000002 <error: Cannot access memory at address 0x55ba00000002>, len = 94257505870192}, v = {ptr = 0x0, len = 0}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x3932008000000000 <error: Cannot access memory at address 0x3932008000000000>, len = 18204466196317010944}}, {n = {ptr = 0x7f8aaaeb1e50 "\001\200\255\373\060", len = 140233549749840}, v = {ptr = 0x3ff <error: Cannot access memory at address 0x3ff>, len = 94257505826200}}, {n = {ptr = 0x7f8aaaeb1fc0 "\001\200\255\373\060", len = 140233433697072}, v = {ptr = 0x7f8aa4004b30 "<134>Oct 24 21:06:29 ", len = 140233564243872}}, {n = {ptr = 0x30fbad8001 <error: Cannot access memory at address 0x30fbad8001>, len = 140233433697072}, v = {ptr = 0x7f8aa4004b30 "<134>Oct 24 21:06:29 ", len = 140233433697072}}, {n = {ptr = 0x7f8aa4004b30 "<134>Oct 24 21:06:29 ", len = 140233433697093}, v = {ptr = 0x7f8aa4004f2f "", len = 140233433697072}}, {n = {ptr = 0x7f8aa4004f2f "", len = 0}, v = {ptr = 0x0, len = 0}}, {n = {ptr = 0x0, len = 0}, v = {ptr = 0x8000000010 <error: Cannot access memory at address 0x8000000010>, len = 140233549750176}}, {n = {ptr = 0x7f8aaaeb0000 "", len = 0}, v = {ptr = 0x7f8aa4005350 "<data here>", len = 140233564243872}}, {n = {ptr = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 0}, v = {ptr = 0x7f8aaaeb1fc0 "\001\200\255\373\060", len = 140233549750212}}, {n = {ptr = 0x7f8affffffff <error: Cannot access memory at address 0x7f8affffffff>, len = 140233433699212}, v = {ptr = 0x7f8aa400574f "", len = 140233565552544}}, {n = {ptr = 0x0, len = 18446744073709551615}, v = {ptr = 0x6 <error: Cannot access memory at address 0x6>, len = 60}}, {n = {ptr = 0x100000004 <error: Cannot access memory at address 0x100000004>, len = 94257504901185}, v = {ptr = 0x80aaeb21b0 <error: Cannot access memory at address 0x80aaeb21b0>, len = 94257514731224}}, {n = {ptr = 0x383131ba092d36e8 <error: Cannot access memory at address 0x383131ba092d36e8>, len = 18204466196317010944}, v = {ptr = 0x7f8aaaeb1fc0 "\001\200\255\373\060", len = 140233549750208}}, {n = {ptr = 0x7f8aabd940e8 <fmt> "%u.%u.%u.%u", len = 140233549750512}, v = {ptr = 0x7f8aaaeb22f0 "0\204\n\244\212\177", len = 140233549751072}}, {n = {ptr = 0x7f8aaaeb21d0 "`\245\257\t\272U", len = 140233564218145}, v = {ptr = 0x30fbad8001 <error: Cannot access memory at address 0x30fbad8001>, len = 140233549750736}}, {n = {ptr = 0x7f8aaaeb21d0 "`\245\257\t\272U", len = 140233549750736}, v = {ptr = 0x7f8aaaeb21d0 "`\245\257\t\272U", len = 140233549750750}}, {n = {ptr = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 140233549750736}, v = {ptr = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 0}}, {n = {ptr = 0x3131363831000000 <error: Cannot access memory at address 0x3131363831000000>, len = 18204466196317010944}, v = {ptr = 0x7f8aaaeb2060 "\001\200\255\373\377\377\377\377PS", len = 140233549750368}}, {n = {ptr = 0x3ff <error: Cannot access memory at address 0x3ff>, len = 94257505873752}, v = {ptr = 0x7f8aaaeb21d8 " \211\n\244\212\177", len = 140233549751072}}, {n = {ptr = 0x7f8aa4005350 "<data here>", len = 140233564243872}, v = {ptr = 0xfffffffffbad8001 <error: Cannot access memory at address 0xfffffffffbad8001>, len = 140233433699152}}, {n = {ptr = 0x7f8aa4005350 "<data here>", len = 140233433699152}, v = {ptr = 0x7f8aa4005350 "<data here>", len = 140233433699301}}, {n = {ptr = 0x7f8aa400574f "", len = 140233571185788}, v = {ptr = 0x7f8aa400574f "", len = 18446744073709551615}}, {n = {ptr = 0x6 <error: Cannot access memory at address 0x6>, len = 149}, v = {ptr = 0x100000004 <error: Cannot access memory at address 0x100000004>, len = 94257504901185}}, {n = {ptr = 0x80aaeb2320 <error: Cannot access memory at address 0x80aaeb2320>, len = 94257514810872}, v = {ptr = 0x55ba092d36e8 <global+296> "", len = 140233549753760}}, {n = {ptr = 0x55ba09acf020 "haproxy", len = 140233549763200}, v = {ptr = 0x1ffffffff <error: Cannot access memory at address 0x1ffffffff>, len = 140233433699152}}, {n = {ptr = 0x55ba092d8540 <default_rfc5424_sd_log_format> "- ", len = 3}, v = {ptr = 0x95 <error: Cannot access memory at address 0x95>, len = 2}}, {n = {ptr = 0x5e <error: Cannot access memory at address 0x5e>, len = 140233565552544}, v = {ptr = 0x0, len = 8192}}, {n = {ptr = 0x1030 <error: Cannot access memory at address 0x1030>, len = 140233433677824}, v = {ptr = 0x7700000000 <error: Cannot access memory at address 0x7700000000>, len = 18446744073709547520}}, {n = {ptr = 0x127000 <error: Cannot access memory at address 0x127000>, len = 1204224}, v = {ptr = 0x7 <error: Cannot access memory at address 0x7>, len = 140233433677952}}, {n = {ptr = 0xff <error: Cannot access memory at address 0xff>, len = 18204466196317010944}, v = {ptr = 0x55ba09affc40 "\002\001\001\001\020'", len = 18204466196317010944}}, {n = {ptr = 0x55ba093e5700 <pool_base_start> "pi\005\244\212\177", len = 381}, v = {ptr = 0x1 <error: Cannot access memory at address 0x1>, len = 1}}, {n = {ptr = 0x55ba09afa560 "", len = 140233434368288}, v = {ptr = 0x7f8aa40a83c0 "N\f\020", len = 140233564112180}}, {n = {ptr = 0x3000000030 <error: Cannot access memory at address 0x3000000030>, len = 140233549751072}, v = {ptr = 0x7f8aaaeb2210 "", len = 18204466196317010944}}, {n = {ptr = 0x0, len = 94257514806480}, v = {ptr = 0x5db21250 <error: Cannot access memory at address 0x5db21250>, len = 297838}}, {n = {ptr = 0x55ba09afa270 "<data here>", len = 18204466196317010944}, v = {ptr = 0x7f8aaaeb2250 " \211\n\244\212\177", len = 140233434026256}}, {n = {ptr = 0x7f8aa40a8920 "\"\"M", len = 140233299869104}, v = {ptr = 0x7f8aa403b7b0 "@\374\257\t\272U", len = 18204466196317010944}}, {n = {ptr = 0x7f8aa40a83c0 "N\f\020", len = 94257505697566}, v = {ptr = 0x7f8aa40a8430 "", len = 4194304}}, {n = {ptr = 0x7f8aa40a85e0 "PQ\005\244\212\177", len = 701760002}, v = {ptr = 0x7f8aa4055110 " \245\257\t\272U", len = 140233564112180}}, {n = {ptr = 0x3000000030 <error: Cannot access memory at address 0x3000000030>, len = 140233549751256}, v = {ptr = 0x7f8aaaeb22d0 " ]\004\240\212\177", len = 18204466196317010944}}, {n = {ptr = 0x7f8aa0045d20 "\b:\005\240", len = 140233549750928}, v = {ptr = 0x5db21250 <error: Cannot access memory at address 0x5db21250>, len = 297838}}, {n = {ptr = 0x7f8aa40a8430 "", len = 140233434366912}, v = {ptr = 0x7f8aa403b7b0 "@\374\257\t\272U", len = 18014398509481987}}, {n = {ptr = 0x7f8aa40a86c0 "\b\b", len = 140232561262593}, v = {ptr = 0x0, len = 94257504374823}}}
        copy = 0x0
        msgf = 0
        htx = 0x0
        flen = 37
        hole = <optimized out>
        ret = 0
        outlen = <optimized out>
        wrap = 16336
        try = 0
#3  0x000055ba09111d41 in h2c_bck_handle_headers (h2s=<optimized out>, h2c=0x7f8aa403bdb0) at src/mux_h2.c:2145
        rxbuf = {size = 0, area = 0x0, data = 0, head = 0}
        body_len = 0
        flags = 0
        error = <optimized out>
        rxbuf = <optimized out>
        body_len = <optimized out>
        flags = <optimized out>
        error = <optimized out>
#4  h2_process_demux (h2c=0x7f8aa403bdb0) at src/mux_h2.c:2617
        ret = 0
        h2s = <optimized out>
        tmp_h2s = <optimized out>
        padlen = <optimized out>
        old_iw = <optimized out>
        hdr = {len = 37, sid = 14033, ft = 1 '\001', ff = 4 '\004'}
        h2s = <optimized out>
        tmp_h2s = <optimized out>
        hdr = <optimized out>
        padlen = <optimized out>
        old_iw = <optimized out>
        ret = <optimized out>
#5  h2_process (h2c=h2c@entry=0x7f8aa403bdb0) at src/mux_h2.c:3007
        conn = 0x7f8aa40a4200
#6  0x000055ba09112d9f in h2_io_cb (t=<optimized out>, ctx=0x7f8aa403bdb0, status=<optimized out>) at src/mux_h2.c:2994
        h2c = 0x7f8aa403bdb0
        ret = <optimized out>
#7  0x000055ba091d9251 in process_runnable_tasks () at src/task.c:414
        t = <optimized out>
        state = <optimized out>
        ctx = <optimized out>
        process = <optimized out>
        lrq = <optimized out>
        grq = <optimized out>
        t = <optimized out>
        max_processed = 200
#8  0x000055ba091448d4 in run_poll_loop () at src/haproxy.c:2516
        next = <optimized out>
        wake = <optimized out>
        next = <optimized out>
        wake = <optimized out>
#9  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2637
        ptaf = <optimized out>
        ptif = <optimized out>
        ptdf = <optimized out>
        ptff = <optimized out>
        init_left = 0
        init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}
        init_cond = {__data = {{__wseq = 13, __wseq32 = {__low = 13, __high = 0}}, {__g1_start = 7, __g1_start32 = {__low = 7, __high = 0}}, __g_refs = {0, 0}, __g_size = {0, 0}, __g1_orig_size = 12, __wrefs = 0, __g_signals = {0, 0}}, __size = "\r\000\000\000\000\000\000\000\a", '\000' <repeats 23 times>, "\f", '\000' <repeats 14 times>, __align = 13}
#10 0x00007f8aac318fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140233549895424, 9062399038263037266, 140725191769182, 140725191769183, 140233549895424, 140725191769808, -9020342544352768686, -9020354317620572846}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#11 0x00007f8aabd054cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
@wtarreau

This comment has been minimized.

Copy link

@wtarreau wtarreau commented Nov 22, 2019

That's odd. It looks like we're trying to allocate a buffer for a constant stream (which must be the closed one then). It does not make much sense to me. Or maybe this could happen when aborting a request before the server responds, but it should not be possible to turn the stream to closed before the server responds, so it still looks particularly odd. I'm interested in any reproducer if there's an easy way to trigger this.

@capflam

This comment has been minimized.

Copy link
Contributor

@capflam capflam commented Nov 26, 2019

This one should be fixed in 2.1, 2.0 and 1.9. For the record, the commit is 57a1816. Could you confirm please ?

@phaze-debug

This comment has been minimized.

Copy link
Author

@phaze-debug phaze-debug commented Nov 26, 2019

Thanks @capflam - Tested with 2.0.10 and didn't notice any segfault after running the test multiple times for extended duration. The issue seems to be fixed!

@capflam

This comment has been minimized.

Copy link
Contributor

@capflam capflam commented Nov 27, 2019

@phaze-debug, Thanks for the feedback !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.