Segfault with http2 with bad connection

[ThomasObenaus]

I did this

Application that communicates using http2 (nghttp2) with our backend.
In case the connection is bad (mobile phone with edge/gsm) the application crashes each time with a segfault and the same back-trace.

Precondition

• Bad connection (low bandwidth) with over mobile phone whose datavolume is empty (fallback to 2G).
• No connection to the internet (firewall closed).
• Application starts requesting (to reproduce the problem) a lot of requests.
• dnsmask already complains about too many requests in parallel (max of 150 reached).
• Opening the fire wall.
• Waiting for ~1 min --> segfault.

Curl errors

• When FW is closed: "Cloud not connect"
• When FW is open: "Timeout reached"

I expected the following

no segfault

curl/libcurl version

Tested with 7.54.0 and 7.60

Libs

nghttp2 1.17.0
c-ares 1.13.9

[ThomasObenaus]

Backtrace

(gdb) bt
#0 0x0000007f935e0988 in __GI__IO_fwrite (buf=..., size=..., count=..., fp=...) at iofwrite.c:37
#1 0x0000007f956708a8 in showit (data=..., type=..., ptr=..., size=...) at sendf.c:826
#2 0x0000007f95670c18 in Curl_debug (data=..., type=..., ptr=..., size=..., conn=...) at sendf.c:878
#3 0x0000007f95670d50 in Curl_infof (data=..., fmt=...) at sendf.c:234
#4 0x0000007f956a1f9c in on_frame_not_send (session=..., frame=..., lib_error_code=..., userp=...) at http2.c:722
#5 0x0000007f95636380 in nghttp2_session_mem_send_internal (session=..., data_ptr=..., fast_cb=...)
at /home/thomaso/work/projects/bob_nav/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:2914
#6 0x0000007f95636c04 in nghttp2_session_send (session=...)
at /home/thomaso/work/projects/bob_nav/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:3215
#7 0x0000007f956a29ec in h2_session_send (data=..., h2=...) at http2.c:1379
#8 0x0000007f956a405c in http2_recv (conn=..., sockindex=..., mem=..., len=..., err=...) at http2.c:1559
#9 0x0000007f95670a64 in Curl_read (conn=..., sockfd=..., buf=..., sizerequested=..., n=...) at sendf.c:747
#10 0x0000007f95684764 in readwrite_data (comeback=..., done=..., didwhat=..., k=..., conn=..., data=...)
at transfer.c:434
#11 Curl_readwrite (conn=..., data=..., done=..., comeback=...) at transfer.c:1102
#12 0x0000007f9568ebac in multi_runsingle (multi=..., now=..., data=...) at multi.c:1908
#13 0x0000007f9568f5a0 in curl_multi_perform (multi=..., running_handles=...) at multi.c:2180
#14 0x00000000017ffeec in tsd::nav::cloud::comm::client::HTTPRequestHandler::main() ()
#15 0x00000000027f5ffc in tsd::nav::cloud::comm::client::HTTPRequestHandler::run() ()
#16 0x0000007f939fc510 in tsd::common::system::Thread::startImpl (arg=...)
at /home/thomaso/work/projects/bob_nav/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/Thread.cpp:310
#17 0x0000007f93a00040 in tsd::common::system::os::thread::pseudoStartThread (arg=...)
at /home/thomaso/work/projects/bob_nav/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/posix/POSIXThread.cpp:367
#18 0x0000007f93955030 in start_thread (arg=...) at pthread_create.c:331
#19 0x0000007f936464c0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89

[bagder]

That's a 7.54.0 backtrace, right? The line numbers seems way off from 7.60.0... Did you build with DEBUG_HTTP2 defined?

So what is the problem in the showit() function? The blanked out arguments make it hard to guess.

[ThomasObenaus]

Current state of investigation

• The guess is that after we got back the easy_handle from curl_multi_info_read processed it and then freed the easy_handle with curl_multi_remove_handle + curl_easy_cleanup there still seams to come traffic for this easy_handle.

[ThomasObenaus]

Yes that's the bt from 7.54.0.
In showit() it seams that the easy_handle (data) is damaged.
The magic field shows 0x877b330 instead of 0xc0dedbad.

It chrashes then at fwrite(s_infotype[type], 2, 1, data->set.err); (in: sendf.c) .

[ThomasObenaus]

• Even at level of Curl_readwrite (frame 11 in the bt) the connection having a pointer to a stream-map referencing another easy_handle (0x7f08776438) than the easy_handle (0x7f08c62258) that owns this connection.
• The first easy_handle is invalid (magic should be 0xc0dedbad).
  (gdb) p data->magic
  $6 = 0x877b330
• It seams that after curl_easy_cleanup was called on the easy_handle (0x7f08776438) it is still part of the nghttp2-stream. Thus when the stream is processed in the next loop it crashes.
• The question here is: Who adds the "wrong" easy_handle (0x7f08776438) as stream_user_data to the "good" easy_handle (0x7f08c62258)

[bagder]

Since I can't reproduce I can only offer general guidance as how to go ahead and debug this:

• build with git master, we've already fixed a few problems with removed easy handles from the multi handle
• define DEBUG_HTTP2 and check if you actually get frames from a stream that is already done/removed when you see this problem - which I think it might be
• if you get frames from a stream that was owned by an easy handle that is now removed, can you figure out why Curl_http2_done was not called on it (and subsequently nghttp2_session_set_stream_user_data cleared the association from frame id to easy handle), or whatever it is that makes libcurl extract a "dead" easy handle for an incoming h2 frame

[ThomasObenaus]

Thanks for the quick feedback.
I'll rebuild it with your hints and come back as soon as possible with a status update.

[ThomasObenaus]

Update

I tried it with 7.60.0 and get now the same bt like mentioned in #2674 for two times and another one at the second try.

BT 1

#0  h2_session_send (data=..., h2=...) at http2.c:1470
#1  0x0000007fbb9564e8 in h2_process_pending_input (conn=..., httpc=..., err=...) at http2.c:1303
#2  0x0000007fbb95667c in http2_connisdead (conn=...) at http2.c:219
#3  http2_conncheck (check=..., checks_to_perform=...) at http2.c:236
#4  0x0000007fbb92c760 in extract_if_dead (conn=..., data=...) at url.c:973
#5  0x0000007fbb92c7ec in call_extract_if_dead (conn=..., param=...) at url.c:1003
#6  0x0000007fbb954914 in Curl_conncache_foreach (data=..., connc=..., param=..., func=...) at conncache.c:382
#7  0x0000007fbb9303d8 in prune_dead_connections (data=...) at url.c:1025
#8  create_conn (async=..., in_connect=..., data=...) at url.c:4382
#9  Curl_connect (data=..., in_connect=..., asyncp=..., protocol_done=...) at url.c:4660
#10 0x0000007fbb940db0 in multi_runsingle (multi=..., now=..., data=...) at multi.c:1421
#11 0x0000007fbb941b0c in curl_multi_perform (multi=..., running_handles=...) at multi.c:2159
#12 0x00000000017ffe10 in tsd::nav::cloud::comm::client::HTTPRequestHandler::main() ()
#13 0x00000000027f500c in tsd::nav::cloud::comm::client::HTTPRequestHandler::run() ()
#14 0x0000007fb9cb0510 in tsd::common::system::Thread::startImpl (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/Thread.cpp:310
#15 0x0000007fb9cb4040 in tsd::common::system::os::thread::pseudoStartThread (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/posix/POSIXThread.cpp:367
#16 0x0000007fb9c09030 in start_thread (arg=...) at pthread_create.c:331
#17 0x0000007fb98fa4c0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89

(gdb) p data->magic
$7 = 0x0

BT 2

#0  0x0000007f88db0504 in Curl_multi_connchanged (multi=...) at multi.c:1111
#1  0x0000007f88dc79cc in on_frame_recv (session=..., frame=..., userp=...) at http2.c:586
#2  0x0000007f88d5b620 in session_call_on_frame_received (frame=..., session=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:3271
#3  nghttp2_session_on_settings_received (session=..., frame=..., noack=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:4367
#4  0x0000007f88d5e7a8 in session_process_settings_frame (session=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:4507
#5  nghttp2_session_mem_recv (session=..., in=..., inlen=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:6161
#6  0x0000007f88dc74c0 in h2_process_pending_input (conn=..., httpc=..., err=...) at http2.c:1279
#7  0x0000007f88dc767c in http2_connisdead (conn=...) at http2.c:219
#8  http2_conncheck (check=..., checks_to_perform=...) at http2.c:236
#9  0x0000007f88d9d760 in extract_if_dead (conn=..., data=...) at url.c:973
#10 0x0000007f88d9d7ec in call_extract_if_dead (conn=..., param=...) at url.c:1003
#11 0x0000007f88dc5914 in Curl_conncache_foreach (data=..., connc=..., param=..., func=...) at conncache.c:382
#12 0x0000007f88da13d8 in prune_dead_connections (data=...) at url.c:1025
#13 create_conn (async=..., in_connect=..., data=...) at url.c:4382
#14 Curl_connect (data=..., in_connect=..., asyncp=..., protocol_done=...) at url.c:4660
#15 0x0000007f88db1db0 in multi_runsingle (multi=..., now=..., data=...) at multi.c:1421
#16 0x0000007f88db2b0c in curl_multi_perform (multi=..., running_handles=...) at multi.c:2159
#17 0x00000000017ffe10 in tsd::nav::cloud::comm::client::HTTPRequestHandler::main() ()
#18 0x00000000027f500c in tsd::nav::cloud::comm::client::HTTPRequestHandler::run() ()
#19 0x0000007f87121510 in tsd::common::system::Thread::startImpl (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/Thread.cpp:310
#20 0x0000007f87125040 in tsd::common::system::os::thread::pseudoStartThread (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/posix/POSIXThread.cpp:367
#21 0x0000007f8707a030 in start_thread (arg=...) at pthread_create.c:331
#22 0x0000007f86d6b4c0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89

(gdb) p multi
$1 = (struct Curl_multi *) 0x

[ThomasObenaus]

Next step: use master from libcurl

[ThomasObenaus]

Still a segfault, even with master from lib-curl:

#0  on_stream_close (session=..., stream_id=..., error_code=..., userp=...) at http2.c:842
#1  0x0000007f8a279c38 in nghttp2_session_close_stream (session=..., stream_id=..., error_code=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:1162
#2  0x0000007f8a27d3ac in nghttp2_session_mem_send_internal (session=..., data_ptr=..., fast_cb=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:2945
#3  0x0000007f8a27dc04 in nghttp2_session_send (session=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_session.c:3215
#4  0x0000007f8a2e81ec in h2_session_send (data=..., h2=...) at http2.c:1476
#5  0x0000007f8a2e9924 in http2_recv (conn=..., sockindex=..., mem=..., len=..., err=...) at http2.c:1656
#6  0x0000007f8a2b82e0 in Curl_read (conn=..., sockfd=..., buf=..., sizerequested=..., n=...) at sendf.c:749
#7  0x0000007f8a2c8db0 in readwrite_data (comeback=..., done=..., didwhat=..., k=..., conn=..., data=...)
    at transfer.c:482
#8  Curl_readwrite (conn=..., data=..., done=..., comeback=...) at transfer.c:1125
#9  0x0000007f8a2d3144 in multi_runsingle (multi=..., now=..., data=...) at multi.c:1914
#10 0x0000007f8a2d3b8c in curl_multi_perform (multi=..., running_handles=...) at multi.c:2181
#11 0x00000000017e6ff0 in tsd::nav::cloud::comm::client::HTTPRequestHandler::main() ()
#12 0x00000000027a3ecc in tsd::nav::cloud::comm::client::HTTPRequestHandler::run() ()
#13 0x0000007f88642510 in tsd::common::system::Thread::startImpl (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/Thread.cpp:310
#14 0x0000007f88646040 in tsd::common::system::os::thread::pseudoStartThread (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/posix/POSIXThread.cpp:367
#15 0x0000007f8859c030 in start_thread (arg=...) at pthread_create.c:331
#16 0x0000007f8828d4c0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89

Now the crash: stream->closed = TRUE;

(gdb) p session
$2 = (nghttp2_session *) 0x7f143c79e0
(gdb) p *session
$3 = {
  streams = {
    table = 0x7f14db83f0, 
    mem = 0x7f143c8260, 
    size = 0xb, 
    tablelen = 0x100
  }, 
  root = {
    map_entry = {
      next = 0x0, 
      key = 0x0
    }, 
    pq_entry = {
      index = 0x0
    }, 
    obq = {
      q = 0x7f1458a590, 
      mem = 0x7f143c8260, 
      length = 0x0, 
      capacity = 0x4, 
      less = 0x7f8a277380 <stream_less at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_stream.c:52>
    }, 
    content_length = 0xffffffffffffffff, 
    recv_content_length = 0x0, 
    descendant_last_cycle = 0x0, 
    cycle = 0x0, 
    descendant_next_seq = 0x5b, 
    seq = 0x0, 
    dep_prev = 0x0, 
    dep_next = 0x7f14afac20, 
    sib_prev = 0x0, 
    sib_next = 0x0, 
    closed_prev = 0x0, 
    closed_next = 0x0, 
    stream_user_data = 0x0, 
    item = 0x0, 
    last_writelen = 0x20, 
    stream_id = 0x0, 
    remote_window_size = 0x0, 
    recv_window_size = 0x0, 
    consumed_size = 0x0, 
    recv_reduction = 0x0, 
    local_window_size = 0x0, 
    weight = 0x10, 
    pending_penalty = 0x0, 
    sum_dep_weight = 0xb0, 
    state = NGHTTP2_STREAM_IDLE, 
    status_code = 0xffff, 
    http_flags = 0x0, 
    flags = 0x0, 
    shut_flags = 0x0, 
    queued = 0x0, 
    window_update_queued = 0x0
  }, 
  ob_urgent = {
    head = 0x0, 
    tail = 0x0, 
    n = 0x0
  }, 
  ob_reg = {
    head = 0x0, 
    tail = 0x0, 
    n = 0x0
  }, 
  ob_syn = {
    head = 0x7f14b37f30, 
    tail = 0x7f143a4820, 
    n = 0x99
  }, 
  aob = {
    item = 0x0, 
    framebufs = {
      head = 0x7f14b52ba0, 
      cur = 0x7f14b52ba0, 
      mem = 0x7f143c8260, 
      chunk_length = 0x400a, 
      max_chunk = 0x4, 
      chunk_used = 0x1, 
      chunk_keep = 0x1, 
      offset = 0xa
    }, 
    state = NGHTTP2_OB_POP_ITEM
  }, 
  iframe = {
    frame = {
      hd = {
        length = 0x0, 
        stream_id = 0x0, 
        type = 0x0, 
        flags = 0x0, 
        reserved = 0x0
      }, 
      data = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        padlen = 0x0
      }, 
      headers = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        padlen = 0x0, 
        pri_spec = {
          stream_id = 0x0, 
          weight = 0x0, 
          exclusive = 0x0
        }, 
        nva = 0x0, 
        nvlen = 0x0, 
        cat = NGHTTP2_HCAT_REQUEST
      }, 
      priority = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        pri_spec = {
          stream_id = 0x0, 
          weight = 0x0, 
          exclusive = 0x0
        }
      }, 
      rst_stream = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        error_code = 0x0
      }, 
      settings = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        niv = 0x0, 
        iv = 0x0
      }, 
      push_promise = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        padlen = 0x0, 
        nva = 0x0, 
        nvlen = 0x0, 
        promised_stream_id = 0x0, 
        reserved = 0x0
      }, 
      ping = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        opaque_data = "\000\000\000\000\000\000\000"
      }, 
      goaway = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        last_stream_id = 0x0, 
        error_code = 0x0, 
        opaque_data = 0x0, 
        opaque_data_len = 0x0, 
        reserved = 0x0
      }, 
      window_update = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        window_size_increment = 0x0, 
        reserved = 0x0
      }, 
      ext = {
        hd = {
          length = 0x0, 
          stream_id = 0x0, 
          type = 0x0, 
          flags = 0x0, 
          reserved = 0x0
        }, 
        payload = 0x0
      }
    }, 
    ext_frame_payload = {
      altsvc = {
        origin = 0x0, 
        origin_len = 0x0, 
        field_value = 0x0, 
        field_value_len = 0x0
      }
    }, 
    iv = 0x0, 
    sbuf = {
      begin = 0x7f143c7c5c "", 
      end = 0x7f143c7c65 "", 
      pos = 0x7f143c7c5c "", 
      last = 0x7f143c7c5c "", 
      mark = 0x7f143c7c65 ""
    }, 
    lbuf = {
      begin = 0x0, 
      end = 0x0, 
      pos = 0x0, 
      last = 0x0, 
      mark = 0x0
    }, 
    raw_lbuf = 0x0, 
    niv = 0x0, 
    max_niv = 0x0, 
    payloadleft = 0x0, 
    padlen = 0x0, 
    state = NGHTTP2_IB_READ_HEAD, 
    raw_sbuf = "\000\000\000\000\001\000\000\000U"
  }, 
  hd_deflater = {
    ctx = {
      hd_table = {
        buffer = 0x7f14db7bd0, 
        mask = 0x7f, 
        first = 0xfffffffffffffffc, 
        len = 0x4
      }, 
      mem = 0x7f143c8260, 
      hd_table_bufsize = 0xf1, 
      hd_table_bufsize_max = 0x1000, 
      next_seq = 0x4, 
      bad = 0x0
    }, 
    map = {
      table = {[0x0] = 0x0 <repeats 21 times>, [0x15] = 0x7f15593e80, [0x16] = 0x0, [0x17] = 0x0, [0x18] = 0x0, 
        [0x19] = 0x7f1491ed00, [0x1a] = 0x0 <repeats 15 times>, [0x29] = 0x7f15614570, 
        [0x2a] = 0x0 <repeats 52 times>, [0x5e] = 0x7f15601800, [0x5f] = 0x0 <repeats 33 times>}
    }, 
    deflate_hd_table_bufsize_max = 0x1000, 
    min_hd_table_bufsize_max = 0xffffffff, 
    notify_table_size_change = 0x0
  }, 
  hd_inflater = {
    ctx = {
      hd_table = {
        buffer = 0x7f14db7fe0, 
        mask = 0x7f, 
        first = 0xffffffffffffff82, 
        len = 0x3f
      }, 
      mem = 0x7f143c8260, 
      hd_table_bufsize = 0xfd9, 
      hd_table_bufsize_max = 0x1000, 
      next_seq = 0x7e, 
      bad = 0x0
    }, 
    huff_decode_ctx = {
      state = 0x38, 
      accept = 0x1
    }, 
    namebuf = {
      begin = 0x7f143cbab8 "x-application-context", 
      end = 0x7f143cbad7 "", 
      pos = 0x7f143cbab8 "x-application-context", 
      last = 0x7f143cbacd "", 
      mark = 0x7f143cbab8 "x-application-context"
    }, 
    valuebuf = {
      begin = 0x7f14aefaf8 "application:60124", 
      end = 0x7f14aefb11 "", 
      pos = 0x7f14aefaf8 "application:60124", 
      last = 0x7f14aefb09 "", 
      mark = 0x7f14aefaf8 "application:60124"
    }, 
    namercbuf = 0x0, 
    valuercbuf = 0x0, 
    nv_name_keep = 0x0, 
    nv_value_keep = 0x0, 
    left = 0x0, 
    index = 0x1b, 
    settings_hd_table_bufsize_max = 0x1000, 
    min_hd_table_bufsize_max = 0xffffffff, 
    shift = 0x0, 
    opcode = NGHTTP2_HD_OPCODE_NEWNAME, 
    state = NGHTTP2_HD_STATE_INFLATE_START, 
    huffman_encoded = 0x1, 
    index_required = 0x0, 
    no_index = 0x0
  }, 
  callbacks = {
    send_callback = 0x7f8a2e8320 <send_callback at http2.c:362>, 
    recv_callback = 0x0, 
    on_frame_recv_callback = 0x7f8a2e87f0 <on_frame_recv at http2.c:557>, 
    on_invalid_frame_recv_callback = 0x7f8a2e86f0 <on_invalid_frame_recv>, 
    on_data_chunk_recv_callback = 0x7f8a2e7ef0 <on_data_chunk_recv at http2.c:708>, 
    before_frame_send_callback = 0x7f8a2e8710 <before_frame_send>, 
    on_frame_send_callback = 0x7f8a2e7af0 <on_frame_send at http2.c:788>, 
    on_frame_not_send_callback = 0x7f8a2e7ad0 <on_frame_not_send at http2.c:802>, 
    on_stream_close_callback = 0x7f8a2e83a0 <on_stream_close at http2.c:826>, 
    on_begin_headers_callback = 0x7f8a2e7e80 <on_begin_headers at http2.c:857>, 
    on_header_callback = 0x7f8a2e7b10 <on_header at http2.c:920>, 
    on_header_callback2 = 0x0, 
    on_invalid_header_callback = 0x0, 
    on_invalid_header_callback2 = 0x0, 
    select_padding_callback = 0x0, 
    read_length_callback = 0x0, 
    on_begin_frame_callback = 0x0, 
    send_data_callback = 0x0, 
    pack_extension_callback = 0x0, 
    unpack_extension_callback = 0x0, 
    on_extension_chunk_recv_callback = 0x0, 
    error_callback = 0x7f8a2e7aa0 <error_callback at http2.c:1081>
  }, 
  mem = {
    mem_user_data = 0x0, 
    malloc = 0x7f8a2851f0 <default_malloc at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_mem.c:28>, 
    free = 0x7f8a2851e0 <default_free at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_mem.c:31>, 
    calloc = 0x7f8a2851d0 <default_calloc at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_mem.c:35>, 
    realloc = 0x7f8a2851c0 <default_realloc at /home/thomaso/work/projects/bob_c_ares/dev/src/libs/ext-nghttp2/1/workspace/nghttp2-1.17.0/lib/nghttp2_mem.c:39>
  }, 
  last_cycle = 0x0, 
  user_data = 0x7f14ae7be8, 
  closed_stream_head = 0x0, 
  closed_stream_tail = 0x0, 
  idle_stream_head = 0x0, 
  idle_stream_tail = 0x0, 
  inflight_settings_head = 0x0, 
  num_outgoing_streams = 0xb, 
  num_incoming_streams = 0x0, 
  num_incoming_reserved_streams = 0x0, 
  max_incoming_reserved_streams = 0xc8, 
  num_closed_streams = 0x0, 
  num_idle_streams = 0x0, 
  nvbuflen = 0x0, 
  obq_flood_counter_ = 0x0, 
  max_send_header_block_length = 0x10000, 
  next_stream_id = 0x223, 
  last_sent_stream_id = 0xd1, 
  last_recv_stream_id = 0x0, 
  last_proc_stream_id = 0x0, 
  next_unique_id = 0x0, 
  local_last_stream_id = 0x7fffffff, 
  remote_last_stream_id = 0x7fffffff, 
  remote_window_size = 0x7ffffddc, 
  recv_window_size = 0x2251e, 
  consumed_size = 0x0, 
  recv_reduction = 0x0, 
  local_window_size = 0x40000000, 
  remote_settings = {
    header_table_size = 0x1000, 
    enable_push = 0x1, 
    max_concurrent_streams = 0x80, 
    initial_window_size = 0x10000, 
    max_frame_size = 0xffffff, 
    max_header_list_size = 0xffffffff
  }, 
  local_settings = {
    header_table_size = 0x1000, 
    enable_push = 0x1, 
    max_concurrent_streams = 0x64, 
    initial_window_size = 0x40000000, 
    max_frame_size = 0x4000, 
    max_header_list_size = 0xffffffff
  }, 
  opt_flags = 0x0, 
  pending_local_max_concurrent_stream = 0x64, 
  builtin_recv_ext_types = 0x0, 
  pending_enable_push = 0x1, 
  server = 0x0, 
  goaway_flags = 0x0, 
  window_update_queued = 0x0, 
  user_recv_ext_types = '\000' <repeats 31 times>
}

[ThomasObenaus]

@bagder: It looks that I can reproduce it with my setup quite good.
Do you have any more ideas, maybe code locations which I should instrument to track down the issue?

[bagder]

That's an incoming stream close. It looks like stream is perhaps pointing to already freed memory there? If that's so, can you figure out why/how it was freed already? It looks like the free in http2_disconnect() is a likely suspect but still seems weird... Can you run your code with valgrind? If so, does it help to point out any problem perhaps earlier?

If you have http2 debugging enabled you can see what stream id that ends there. Can you see anything wrong with that? Is there other streams going on at the same time?

p *session

That's the libnghttp2 handle, I don't think that helps us much because I think the problem is on the curl side.

[ThomasObenaus]

With HTTP2_DEBUG enabled

Now I get an assert (only one time) at DEBUGASSERT(httpc->drain_total >= data->state.drain);

(gdb) p httpc->drain_total
$1 = 0x0
(gdb) p data->state.drain
$2 = 0x1
(gdb)

#0  0x0000007faa6851e8 in __GI_raise (sig=...) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x0000007faa6865dc in __GI_abort () at abort.c:89
#2  0x0000007faa67e94c in __assert_fail_base (fmt=..., assertion=..., file=..., line=..., function=...) at assert.c:92
#3  0x0000007faa67e9e4 in __GI___assert_fail (assertion=..., file=..., line=..., function=...) at assert.c:101
#4  0x0000007fac775194 in http2_handle_stream_close (conn=..., data=..., stream=..., err=...) at http2.c:1370
#5  0x0000007fac775c68 in http2_recv (conn=..., sockindex=..., mem=..., len=..., err=...) at http2.c:1693
#6  0x0000007fac7442e0 in Curl_read (conn=..., sockfd=..., buf=..., sizerequested=..., n=...) at sendf.c:749
#7  0x0000007fac754db0 in readwrite_data (comeback=..., done=..., didwhat=..., k=..., conn=..., data=...)
    at transfer.c:482
#8  Curl_readwrite (conn=..., data=..., done=..., comeback=...) at transfer.c:1125
#9  0x0000007fac75f144 in multi_runsingle (multi=..., now=..., data=...) at multi.c:1914
#10 0x0000007fac75fb8c in curl_multi_perform (multi=..., running_handles=...) at multi.c:2181
#11 0x00000000017e6ff0 in tsd::nav::cloud::comm::client::HTTPRequestHandler::main() ()
#12 0x00000000027a3ecc in tsd::nav::cloud::comm::client::HTTPRequestHandler::run() ()
#13 0x0000007faaace510 in tsd::common::system::Thread::startImpl (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/Thread.cpp:310
#14 0x0000007faaad2040 in tsd::common::system::os::thread::pseudoStartThread (arg=...)
    at /home/thomaso/work/projects/bob_c_ares/dev/src/system/tsd-common/1/workspace/tsd.common/src/tsd/common/system/posix/POSIXThread.cpp:367
#15 0x0000007faaa28030 in start_thread (arg=...) at pthread_create.c:331
#16 0x0000007faa7194c0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89

Last Log-Lines

16:50:35.040    [Debug]  [0x7f352d6a88] cURL [TEXT] Using Stream ID: e5 (easy handle 0x7f352d6a88)
16:50:35.040     [Warn]  [0x7f352d6a88] cURL [TEXT] h2 header: Authorization:Bearer bla
16:50:35.040    [Debug]  [0x7f352d6a88] cURL [TEXT] STATE: DO => DO_DONE handle 0x7f352d6a88; line 1695 (connection #3892)
16:50:35.040     [Warn]  [0x7f352d6a88] cURL [HEADER_OUT] GET /api/traffic/v2/tile/134766960/current HTTP/2
16:50:35.040    [Debug]  [0x7f352d6a88] cURL [TEXT] STATE: DO_DONE => WAITPERFORM handle 0x7f352d6a88; line 1822 (connection #3892)
16:50:35.044    [Debug]  [0x7f352d6a88] cURL [TEXT] STATE: WAITPERFORM => PERFORM handle 0x7f352d6a88; line 1837 (connection #3892)
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] STATE: INIT => CONNECT handle 0x7f34657d88; line 1429 (connection #-5000)
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] Found bundle for host blubb: 0x7f350ec088 [can multiplex]
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] MAX_CONCURRENT_STREAMS reached, skip (128)
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] Conn: 3892 (0x7f34af6ca8) Receive pipe weight: (-1/0), penalized: FALSE
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] Multiplexed connection found!
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] Found connection 3892, with requests in the pipe (69)
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] Re-using existing connection! (#3892) with host blubb
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] STATE: CONNECT => DO handle 0x7f34657d88; line 1474 (connection #3892)
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] http2_send len=1074
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: :method:GET
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: :path:/api/traffic/v2/tile/134766962/current
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: :scheme:https
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: :authority:blubb
16:50:35.044    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: Accept-Encoding:deflate, gzip
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: accept:application/x-protobuf
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] h2 header: content-type:application/x-protobuf
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] Using Stream ID: e7 (easy handle 0x7f34657d88)
16:50:35.045     [Warn]  [0x7f34657d88] cURL [TEXT] h2 header: Authorization:Bearer bla
16:50:35.045     [Warn]  [0x7f34657d88] cURL [HEADER_OUT] GET /api/traffic/v2/tile/134766962/current HTTP/2
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] STATE: DO => DO_DONE handle 0x7f34657d88; line 1695 (connection #3892)
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] STATE: DO_DONE => WAITPERFORM handle 0x7f34657d88; line 1822 (connection #3892)
16:50:35.045    [Debug]  [0x7f34657d88] cURL [TEXT] STATE: WAITPERFORM => PERFORM handle 0x7f34657d88; line 1837 (connection #3892)
16:50:35.046    [Debug]  [0x7f34c84ab8] cURL [TEXT] Expire cleared
16:50:35.048    [Debug]  [0x7f34c69578] cURL [TEXT] Curl_readwrite: forcibly told to drain data
16:50:35.048    [Debug]  [0x7f34c69578] cURL [TEXT] http2_recv: easy 0x7f34c69578 (stream 411)