Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assert error in HTC_RxStuff(), cache/cache_session.c line 294 #2285

Closed
simonvik opened this issue Mar 28, 2017 · 1 comment
Closed

Assert error in HTC_RxStuff(), cache/cache_session.c line 294 #2285

simonvik opened this issue Mar 28, 2017 · 1 comment
Labels
b=bug c=H/2 c=varnishd

Comments

@simonvik
Copy link
Contributor 

Panic at: Tue, 28 Mar 2017 09:53:14 GMT
Assert error in HTC_RxStuff(), cache/cache_session.c line 294:
  Condition(i >= 0) not true.
version = varnish-5.1.1 revision de38712, vrt api = 6.0
ident = Linux,4.4.30-20161130-1423-0a2a05d,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll
now = 5510903.711436 (mono), 1490694793.346442 (real)
Backtrace:
  0x438975: /usr/sbin/varnishd() [0x438975]
  0x440c92: /usr/sbin/varnishd(HTC_RxStuff+0x352) [0x440c92]
  0x461a71: /usr/sbin/varnishd(h2_rxframe+0x81) [0x461a71]
  0x4624b5: /usr/sbin/varnishd() [0x4624b5]
  0x451de2: /usr/sbin/varnishd() [0x451de2]
  0x45228b: /usr/sbin/varnishd() [0x45228b]
  0x7f7d5a1316ba: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f7d5a1316ba]
  0x7f7d59e6782d: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f7d59e6782d]
thread = (cache-worker)
thr.req = 0x7f7d38817020 {
  vxid = 425985, transport = H2
  step = 0x0,
  req_body = R_BODY_INIT,
  err_code = 1, err_reason = (null),
  restarts = 0, esi_level = 0,
  sp = 0x7f7d3bc0e220 {
    fd = 21, vxid = 425985,
    t_open = 1490694793.208348,
    t_idle = 1490694793.346433,
    transport = H2 {
      streams {
        0x00000000 idle  
        0x00000001 c-rem 
        0x00000003 c-rem 
      }
    }
    client = ********* 64787,
    privs = 0x7f7d3bc0e288 {
    },
  },
  ws = 0x7f7d38817208 {
    id = \"req\",
    {s, f, r, e} = {0x7f7d38819008, +16648, +122864, +122864},
  },
  http_conn = 0x7f7d38817130 {
    fd = 21 (@0x7f7d3bc0e238),
    doclose = NULL,
    ws = 0x7f7d38817208 {
      [Already dumped, see above]
    },
    {rxbuf_b, rxbuf_e} = {0x7f7d3881d110, 0x7f7d38820ed3},
    {pipeline_b, pipeline_e} = {(nil), (nil)},
    content_length = 0,
    body_status = none,
    first_byte_timeout = 0.000000,
    between_bytes_timeout = 0.000000,
  },
  http[req] = 0x7f7d388172a0 {
    ws = (nil) {
    },
    hdrs {
    },
  },
  vmods = {
    std = {Varnish 5.1.1 de38712, 6.0},
    one = {Varnish 5.1.1 de38712, 6.0},
    maxminddb = {Varnish 5.1.1 de38712, 6.0},
    directors = {Varnish 5.1.1 de38712, 6.0},
    xkey = {Varnish 5.1.1 de38712, 6.0},
    header = {Varnish 5.1.1 de38712, 6.0},
    cookie = {Varnish 5.1.1 de38712, 6.0},
    vsthrottle = {Varnish 5.1.1 de38712, 6.0},
    tcp = {Varnish 5.1.1 de38712, 6.0},
    bodyaccess = {Varnish 5.1.1 de38712, 6.0},
  },
  flags = {
  },
},
thr.busyobj = (nil) {
},

Running on ubuntu 16.04
Varnish 5.1.1 patched to 15f5c12
@simonvik
Copy link
Contributor Author

Still see this, running varnish patched to 3db481a

@bsdphk bsdphk closed this as completed in 20adece Apr 3, 2017
mbgrydeland added a commit to mbgrydeland/varnish-cache that referenced this issue Jun 7, 2018
@mbgrydeland
Make HTC_RxStuff reserve a byte for NUL character again and fix maxbytes
ae74e90 
handling

This partly reverses the fix for varnishcache#2285, as that fix was missdiagnosed. The
original fix for varnishcache#2285 also introduced a bug similar to varnishcache#2219 where the
proxy code would not find space to insert a NUL character.

The real problem in varnishcache#2285 (and the duplicate varnishcache#2624) appears during H/2
prior knowledge upgrade, and the difference in maxbytes argument to
HTC_RxStuff between the H/1 and H/2 code paths. If the pipelined bytes
exceeded the maxbytes input on the second run we would run into an assert.

(varnishcache#2624 has received a workaround in the H/2 code that perhaps should be
reverted).

Changes in this patch:

HTC_RxStuff will now reserve a space to insert a NUL character for those
completion functions that needs that (HTTP/1 and Proxy_V1).

HTC_RxStuff will no longer assert on seeing pipelined data exceeding the
maxbytes input.
@mbgrydeland mbgrydeland mentioned this issue Jun 7, 2018
Closed
mbgrydeland added a commit to mbgrydeland/varnish-cache that referenced this issue Jun 7, 2018
@mbgrydeland
Make HTC_RxStuff reserve a byte for NUL character again and fix maxbytes
3e91319 
handling

This partly reverses the fix for varnishcache#2285, as that fix was missdiagnosed. The
original fix for varnishcache#2285 also introduced a bug similar to varnishcache#2219 where the
proxy code would not find space to insert a NUL character.

The real problem in varnishcache#2285 (and the duplicate varnishcache#2624) appears during H/2
prior knowledge upgrade, and the difference in maxbytes argument to
HTC_RxStuff between the H/1 and H/2 code paths. If the pipelined bytes
exceeded the maxbytes input on the second run we would run into an assert.

(varnishcache#2624 has received a workaround in the H/2 code that perhaps should be
reverted).

Changes in this patch:

HTC_RxStuff will now reserve a space to insert a NUL character for those
completion functions that needs that (HTTP/1 and Proxy_V1).

HTC_RxStuff will no longer assert on seeing pipelined data exceeding the
maxbytes input.
mbgrydeland added a commit that referenced this issue Jun 11, 2018
@mbgrydeland
New fix for #2285 and #2624
ed5c43b 
Previous fix for #2285 (and the duplicate #2624) was missdiagnosed. The
problem stems from a wrong assumption that the number of bytes already
pipelined will be less than maxbytes, with maxbytes beeing the maximum
number of bytes the HTC_RxStuff may need to get a full work unit. That
assumption may fail during the H/1 to H/2 upgrade path where maxbytes
change with the context, or during runtime changing of parameters.

This patch makes HTC_RxStuff not assert if the pipelined data turned out
to exceed maxbytes, but return overflow if we run out of workspace.

(#2624 has received a workaround in the H/2 code that perhaps should be
reverted).
Dridi pushed a commit to Dridi/varnish-cache that referenced this issue Jun 20, 2018
@mbgrydeland @Dridi
New fix for varnishcache#2285 and varnishcache#2624
a61b713 
Previous fix for varnishcache#2285 (and the duplicate varnishcache#2624) was missdiagnosed. The
problem stems from a wrong assumption that the number of bytes already
pipelined will be less than maxbytes, with maxbytes beeing the maximum
number of bytes the HTC_RxStuff may need to get a full work unit. That
assumption may fail during the H/1 to H/2 upgrade path where maxbytes
change with the context, or during runtime changing of parameters.

This patch makes HTC_RxStuff not assert if the pipelined data turned out
to exceed maxbytes, but return overflow if we run out of workspace.

(varnishcache#2624 has received a workaround in the H/2 code that perhaps should be
reverted).
dmatetelki pushed a commit to dmatetelki/varnish-cache that referenced this issue Mar 14, 2019
@mbgrydeland @dmatetelki
New fix for varnishcache#2285 and varnishcache#2624
bef9995 
Previous fix for varnishcache#2285 (and the duplicate varnishcache#2624) was missdiagnosed. The
problem stems from a wrong assumption that the number of bytes already
pipelined will be less than maxbytes, with maxbytes beeing the maximum
number of bytes the HTC_RxStuff may need to get a full work unit. That
assumption may fail during the H/1 to H/2 upgrade path where maxbytes
change with the context, or during runtime changing of parameters.

This patch makes HTC_RxStuff not assert if the pipelined data turned out
to exceed maxbytes, but return overflow if we run out of workspace.

(varnishcache#2624 has received a workaround in the H/2 code that perhaps should be
reverted).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
b=bug c=H/2 c=varnishd
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@simonvik @nigoroll @vssync @hermunn