Skip to content
This repository was archived by the owner on Feb 16, 2026. It is now read-only.

bugfix : coredump causes by memcpy in vbf_stp_error - #2429

Closed
shamger wants to merge 1 commit into
varnishcache:masterfrom
shamger:my-varnish
Closed

shamger wants to merge 1 commit into
varnishcache:masterfrom
shamger:my-varnish

Conversation

@shamger

@shamger shamger commented Sep 18, 2017 •

Copy link
Copy Markdown

See the code in vbf_stp_error(cache_fetch.c). The variable of l is the storage size that has been got, while ll is the length of VSB_data(synth_body).

In this case, the third parameter of memcpy is 4096, but the actual length of VSB_data(synth_body) is only 284. If the memory address from VSB_data(synth_body)+284 is unavailable, coredump will occur!

@dridi

dridi commented Sep 18, 2017

Copy link
Copy Markdown
Member

See #2427 for more details.

@shamger can you reproduce this with a test case?

@shamger

shamger commented Sep 18, 2017 •

Copy link
Copy Markdown
Author

@dridi
Yes. It only reappears when varnish is working under highly concurrent and network is weak. But I can reproduce it by changing the vcl file. See below:

  1. Using file storage, varnish will return one page size(4096 bytes) to vbf_stp_error at least.
  2. Backend don't response anything for tcp connection, so varnish will run the fetch_error process.
  3. Add "set beresp.ttl = 2m; set beresp.grace = 5m;" in vcl_backend_error subroutine to avoid varnish using stv_transient to store an obj.
  4. Add "set req.hash_always_miss = true", varnish always will fetch obj from backend.
  5. Use appache bench as: ab -n 1000 -c 100 http://localhost/test.html

@fgsch

fgsch commented Sep 18, 2017 •

Copy link
Copy Markdown
Member

IMO the correct fix is to ensure that GetStorage never returns a size larger than what we asked for, but might be more complicated than that.

@fgsch

fgsch commented Sep 18, 2017 •

Copy link
Copy Markdown
Member

Testcase (with ASAN):

varnishtest ""

varnish v1 -arg "-sfoo=file,${tmpdir}/_.file,10m" -vcl {
        backend default {
                .host = "${bad_ip}";
                .connect_timeout = 0.1s;
        }
        sub vcl_recv {
                set req.hash_always_miss = true;
        }
        sub vcl_backend_error {
                set beresp.ttl = 2m;
                synthetic("error");
                return (deliver);
        }
} -start

client c1 -repeat 10 {
        txreq
        rxresp
} -run

@fgsch

fgsch commented Sep 18, 2017 •

Copy link
Copy Markdown
Member

Test output (panic only):

***  v1    0.9 debug|Info: Child (14386) said =================================================================
***  v1    0.9 debug|Info: Child (14386) said ==14386==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000009b00 at pc 0x000108f3d2f8 bp 0x700002632c30 sp 0x7000026323e0
***  v1    0.9 debug|Info: Child (14386) said READ of size 4096 at 0x602000009b00 thread T12
***  v1    1.0 debug|Info: Child (14386) said     #0 0x108f3d2f7 in __asan_memcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4d2f7)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x1082542ab in vbf_stp_error cache_fetch.c:902
***  v1    1.2 debug|Info: Child (14386) said     #2 0x10822f81f in vbf_fetch_thread steps.h:58
***  v1    1.2 debug|Info: Child (14386) said     #3 0x1083fef58 in Pool_Work_Thread cache_wrk.c:375
***  v1    1.2 debug|Info: Child (14386) said     #4 0x1083fa276 in WRK_Thread cache_wrk.c:128
***  v1    1.2 debug|Info: Child (14386) said     #5 0x1083f94f0 in pool_thread cache_wrk.c:406
***  v1    1.2 debug|Info: Child (14386) said     #6 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #7 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #8 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said 0x602000009b00 is located 0 bytes to the right of 16-byte region [0x602000009af0,0x602000009b00)
***  v1    1.2 debug|Info: Child (14386) said allocated by thread T12 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f4618c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5618c)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x10865a461 in VSB_newbuf vsb.c:201
***  v1    1.2 debug|Info: Child (14386) said     #2 0x108659920 in VSB_new vsb.c:229
***  v1    1.2 debug|Info: Child (14386) said     #3 0x108252a55 in vbf_stp_error cache_fetch.c:862
***  v1    1.2 debug|Info: Child (14386) said     #4 0x10822f81f in vbf_fetch_thread steps.h:58
***  v1    1.2 debug|Info: Child (14386) said     #5 0x1083fef58 in Pool_Work_Thread cache_wrk.c:375
***  v1    1.2 debug|Info: Child (14386) said     #6 0x1083fa276 in WRK_Thread cache_wrk.c:128
***  v1    1.2 debug|Info: Child (14386) said     #7 0x1083f94f0 in pool_thread cache_wrk.c:406
***  v1    1.2 debug|Info: Child (14386) said     #8 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #9 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #10 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T12 created by T4 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x1083f7bc4 in pool_breed cache_wrk.c:431
***  v1    1.2 debug|Info: Child (14386) said     #2 0x1083f3052 in pool_herder cache_wrk.c:490
***  v1    1.2 debug|Info: Child (14386) said     #3 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #4 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #5 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T4 created by T3 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x10830c143 in pool_mkpool cache_pool.c:162
***  v1    1.2 debug|Info: Child (14386) said     #2 0x108306f84 in pool_poolherder cache_pool.c:196
***  v1    1.2 debug|Info: Child (14386) said     #3 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #4 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #5 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T3 created by T0 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x108306b6a in Pool_Init cache_pool.c:255
***  v1    1.2 debug|Info: Child (14386) said     #2 0x1082d10f2 in child_main cache_main.c:260
***  v1    1.2 debug|Info: Child (14386) said     #3 0x1084e0347 in mgt_launch_child mgt_child.c:399
***  v1    1.2 debug|Info: Child (14386) said     #4 0x1084e3e87 in mch_cli_server_start mgt_child.c:665
***  v1    1.2 debug|Info: Child (14386) said     #5 0x10863490f in cls_dispatch vcli_serve.c:229
***  v1    1.2 debug|Info: Child (14386) said     #6 0x1086324f7 in cls_vlu2 vcli_serve.c:289
***  v1    1.2 debug|Info: Child (14386) said     #7 0x108620290 in cls_vlu vcli_serve.c:364
***  v1    1.2 debug|Info: Child (14386) said     #8 0x10864b742 in LineUpProcess vlu.c:98
***  v1    1.2 debug|Info: Child (14386) said     #9 0x10864ad92 in VLU_Fd vlu.c:123
***  v1    1.2 debug|Info: Child (14386) said     #10 0x10862a5d9 in VCLS_PollFd vcli_serve.c:554
***  v1    1.2 debug|Info: Child (14386) said     #11 0x1084e7e31 in mgt_cli_callback2 mgt_cli.c:397
***  v1    1.2 debug|Info: Child (14386) said     #12 0x10863f33e in vev_schedule_one vev.c:450
***  v1    1.2 debug|Info: Child (14386) said     #13 0x10863d01b in vev_schedule vev.c:344
***  v1    1.2 debug|Info: Child (14386) said     #14 0x1084fdd4b in main mgt_main.c:929
***  v1    1.2 debug|Info: Child (14386) said     #15 0x7fff9d87d234 in start (libdyld.dylib:x86_64+0x5234)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4d2f7) in __asan_memcpy
***  v1    1.2 debug|Info: Child (14386) said Shadow bytes around the buggy address:
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001310: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001330: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001340: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa 00 00
***  v1    1.2 debug|Info: Child (14386) said =>0x1c0400001360:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c04000013a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c04000013b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said Shadow byte legend (one shadow byte represents 8 application bytes):
***  v1    1.2 debug|Info: Child (14386) said   Addressable:           00
***  v1    1.2 debug|Info: Child (14386) said   Partially addressable: 01 02 03 04 05 06 07 
***  v1    1.2 debug|Info: Child (14386) said   Heap left redzone:       fa
***  v1    1.2 debug|Info: Child (14386) said   Freed heap region:       fd
***  v1    1.2 debug|Info: Child (14386) said   Stack left redzone:      f1
***  v1    1.2 debug|Info: Child (14386) said   Stack mid redzone:       f2
***  v1    1.2 debug|Info: Child (14386) said   Stack right redzone:     f3
***  v1    1.2 debug|Info: Child (14386) said   Stack after return:      f5
***  v1    1.2 debug|Info: Child (14386) said   Stack use after scope:   f8
***  v1    1.2 debug|Info: Child (14386) said   Global redzone:          f9
***  v1    1.2 debug|Info: Child (14386) said   Global init order:       f6
***  v1    1.2 debug|Info: Child (14386) said   Poisoned by user:        f7
***  v1    1.2 debug|Info: Child (14386) said   Container overflow:      fc
***  v1    1.2 debug|Info: Child (14386) said   Array cookie:            ac
***  v1    1.2 debug|Info: Child (14386) said   Intra object redzone:    bb
***  v1    1.2 debug|Info: Child (14386) said   ASan internal:           fe
***  v1    1.2 debug|Info: Child (14386) said   Left alloca redzone:     ca
***  v1    1.2 debug|Info: Child (14386) said   Right alloca redzone:    cb
***  v1    1.2 debug|Info: Child (14386) said ==14386==ABORTING
---- c1    1.2 HTTP rx EOF (fd:15 read: Undefined error: 0) 1
*    top   1.2 RESETTING after a.vtc
**   v1    1.2 Wait
**** v1    1.2 CLI TX|panic.clear
***  v1    1.2 debug|Error: Child (14386) died signal=6
***  v1    1.2 debug|Error: Child (14386) Panic at: Mon, 18 Sep 2017 11:24:07 GMT
***  v1    1.2 debug|Wrong turn at mgt/mgt_child.c:287:
***  v1    1.2 debug|Signal 6 (Abort trap: 6) received at 0x7fff9d9abd42 si_code 0
***  v1    1.2 debug|version = varnish-trunk revision 8e0b7b204, vrt api = 6.1
***  v1    1.2 debug|ident = Darwin,16.7.0,x86_64,-jnone,-sfile,-smalloc,-hcritbit,kqueue
***  v1    1.2 debug|now = 426422.783616 (mono), 1505733847.909924 (real)
***  v1    1.2 debug|Backtrace:
***  v1    1.2 debug|  0x1082f5c59: 0   varnishd                            0x00000001082f5c59 pan_backtrace + 361
***  v1    1.2 debug|  0x1082f5875: 0   varnishd                            0x00000001082f5875 pan_ic + 1525
***  v1    1.2 debug|  0x10861189b: 0   varnishd                            0x000000010861189b VAS_Fail + 379
***  v1    1.2 debug|  0x1084e3928: 0   varnishd                            0x00000001084e3928 child_signal_handler + 1272
***  v1    1.2 debug|  0x7fff9da8cb3a: 0   libsystem_platform.dylib            0x00007fff9da8cb3a _sigtramp + 26
***  v1    1.2 debug|  0x7000026323e0: 0   ???                                 0x00007000026323e0 0x0 + 123145342362592
***  v1    1.2 debug|  0x7fff9d911420: 0   libsystem_c.dylib                   0x00007fff9d911420 abort + 129
***  v1    1.2 debug|  0x108f68996: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f68996 _ZN11__sanitizer5AbortEv + 70
***  v1    1.2 debug|  0x108f64268: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f64268 _ZN11__sanitizer3DieEv + 120
***  v1    1.2 debug|  0x108f4b207: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f4b207 _ZN6__asan19ScopedInErrorReportD2Ev + 311
***  v1    1.2 debug|errno = 25 (Inappropriate ioctl for device)
***  v1    1.2 debug|thread = (cache-worker)
***  v1    1.2 debug|thr.req = 0x0 {
***  v1    1.2 debug|},
***  v1    1.2 debug|thr.busyobj = 0x631000050820 {
***  v1    1.2 debug|  ws = 0x6310000508a0 {
***  v1    1.2 debug|    id = \"bo\",
***  v1    1.2 debug|    {s, f, r, e} = {0x631000052760, +264, 0x0, +57496},
***  v1    1.2 debug|  },
***  v1    1.2 debug|  retries = 0, failed = 0, flags = {do_stream},
***  v1    1.2 debug|  director_req = 0x61200000b378 {
***  v1    1.2 debug|    vcl_name = default,
***  v1    1.2 debug|    type = backend {
***  v1    1.2 debug|      display_name = vcl1.default,
***  v1    1.2 debug|      ipv4 = 192.0.2.255,
***  v1    1.2 debug|      port = 80,
***  v1    1.2 debug|      hosthdr = 192.0.2.255,
***  v1    1.2 debug|      health = healthy,
***  v1    1.2 debug|      admin_health = probe, changed = 1505733847.432227,
***  v1    1.2 debug|      n_conn = 0,
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  director_resp = director_req,
***  v1    1.2 debug|  http[bereq] = 0x631000050e70 {
***  v1    1.2 debug|    ws = 0x6310000508a0 {
***  v1    1.2 debug|      [Already dumped, see above]
***  v1    1.2 debug|    },
***  v1    1.2 debug|    hdrs {
***  v1    1.2 debug|      \"GET\",
***  v1    1.2 debug|      \"/\",
***  v1    1.2 debug|      \"HTTP/1.1\",
***  v1    1.2 debug|      \"X-Forwarded-For: 127.0.0.1\",
***  v1    1.2 debug|      \"Accept-Encoding: gzip\",
***  v1    1.2 debug|      \"X-Varnish: 1002\",
***  v1    1.2 debug|      \"Host: 192.0.2.255\",
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  http[beresp] = 0x6310000512e8 {
***  v1    1.2 debug|    ws = 0x6310000508a0 {
***  v1    1.2 debug|      [Already dumped, see above]
***  v1    1.2 debug|    },
***  v1    1.2 debug|    hdrs {
***  v1    1.2 debug|      \"HTTP/1.1\",
***  v1    1.2 debug|      \"503\",
***  v1    1.2 debug|      \"Backend fetch failed\",
***  v1    1.2 debug|      \"Date: Mon, 18 Sep 2017 11:24:07 GMT\",
***  v1    1.2 debug|      \"Server: Varnish\",
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  objcore[fetch] = 0x60e000006be0 {
***  v1    1.2 debug|    refcnt = 2,
***  v1    1.2 debug|    flags = {busy},
***  v1    1.2 debug|    exp_flags = {},
***  v1    1.2 debug|    boc = 0x608000007b20 {
***  v1    1.2 debug|      refcnt = 2,
***  v1    1.2 debug|      state = req_done,
***  v1    1.2 debug|      vary = 0x0,
***  v1    1.2 debug|      stevedore_priv = 0x0,
***  v1    1.2 debug|    },
***  v1    1.2 debug|    exp = {1505733847.643175, 120.000000, 0.000000, 0.000000},
***  v1    1.2 debug|    objhead = 0x60b0000054d0,
***  v1    1.2 debug|    stevedore = 0x60f00000e230 (file foo) {
***  v1    1.2 debug|      Simple = 0x10dad6000,
***  v1    1.2 debug|      Obj = 0x60d000006438 {priv=0x60d000006430, ptr=0x10dad6000, len=216, space=4096},
***  v1    1.2 debug|      LEN = 0x0...0,
***  v1    1.2 debug|      VXID = 0x000003ea,
***  v1    1.2 debug|      FLAGS = 0x00,
***  v1    1.2 debug|      GZIPBITS = 0x0...0,
***  v1    1.2 debug|      LASTMODIFIED = 0x41d66feb35c00000,
***  v1    1.2 debug|      VARY = {len=0, ptr=0x0},
***  v1    1.2 debug|      HEADERS = {len=96, ptr=0x10dad6078},
***  v1    1.2 debug|      Body = 0x60d000006368 {priv=0x60d000006360, ptr=0x10dad7000, len=0, space=4096},
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  vcl = {
***  v1    1.2 debug|    name = \"vcl1\",
***  v1    1.2 debug|    busy = 2,
***  v1    1.2 debug|    discard = 0,
***  v1    1.2 debug|    state = auto,
***  v1    1.2 debug|    temp = warm,
***  v1    1.2 debug|    conf = {
***  v1    1.2 debug|      srcname = {
***  v1    1.2 debug|        \"<vcl.inline>\",
***  v1    1.2 debug|        \"Builtin\",
***  v1    1.2 debug|      },
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  vmods = {
***  v1    1.2 debug|  },
***  v1    1.2 debug|},

mbgrydeland added a commit to mbgrydeland/varnish-cache that referenced this pull request Sep 18, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Fixes: varnishcache#2429
hermunn pushed a commit that referenced this pull request Sep 19, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: #2429
@hermunn

hermunn commented Sep 19, 2017

Copy link
Copy Markdown
Member

@shamger Thanks for the pull request. As you can see, @mbgrydeland has commited the fix you proposed, and I have back ported it to 4.1 as 19a7318.

@shamger
shamger deleted the my-varnish branch September 19, 2017 11:45
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: #2429
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
@carnil

carnil commented Nov 15, 2017

Copy link
Copy Markdown

This is CVE-2017-8807

@mbgrydeland

Copy link
Copy Markdown
Contributor

This is Varnish Security Vulnerability VSV00002. See https://varnish-cache.org/security/VSV00002.html for details.

dmatetelki pushed a commit to dmatetelki/varnish-cache that referenced this pull request Mar 14, 2019
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: varnishcache#2429
dmatetelki pushed a commit to dmatetelki/varnish-cache that referenced this pull request Mar 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants