Skip to content

bugfix : coredump causes by memcpy in vbf_stp_error #2429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

shamger
Copy link

@shamger shamger commented Sep 18, 2017

See the code in vbf_stp_error(cache_fetch.c). The variable of l is the storage size that has been got, while ll is the length of VSB_data(synth_body).

In this case, the third parameter of memcpy is 4096, but the actual length of VSB_data(synth_body) is only 284. If the memory address from VSB_data(synth_body)+284 is unavailable, coredump will occur!

@dridi
Copy link
Member

dridi commented Sep 18, 2017

See #2427 for more details.

@shamger can you reproduce this with a test case?

@shamger
Copy link
Author

shamger commented Sep 18, 2017

@dridi
Yes. It only reappears when varnish is working under highly concurrent and network is weak. But I can reproduce it by changing the vcl file. See below:

  1. Using file storage, varnish will return one page size(4096 bytes) to vbf_stp_error at least.
  2. Backend don't response anything for tcp connection, so varnish will run the fetch_error process.
  3. Add "set beresp.ttl = 2m; set beresp.grace = 5m;" in vcl_backend_error subroutine to avoid varnish using stv_transient to store an obj.
  4. Add "set req.hash_always_miss = true", varnish always will fetch obj from backend.
  5. Use appache bench as: ab -n 1000 -c 100 http://localhost/test.html

@fgsch
Copy link
Member

fgsch commented Sep 18, 2017

IMO the correct fix is to ensure that GetStorage never returns a size larger than what we asked for, but might be more complicated than that.

@fgsch
Copy link
Member

fgsch commented Sep 18, 2017

Testcase (with ASAN):

varnishtest ""

varnish v1 -arg "-sfoo=file,${tmpdir}/_.file,10m" -vcl {
        backend default {
                .host = "${bad_ip}";
                .connect_timeout = 0.1s;
        }
        sub vcl_recv {
                set req.hash_always_miss = true;
        }
        sub vcl_backend_error {
                set beresp.ttl = 2m;
                synthetic("error");
                return (deliver);
        }
} -start

client c1 -repeat 10 {
        txreq
        rxresp
} -run

@fgsch
Copy link
Member

fgsch commented Sep 18, 2017

Test output (panic only):

***  v1    0.9 debug|Info: Child (14386) said =================================================================
***  v1    0.9 debug|Info: Child (14386) said ==14386==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000009b00 at pc 0x000108f3d2f8 bp 0x700002632c30 sp 0x7000026323e0
***  v1    0.9 debug|Info: Child (14386) said READ of size 4096 at 0x602000009b00 thread T12
***  v1    1.0 debug|Info: Child (14386) said     #0 0x108f3d2f7 in __asan_memcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4d2f7)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x1082542ab in vbf_stp_error cache_fetch.c:902
***  v1    1.2 debug|Info: Child (14386) said     #2 0x10822f81f in vbf_fetch_thread steps.h:58
***  v1    1.2 debug|Info: Child (14386) said     #3 0x1083fef58 in Pool_Work_Thread cache_wrk.c:375
***  v1    1.2 debug|Info: Child (14386) said     #4 0x1083fa276 in WRK_Thread cache_wrk.c:128
***  v1    1.2 debug|Info: Child (14386) said     #5 0x1083f94f0 in pool_thread cache_wrk.c:406
***  v1    1.2 debug|Info: Child (14386) said     #6 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #7 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #8 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said 0x602000009b00 is located 0 bytes to the right of 16-byte region [0x602000009af0,0x602000009b00)
***  v1    1.2 debug|Info: Child (14386) said allocated by thread T12 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f4618c in wrap_malloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5618c)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x10865a461 in VSB_newbuf vsb.c:201
***  v1    1.2 debug|Info: Child (14386) said     #2 0x108659920 in VSB_new vsb.c:229
***  v1    1.2 debug|Info: Child (14386) said     #3 0x108252a55 in vbf_stp_error cache_fetch.c:862
***  v1    1.2 debug|Info: Child (14386) said     #4 0x10822f81f in vbf_fetch_thread steps.h:58
***  v1    1.2 debug|Info: Child (14386) said     #5 0x1083fef58 in Pool_Work_Thread cache_wrk.c:375
***  v1    1.2 debug|Info: Child (14386) said     #6 0x1083fa276 in WRK_Thread cache_wrk.c:128
***  v1    1.2 debug|Info: Child (14386) said     #7 0x1083f94f0 in pool_thread cache_wrk.c:406
***  v1    1.2 debug|Info: Child (14386) said     #8 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #9 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #10 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T12 created by T4 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x1083f7bc4 in pool_breed cache_wrk.c:431
***  v1    1.2 debug|Info: Child (14386) said     #2 0x1083f3052 in pool_herder cache_wrk.c:490
***  v1    1.2 debug|Info: Child (14386) said     #3 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #4 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #5 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T4 created by T3 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x10830c143 in pool_mkpool cache_pool.c:162
***  v1    1.2 debug|Info: Child (14386) said     #2 0x108306f84 in pool_poolherder cache_pool.c:196
***  v1    1.2 debug|Info: Child (14386) said     #3 0x7fff9da9693a in _pthread_body (libsystem_pthread.dylib:x86_64+0x393a)
***  v1    1.2 debug|Info: Child (14386) said     #4 0x7fff9da96886 in _pthread_start (libsystem_pthread.dylib:x86_64+0x3886)
***  v1    1.2 debug|Info: Child (14386) said     #5 0x7fff9da9608c in thread_start (libsystem_pthread.dylib:x86_64+0x308c)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said Thread T3 created by T0 here:
***  v1    1.2 debug|Info: Child (14386) said     #0 0x108f3cca6 in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4cca6)
***  v1    1.2 debug|Info: Child (14386) said     #1 0x108306b6a in Pool_Init cache_pool.c:255
***  v1    1.2 debug|Info: Child (14386) said     #2 0x1082d10f2 in child_main cache_main.c:260
***  v1    1.2 debug|Info: Child (14386) said     #3 0x1084e0347 in mgt_launch_child mgt_child.c:399
***  v1    1.2 debug|Info: Child (14386) said     #4 0x1084e3e87 in mch_cli_server_start mgt_child.c:665
***  v1    1.2 debug|Info: Child (14386) said     #5 0x10863490f in cls_dispatch vcli_serve.c:229
***  v1    1.2 debug|Info: Child (14386) said     #6 0x1086324f7 in cls_vlu2 vcli_serve.c:289
***  v1    1.2 debug|Info: Child (14386) said     #7 0x108620290 in cls_vlu vcli_serve.c:364
***  v1    1.2 debug|Info: Child (14386) said     #8 0x10864b742 in LineUpProcess vlu.c:98
***  v1    1.2 debug|Info: Child (14386) said     #9 0x10864ad92 in VLU_Fd vlu.c:123
***  v1    1.2 debug|Info: Child (14386) said     #10 0x10862a5d9 in VCLS_PollFd vcli_serve.c:554
***  v1    1.2 debug|Info: Child (14386) said     #11 0x1084e7e31 in mgt_cli_callback2 mgt_cli.c:397
***  v1    1.2 debug|Info: Child (14386) said     #12 0x10863f33e in vev_schedule_one vev.c:450
***  v1    1.2 debug|Info: Child (14386) said     #13 0x10863d01b in vev_schedule vev.c:344
***  v1    1.2 debug|Info: Child (14386) said     #14 0x1084fdd4b in main mgt_main.c:929
***  v1    1.2 debug|Info: Child (14386) said     #15 0x7fff9d87d234 in start (libdyld.dylib:x86_64+0x5234)
***  v1    1.2 debug|Info: Child (14386) said 
***  v1    1.2 debug|Info: Child (14386) said SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x4d2f7) in __asan_memcpy
***  v1    1.2 debug|Info: Child (14386) said Shadow bytes around the buggy address:
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001310: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001320: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001330: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001340: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa 00 00
***  v1    1.2 debug|Info: Child (14386) said =>0x1c0400001360:[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c0400001390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c04000013a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said   0x1c04000013b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
***  v1    1.2 debug|Info: Child (14386) said Shadow byte legend (one shadow byte represents 8 application bytes):
***  v1    1.2 debug|Info: Child (14386) said   Addressable:           00
***  v1    1.2 debug|Info: Child (14386) said   Partially addressable: 01 02 03 04 05 06 07 
***  v1    1.2 debug|Info: Child (14386) said   Heap left redzone:       fa
***  v1    1.2 debug|Info: Child (14386) said   Freed heap region:       fd
***  v1    1.2 debug|Info: Child (14386) said   Stack left redzone:      f1
***  v1    1.2 debug|Info: Child (14386) said   Stack mid redzone:       f2
***  v1    1.2 debug|Info: Child (14386) said   Stack right redzone:     f3
***  v1    1.2 debug|Info: Child (14386) said   Stack after return:      f5
***  v1    1.2 debug|Info: Child (14386) said   Stack use after scope:   f8
***  v1    1.2 debug|Info: Child (14386) said   Global redzone:          f9
***  v1    1.2 debug|Info: Child (14386) said   Global init order:       f6
***  v1    1.2 debug|Info: Child (14386) said   Poisoned by user:        f7
***  v1    1.2 debug|Info: Child (14386) said   Container overflow:      fc
***  v1    1.2 debug|Info: Child (14386) said   Array cookie:            ac
***  v1    1.2 debug|Info: Child (14386) said   Intra object redzone:    bb
***  v1    1.2 debug|Info: Child (14386) said   ASan internal:           fe
***  v1    1.2 debug|Info: Child (14386) said   Left alloca redzone:     ca
***  v1    1.2 debug|Info: Child (14386) said   Right alloca redzone:    cb
***  v1    1.2 debug|Info: Child (14386) said ==14386==ABORTING
---- c1    1.2 HTTP rx EOF (fd:15 read: Undefined error: 0) 1
*    top   1.2 RESETTING after a.vtc
**   v1    1.2 Wait
**** v1    1.2 CLI TX|panic.clear
***  v1    1.2 debug|Error: Child (14386) died signal=6
***  v1    1.2 debug|Error: Child (14386) Panic at: Mon, 18 Sep 2017 11:24:07 GMT
***  v1    1.2 debug|Wrong turn at mgt/mgt_child.c:287:
***  v1    1.2 debug|Signal 6 (Abort trap: 6) received at 0x7fff9d9abd42 si_code 0
***  v1    1.2 debug|version = varnish-trunk revision 8e0b7b204, vrt api = 6.1
***  v1    1.2 debug|ident = Darwin,16.7.0,x86_64,-jnone,-sfile,-smalloc,-hcritbit,kqueue
***  v1    1.2 debug|now = 426422.783616 (mono), 1505733847.909924 (real)
***  v1    1.2 debug|Backtrace:
***  v1    1.2 debug|  0x1082f5c59: 0   varnishd                            0x00000001082f5c59 pan_backtrace + 361
***  v1    1.2 debug|  0x1082f5875: 0   varnishd                            0x00000001082f5875 pan_ic + 1525
***  v1    1.2 debug|  0x10861189b: 0   varnishd                            0x000000010861189b VAS_Fail + 379
***  v1    1.2 debug|  0x1084e3928: 0   varnishd                            0x00000001084e3928 child_signal_handler + 1272
***  v1    1.2 debug|  0x7fff9da8cb3a: 0   libsystem_platform.dylib            0x00007fff9da8cb3a _sigtramp + 26
***  v1    1.2 debug|  0x7000026323e0: 0   ???                                 0x00007000026323e0 0x0 + 123145342362592
***  v1    1.2 debug|  0x7fff9d911420: 0   libsystem_c.dylib                   0x00007fff9d911420 abort + 129
***  v1    1.2 debug|  0x108f68996: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f68996 _ZN11__sanitizer5AbortEv + 70
***  v1    1.2 debug|  0x108f64268: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f64268 _ZN11__sanitizer3DieEv + 120
***  v1    1.2 debug|  0x108f4b207: 0   libclang_rt.asan_osx_dynamic.dylib  0x0000000108f4b207 _ZN6__asan19ScopedInErrorReportD2Ev + 311
***  v1    1.2 debug|errno = 25 (Inappropriate ioctl for device)
***  v1    1.2 debug|thread = (cache-worker)
***  v1    1.2 debug|thr.req = 0x0 {
***  v1    1.2 debug|},
***  v1    1.2 debug|thr.busyobj = 0x631000050820 {
***  v1    1.2 debug|  ws = 0x6310000508a0 {
***  v1    1.2 debug|    id = \"bo\",
***  v1    1.2 debug|    {s, f, r, e} = {0x631000052760, +264, 0x0, +57496},
***  v1    1.2 debug|  },
***  v1    1.2 debug|  retries = 0, failed = 0, flags = {do_stream},
***  v1    1.2 debug|  director_req = 0x61200000b378 {
***  v1    1.2 debug|    vcl_name = default,
***  v1    1.2 debug|    type = backend {
***  v1    1.2 debug|      display_name = vcl1.default,
***  v1    1.2 debug|      ipv4 = 192.0.2.255,
***  v1    1.2 debug|      port = 80,
***  v1    1.2 debug|      hosthdr = 192.0.2.255,
***  v1    1.2 debug|      health = healthy,
***  v1    1.2 debug|      admin_health = probe, changed = 1505733847.432227,
***  v1    1.2 debug|      n_conn = 0,
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  director_resp = director_req,
***  v1    1.2 debug|  http[bereq] = 0x631000050e70 {
***  v1    1.2 debug|    ws = 0x6310000508a0 {
***  v1    1.2 debug|      [Already dumped, see above]
***  v1    1.2 debug|    },
***  v1    1.2 debug|    hdrs {
***  v1    1.2 debug|      \"GET\",
***  v1    1.2 debug|      \"/\",
***  v1    1.2 debug|      \"HTTP/1.1\",
***  v1    1.2 debug|      \"X-Forwarded-For: 127.0.0.1\",
***  v1    1.2 debug|      \"Accept-Encoding: gzip\",
***  v1    1.2 debug|      \"X-Varnish: 1002\",
***  v1    1.2 debug|      \"Host: 192.0.2.255\",
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  http[beresp] = 0x6310000512e8 {
***  v1    1.2 debug|    ws = 0x6310000508a0 {
***  v1    1.2 debug|      [Already dumped, see above]
***  v1    1.2 debug|    },
***  v1    1.2 debug|    hdrs {
***  v1    1.2 debug|      \"HTTP/1.1\",
***  v1    1.2 debug|      \"503\",
***  v1    1.2 debug|      \"Backend fetch failed\",
***  v1    1.2 debug|      \"Date: Mon, 18 Sep 2017 11:24:07 GMT\",
***  v1    1.2 debug|      \"Server: Varnish\",
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  objcore[fetch] = 0x60e000006be0 {
***  v1    1.2 debug|    refcnt = 2,
***  v1    1.2 debug|    flags = {busy},
***  v1    1.2 debug|    exp_flags = {},
***  v1    1.2 debug|    boc = 0x608000007b20 {
***  v1    1.2 debug|      refcnt = 2,
***  v1    1.2 debug|      state = req_done,
***  v1    1.2 debug|      vary = 0x0,
***  v1    1.2 debug|      stevedore_priv = 0x0,
***  v1    1.2 debug|    },
***  v1    1.2 debug|    exp = {1505733847.643175, 120.000000, 0.000000, 0.000000},
***  v1    1.2 debug|    objhead = 0x60b0000054d0,
***  v1    1.2 debug|    stevedore = 0x60f00000e230 (file foo) {
***  v1    1.2 debug|      Simple = 0x10dad6000,
***  v1    1.2 debug|      Obj = 0x60d000006438 {priv=0x60d000006430, ptr=0x10dad6000, len=216, space=4096},
***  v1    1.2 debug|      LEN = 0x0...0,
***  v1    1.2 debug|      VXID = 0x000003ea,
***  v1    1.2 debug|      FLAGS = 0x00,
***  v1    1.2 debug|      GZIPBITS = 0x0...0,
***  v1    1.2 debug|      LASTMODIFIED = 0x41d66feb35c00000,
***  v1    1.2 debug|      VARY = {len=0, ptr=0x0},
***  v1    1.2 debug|      HEADERS = {len=96, ptr=0x10dad6078},
***  v1    1.2 debug|      Body = 0x60d000006368 {priv=0x60d000006360, ptr=0x10dad7000, len=0, space=4096},
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  vcl = {
***  v1    1.2 debug|    name = \"vcl1\",
***  v1    1.2 debug|    busy = 2,
***  v1    1.2 debug|    discard = 0,
***  v1    1.2 debug|    state = auto,
***  v1    1.2 debug|    temp = warm,
***  v1    1.2 debug|    conf = {
***  v1    1.2 debug|      srcname = {
***  v1    1.2 debug|        \"<vcl.inline>\",
***  v1    1.2 debug|        \"Builtin\",
***  v1    1.2 debug|      },
***  v1    1.2 debug|    },
***  v1    1.2 debug|  },
***  v1    1.2 debug|  vmods = {
***  v1    1.2 debug|  },
***  v1    1.2 debug|},

mbgrydeland added a commit to mbgrydeland/varnish-cache that referenced this pull request Sep 18, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Fixes: varnishcache#2429
hermunn pushed a commit that referenced this pull request Sep 19, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: #2429
@hermunn
Copy link
Member

hermunn commented Sep 19, 2017

@shamger Thanks for the pull request. As you can see, @mbgrydeland has commited the fix you proposed, and I have back ported it to 4.1 as 19a7318.

@shamger shamger deleted the my-varnish branch September 19, 2017 11:45
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: #2429
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
mbgrydeland added a commit that referenced this pull request Nov 14, 2017
@carnil
Copy link

carnil commented Nov 15, 2017

This is CVE-2017-8807

@mbgrydeland
Copy link
Contributor

This is Varnish Security Vulnerability VSV00002. See https://varnish-cache.org/security/VSV00002.html for details.

dmatetelki pushed a commit to dmatetelki/varnish-cache that referenced this pull request Mar 14, 2019
The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: varnishcache#2429
dmatetelki pushed a commit to dmatetelki/varnish-cache that referenced this pull request Mar 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants