Recent kernel causes -fPIE ASan executables to abort on x86_64

[jcowgill]

This is just a heads-up about this Linux kernel commit recently committed and pending on a number of stable queues:
torvalds/linux@eab0953

It seems to adjust move the default load address for -fPIE executables into the location ASan uses for its shadow memory map (on x86_64). This then causes ASan to abort on startup. Example error:

$ ./a.out
==5661==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING.
==5661==ASan shadow was supposed to be located in the [0x00007fff7000-0x10007fff7fff] range.
==5661==Process memory map follows:
	0x000cb5280000-0x000cb5281000	/var/tmp/a.out
	0x000cb5480000-0x000cb5481000	/var/tmp/a.out
	0x000cb5481000-0x000cb5482000	/var/tmp/a.out
	0x7f6d4f9ca000-0x7f6d4fd1c000	
	0x7f6d4fd1c000-0x7f6d4fd32000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4fd32000-0x7f6d4ff31000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4ff31000-0x7f6d4ff32000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4ff32000-0x7f6d4ff33000	/lib/x86_64-linux-gnu/libgcc_s.so.1
[...]

With ASLR enabled, you can sometimes get lucky with the load address and the program runs, but most of the time ASan aborts with this error.

Is it possible for ASan to be a bit more flexible about where it places the shadow map on startup to fix this?

[kcc]

It is possible at the code size and execution time cost, which we are not willing to pay.
Any chance to get the kernel to cooperate?

[kcc]

This would not be the first time when the kernel change breaks the sanitizers.
The last significant one was by H.J. Lu when he changed the based from 0x7.... to 0x555....
It caused lots of trouble for us in msan and tsan.

What we really need here is to tell at link time where the shadow is.
AFAICT, there is no such capability currently.

[rnk]

I always wondered if it would be possible to express the shadow mapping as an ELF program header. That would be the ultimate way to communicate shadow memory needs to the kernel.

[jcowgill]

I'm not sure - I'm just a user who happened to stumble across the bug. You might be able to get them to change where the executable gets mapped, but they could argue that PIE executables should be prepared to be loaded at any address.

What we really need here is to tell at link time where the shadow is.

I don't see how that is possible with PIE / ASLR. The entire point is that you don't know where the executable will be loaded, so you can't know what bits of memory will be free until runtime.

[kcc]

@dvyukov @xairy @ramosian-glider FYI

[pcc]

We could have a program header that means "please reserve the first N bytes of the address space for the application". Then the kernel can use that as a minimum for ELF_ET_DYN_BASE.

[kcc]

@dvyukov can you confirm that the fresh kernel breaks the sanitizers?

[bennofs]

I think I am hitting this bug:

$  ./loadaddr 
==16572==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING.
==16572==ASan shadow was supposed to be located in the [0x00007fff7000-0x10007fff7fff] range.
==16572==Process memory map follows:
	0x04daa6e91000-0x04daa6fc6000	/tmp/loadaddr
	0x04daa71c6000-0x04daa71c7000	/tmp/loadaddr
	0x04daa71c7000-0x04daa71ca000	/tmp/loadaddr
	0x04daa71ca000-0x04daa7e2f000	
	0x7b742c072000-0x7b742c3c4000	
	0x7b742c3c4000-0x7b742c559000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c559000-0x7b742c759000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c759000-0x7b742c75d000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c75d000-0x7b742c75f000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c75f000-0x7b742c763000	
	0x7b742c763000-0x7b742c779000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c779000-0x7b742c978000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c978000-0x7b742c979000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c979000-0x7b742c97c000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742c97c000-0x7b742cb7b000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7b000-0x7b742cb7c000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7c000-0x7b742cb7d000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7d000-0x7b742cc8e000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742cc8e000-0x7b742ce8e000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce8e000-0x7b742ce8f000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce8f000-0x7b742ce90000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce90000-0x7b742ce97000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742ce97000-0x7b742d096000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d096000-0x7b742d097000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d097000-0x7b742d098000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d098000-0x7b742d0b1000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d0b1000-0x7b742d2b0000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b0000-0x7b742d2b1000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b1000-0x7b742d2b2000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b2000-0x7b742d2b6000	
	0x7b742d2b6000-0x7b742d2d9000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4a8000-0x7b742d4bc000	
	0x7b742d4c0000-0x7b742d4d9000	
	0x7b742d4d9000-0x7b742d4da000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4da000-0x7b742d4db000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4db000-0x7b742d4dc000	
	0x7fff06e24000-0x7fff06e46000	[stack]
	0x7fff06ef0000-0x7fff06ef2000	[vvar]
	0x7fff06ef2000-0x7fff06ef4000	[vdso]
	0xffffffffff600000-0xffffffffff601000	[vsyscall]
==16572==End of process memory map.
c-cube:/tmp uname -a
Linux c-cube 4.9.39 #1-NixOS SMP Fri Jul 21 05:42:36 UTC 2017 x86_64 GNU/Linux

(Just compiled a trivial Hello world with -fsanitize=address)

[bennofs]

A possible workaround seems to be the following:

$  .../ld-2.25.so ./loadaddr

That way, loadaddr will be loaded by ld.so, which uses mmap so loadaddr ends up in the mmap region which is way higher than the PIE base.

(Yes, my ld.so is in weird path, that's just NixOS things :)

[FSMaxB]

I independently bisected this in the kernel and opened a bug there: https://bugzilla.kernel.org/show_bug.cgi?id=196537 but didn't have a lot of knowledge about the underlying issues.

[richfelker]

Bringing this over from twitter (https://twitter.com/kayseesee/status/894594085608013825), my basic view is that this is a bug in the ASAN library code. Assuming you can use a particular virtual address range is not valid (it could already be in use for some reason, as you're now seeing), and even if it were valid, it's not safe for something that can be used in deployment; it exposes potentially sensitive information at an attacker-known address. ASAN simply needs to pay the cost of using a variable address chosen at runtime.

[kcc]

@richfelker ASAN has been using fixed addresses since 2011.
I know kernel does not guarantee anything like this, but it worked, and it provided performance and code size benefits over using a dynamic shadow base (which we also have now, as an option, off by default on linux)

ASAN simply needs to pay the cost of using a variable address chosen at runtime.

That's one way to look at it. But a much better resolution would be to have a kernel<=>userspace interface that allows to use a fixed address. And in the meantime, revert the change that broke ASAN.

safe for something that can be used in deployment

If you want to discuss this topic, please open a separate issue, let's not mix too many things in a single place.

[richfelker]

Like I said on on the initial Twitter thread, I don't think I have much of value to say beyond "I think what you're doing is badly wrong" and "it happened to work before is not a good argument to do it (or for changing the kernel)". If we disagree then we disagree...

[FSMaxB]

@kcc: You mentioned a dynamic shadow base. Could you please elaborate on that.

Is that available in the current stable release of LLVM? And if yes, can you point me to some documentation please.

I think that information would be useful for downstream projects that find the runtime overhead of a dynamic shadow base is acceptable.

[bennofs]

And in the meantime, revert the change that broke ASAN.

@kcc I don't think this is good advice. Pretty sure that the change fixes some security issue, so you shouldn't revert that.

[richfelker]

I agree strongly with @bennofs. Address assignment/ASLR for production systems should not be tiptoeing around (and possibly impacting security) for the sake of a tool that's only suitable in debugging situations and not production. I'd like ASAN to be usable in production (which is why I mentioned that above) but at present it's not.

[kcc]

One more discussion thread is here: http://marc.info/?t=149973272100048&r=1&w=2

[kcc]

@kcc: You mentioned a dynamic shadow base. Could you please elaborate on that.

In clang there is -mllvm -asan-force-dynamic-shadow=1, which is the default on Windows.
I don't think this has been implemented in GCC.
This is currently an implementation detail (on windows), not documented.

should not be tiptoeing around

All these arguments are perfectly valid, but who is going to pay for the increased CPU usage and code size? Or, if we end up supporting both configurations on linux (dynamic and static) who is going to pay for the extra maintenance overhead?

We really need to come up with a solution where the application requests a fixed address range at startup and the kernel can't refuse.

[FSMaxB]

@kcc: Forcing the dynamic shadow doesn't work on my system! (Archlinux x86_64 with clang 4.0.1)

[kcc]

@FSMaxB please open a separate bug with details.
But please note: this flag is not officially supported.

[richfelker]

Requesting a fixed address range at startup is non-PIE. Normal non-PIE ELF already has a way to do that: PT_LOAD segments (e.g. with PROT_NONE or just BSS you can MAP_FIXED over later). The whole point of an executable being PIE is that it doesn't demand specific addresses.

Being that current kernels don't, and future kernels probably won't, support the invalid usage of assuming a particular fixed address range is free, the fixed address mode should just be removed and dynamic always used. This will simplify the amount of code that needs to be maintained anyway (since Windows already needs dynamic). Performance is not likely to be significantly worse, but ASAN already performs badly and is intended and understood as a costly (but less so than some other approaches) tool for debugging (and possibly in the future, for hardening).

[kcc]

Asan's shadow being at a fixed offset does not really contradict PIE -- the rest of the addresses could be anywhere they want to (except for the shadow region).

BTW, I am trying to get the fresh perf numbers on spec for static vs dynamic shadow.

[richfelker]

The view I'm putting forward, which you're free to disagree with but I think is worthwhile, is that the definition of PIE is "no fixed mappings", not "some non-fixed mappings". In this definition, PIE ELF programs can even be loaded in rather esoteric environments like a shared address space (multiple programs in the same process) or a nommu system (where all processes share an address space). There are very good reasons to consider any fixed mappings a design bug; in places where they've been used recently, they've repeatedly come back to bite the designers and users. The Linux/glibc x86_64 "vsyscall" mess, ARM kuserhelper page, etc. come to mind.

[richfelker]

BTW my view of these matters is somewhat broader than "Linux" because I'm thinking of/interested in the usage case of non-Linux implementations loading and executing programs using the Linux user-kernel ABI. This sort of generality is part of why I disagree with the view that the kernel is obligated to lay out memory the same way past versions did.

[yugr]

@kcc

BTW, I am trying to get the fresh perf numbers on spec for static vs dynamic shadow.

May make sense to measure sanitized DSOs (where __asan_shadow_memory_dynamic_address is GOT-relocated), rather than sanitized executables.

[yugr]

@richfelker

I'd like ASAN to be usable in production (which is why I mentioned that above) but at present it's not.

Relevant discussion in oss-security

[kcc]

I've done an overnight run of SPEC2006 on my machine.
The results are surprisingly close.
But the run-to-run variation is too high, I'll need to find a less noisy machine.

                                   static              dynamic
       400.perlbench,      1605.00,      1647.00,         1.03   << dynamic is 3% slower
           401.bzip2,       779.00,       797.00,         1.02
             403.gcc,       660.00,       686.00,         1.04
             429.mcf,       593.00,       503.00,         0.85   << very noisy test
           445.gobmk,       960.00,       956.00,         1.00
           456.hmmer,       809.00,       812.00,         1.00
           458.sjeng,      1214.00,      1227.00,         1.01
      462.libquantum,       435.00,       442.00,         1.02
         464.h264ref,      1193.00,      1207.00,         1.01
         471.omnetpp,       881.00,       904.00,         1.03
           473.astar,       704.00,       672.00,         0.95  << dynamic is 5% faster!
       483.xalancbmk,      1252.00,      1216.00,         0.97
            433.milc,       860.00,       837.00,         0.97
            444.namd,       583.00,       590.00,         1.01
          447.dealII,      1659.00,      1627.00,         0.98
          450.soplex,       454.00,       476.00,         1.05
          453.povray,       648.00,       630.00,         0.97
             470.lbm,       478.00,       460.00,         0.96
         482.sphinx3,       811.00,       798.00,         0.98

I was also surprised to see that the code size with dynamic shadow is actually better (~0.3%).
Well, looking at the objdump it makes sense:

Dynamic:

 9a8a66:       80 3c 01 00             cmpb   $0x0,(%rcx,%rax,1)

Static:

  41fd36:       80 b8 00 80 ff 7f 00    cmpb   $0x0,0x7fff8000(%rax)

Next steps:

• find a proper noise-free machine for benchmarking
• check what happens with PIC/PIE, where loading the shadow base is more expensive
• check what's going on on ARM (I'll certainly need help with that)

[kcc]

The difference between regular executables and PIE:
Regular:

  4e7f74:	4c 8b 35 9d 2c 44 00 	mov    0x442c9d(%rip),%r14        # 92ac18 <__asan_shadow_memory_dynamic_address>

PIE (or -shared-libasan):

   e9504:	48 8d 05 0d 27 44 00 	lea    0x44270d(%rip),%rax        # 52bc18 <__asan_shadow_memory_dynamic_address>
   e950b:	4c 8b 30             	mov    (%rax),%r14

[eugenis]

Btw, ASan on 32-bit Android maps shadow at 0000 0000 .. 2000 0000, because all executables are PIE, and it is slightly faster that way (and requires less code). This is now broken.

[eugenis]

If we care about ELF + dynamic shadow base, we should duplicate the shadow base global into every DSO. We could add a hidden visibility comdat global with the shadow base to every object file and let the linker merge them. A high priority initializer would set it. This is similar to what we do on Windows.

This will not always work. If library A depends on library B, then a constructor of B may call A before A's constructors have ran.

[morehouse]

The kernel commit was ultimately reverted. Do we want to keep this issue open?

[eugenis]

I don't think it was reverted.

[eugenis]

Oh, I think it was reverted in Ubuntu kernel, but not in upstream.

[CLanguagePurist]

I am writing this for everyone who are trying to find a solution to the problem of running sanitizer on Linux and arrive at this thread from googling. As you might infers from the problem described in this thread, you have to disable ASLR on Linux via "nokaslr" option to be able to run sanitizer, but that put you at a potential security risk, so what I would recommends is to do the followings:

1. Create a virtual machine
2. Install Linux Distro of your choice
3. Ensure that Distro does not use hardened linux kernel variant
4. Configure grub.cfg or whatever boot configuration to include "nokaslr" option at the end of kernel line in the VM
5. Compile your binary with "-fsanitize=address" for both compile arguments and linker arguments
6. Reboot VM and simply copy your binary over to VM and run it to view sanitizer output

[rnk]

If we need to fix this, I think the best solution would be to use the dynamic shadow offset feature (-mllvm -asan-force-dynamic-shadow=1) already used on other OSs. My understanding is that the majority of non-Linux platforms (Windows, Android, Mac, iOS) use a dynamic shadow memory base address.

@kcc had concerns in 2017 about the performance of this change. He ran some benchmarks in this comment, and the results were in the noise. If someone can produce new results on a less noisy machine, I don't think there are any other objections. We can make the change and fix this issue for good.

Maybe this interacts with the new ASan codegen that @kda added, I'm not sure.

[CLanguagePurist]

I don't know if they will ever come around to fixing this issue since this have been around for 5 years, I submitted a workaround for this until then.

[thurstond]

Oh, I think it was reverted in Ubuntu kernel, but not in upstream.

It looks like it was reverted upstream in August 2017: torvalds/linux@c715b72

There was then a minor fix in November 2017 for 5-level-paging (torvalds/linux@be739f4), but it has no impact on this issue; that's the last time ELF_ET_DYN_BASE was modified for x64.

This means there is only a very narrow time window from when the breaking change was made (July 2017) and reverted (August 2017); any kernel outside of that 5-week period should be compatible with ASan.

[anonymouspage]

Since this is still open, I thought I'd comment that I ran into this today on Ubuntu 22.04, kernel 6.2.0-34-generic (a little out of date).

[thurstond]

Since this is still open, I thought I'd comment that I ran into this today on Ubuntu 22.04, kernel 6.2.0-34-generic (a little out of date).

Perhaps your Ubuntu installation has an aggressive address-space layout randomization (ASLR) setting. Could you please run:

sudo cat /proc/sys/vm/mmap_rnd_bits

and report what it prints?

N.B. the ASan layout was slightly tweaked in April (https://reviews.llvm.org/D148280) - perhaps just missing the release date of Ubuntu 22.04 - which increases ASan's compatibility with ASLR.

[anonymouspage]

and report what it prints?

[thurstond]

and report what it prints?

Thanks! That's weird, even 2022 ASan should not have any problems with 28-bit ASLR.

Could you please provide an example program and command line where you encounter the issue?

FWIW I spun up an Ubuntu 22.04 VM with a toy example, and it worked fine:

thurston@thurston-del-test:~$ cat /tmp/hello.c
#include <stdio.h>
#include <malloc.h>


int main (int argc, char* argv[]) {
    char* a = malloc (10);
    free (a);
    printf ("%d\n", a[0]);
    return 0;
}
thurston@thurston-del-test:~$ clang -fsanitize=address /tmp/hello.c -o /tmp/hello -fPIE
thurston@thurston-del-test:~$ file /tmp/hello
/tmp/hello: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=e7a08a803ea6092932b31ae7fb02f1ce213d212a, for GNU/Linux 3.2.0, not stripped
thurston@thurston-del-test:~$ /tmp/hello
=================================================================
==6604==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010 at pc 0x5558f1c98f06 bp 0x7ffe3ed7f0d0 sp 0x7ffe3ed7f0c8
READ of size 1 at 0x602000000010 thread T0
    #0 0x5558f1c98f05 in main (/tmp/hello+0xdbf05) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)
    #1 0x7f1dbf629d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)
    #2 0x7f1dbf629e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)
    #3 0x5558f1bdb304 in _start (/tmp/hello+0x1e304) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)

0x602000000010 is located 0 bytes inside of 10-byte region [0x602000000010,0x60200000001a)
freed by thread T0 here:
    #0 0x5558f1c5dea2 in free (/tmp/hello+0xa0ea2) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)
    #1 0x5558f1c98ecc in main (/tmp/hello+0xdbecc) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)
    #2 0x7f1dbf629d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)

previously allocated by thread T0 here:
    #0 0x5558f1c5e14e in __interceptor_malloc (/tmp/hello+0xa114e) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)
    #1 0x5558f1c98ebf in main (/tmp/hello+0xdbebf) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a)
    #2 0x7f1dbf629d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)

SUMMARY: AddressSanitizer: heap-use-after-free (/tmp/hello+0xdbf05) (BuildId: e7a08a803ea6092932b31ae7fb02f1ce213d212a) in main
Shadow bytes around the buggy address:
  0x0c047fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c047fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c047fff8000: fa fa[fd]fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==6604==ABORTING
thurston@thurston-del-test:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
thurston@thurston-del-test:~$ uname -a
Linux thurston-del-test 6.2.0-1019-gcp #21~22.04.1-Ubuntu SMP Thu Nov 16 18:18:34 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux