FFI gem doesn't install under Ruby 2.7.8 with Address Sanitizer enabled.

[JasonLunn]

I'm troubleshooting a flaky test where I suspect a memory corruption issue.

I'm trying to in use Address Sanitizer (ASAN) to identify the problem. I able to successfully install the ruby interpreter with ASAN enabled using:

ruby-install ruby 2.7.8 --rubies-dir ~/.rvm/rubies -- CFLAGS="-g -O0 -fsanitize=address"

However, when I try to install the FFI gem on that ruby, ASAN blows up. This doesn't happen when I install other gems:

gem install ffi -- --with-cflags=\"-g -O0 -fsanitize=address\" results in:

==14576==ERROR: AddressSanitizer failed to deallocate 0x8000 (32768) bytes at address 0x62d0004ec400
AddressSanitizer: CHECK failed: sanitizer_posix.cpp:61 "(("unable to unmap" && 0)) != (0)" (0x0, 0x0) (tid=2102857)
    <empty stack>

/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:189: [BUG] Illegal instruction at 0x00007ff806d3ebfd
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-darwin22]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:                    
     * ~/Library/Logs/DiagnosticReports                                     
     * /Library/Logs/DiagnosticReports                                      
   for more details.                                                        
Don't forget to include the above Crash Report log file in bug reports.     

-- Control frame information -----------------------------------------------
c:0030 p:---- s:0201 e:000200 CFUNC  :load
c:0029 p:0181 s:0196 e:000194 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:189
c:0028 p:0030 s:0182 e:000178 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:254
c:0027 p:0115 s:0172 e:000170 BLOCK  /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:224 [FINISH]
c:0026 p:---- s:0165 e:000164 CFUNC  :each
c:0025 p:0009 s:0161 e:000160 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100
c:0024 p:0014 s:0156 e:000155 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:214
c:0023 p:0093 s:0149 E:0014f0 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/index_set.rb:22 [FINISH]
c:0022 p:---- s:0141 e:000140 CFUNC  :new
c:0021 p:0043 s:0136 e:000135 RESCUE /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:90
c:0020 p:0126 s:0132 e:000131 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:86
c:0019 p:0008 s:0125 e:000123 BLOCK  /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:24 [FINISH]
c:0018 p:---- s:0120 e:000119 CFUNC  :each
c:0017 p:0009 s:0116 e:000115 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100
c:0016 p:0006 s:0111 e:000110 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:23
c:0015 p:0015 s:0107 e:000106 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:29
c:0014 p:0155 s:0101 e:000099 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/installer_set.rb:155
c:0013 p:0029 s:0090 e:000089 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/installer_set.rb:56
c:0012 p:0311 s:0081 e:000080 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/dependency_installer.rb:394
c:0011 p:0085 s:0070 e:000069 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:199
c:0010 p:0055 s:0061 e:000060 BLOCK  /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:224 [FINISH]
c:0009 p:---- s:0053 e:000052 CFUNC  :each
c:0008 p:0009 s:0049 e:000048 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:217
c:0007 p:0060 s:0044 e:000043 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:165
c:0006 p:0084 s:0039 e:000038 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command.rb:325
c:0005 p:0098 s:0031 e:000030 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command_manager.rb:178
c:0004 p:0010 s:0023 e:000022 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command_manager.rb:148
c:0003 p:0048 s:0016 e:000015 METHOD /Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/gem_runner.rb:59
c:0002 p:0127 s:0009 E:0006d8 EVAL   /Users/jatl/.rvm/rubies/ruby-2.7.8/bin/gem:21 [FINISH]
c:0001 p:0000 s:0003 E:002520 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/Users/jatl/.rvm/rubies/ruby-2.7.8/bin/gem:21:in `<main>'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/gem_runner.rb:59:in `run'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command_manager.rb:148:in `run'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command_manager.rb:178:in `process_args'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/command.rb:325:in `invoke_with_build_args'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:165:in `execute'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:217:in `install_gems'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:217:in `each'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:224:in `block in install_gems'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/commands/install_command.rb:199:in `install_gem'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/dependency_installer.rb:394:in `resolve_dependencies'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/installer_set.rb:56:in `add_always_install'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/installer_set.rb:155:in `find_all'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:29:in `find_all'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:23:in `pick_sets'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100:in `each_source'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100:in `each'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/best_set.rb:24:in `block in pick_sets'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:86:in `dependency_resolver_set'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:90:in `rescue in dependency_resolver_set'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:90:in `new'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/resolver/index_set.rb:22:in `initialize'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:214:in `available_specs'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100:in `each_source'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source_list.rb:100:in `each'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:224:in `block in available_specs'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/spec_fetcher.rb:254:in `tuples_for'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:189:in `load_specs'
/Users/jatl/.rvm/rubies/ruby-2.7.8/lib/ruby/2.7.0/rubygems/source.rb:189:in `load'

-- Machine register context ------------------------------------------------
 rax: 0x0000000000000000 rbx: 0x0000000000000003 rcx: 0x00007ff806ddc056
 rdx: 0x0000000000000000 rdi: 0x0000000000000003 rsi: 0x000070000f485db0
 rbp: 0x000070000f485de0 rsp: 0x000070000f485da0  r8: 0x0000000000000000
  r9: 0x0000000000989680 r10: 0x0000000000000000 r11: 0x0000000000000246
 r12: 0x000070000f485db0 r13: 0x000000000000003d r14: 0x000070000f485da8
 r15: 0x0000000000000006 rip: 0x00007ff806d3ebfd rfl: 0x0000000000010202

-- C level backtrace information -------------------------------------------
=================================================================
==14576==ERROR: AddressSanitizer: global-buffer-overflow on address 0x000103e304d8 at pc 0x000103bf88a8 bp 0x70000f483ab0 sp 0x70000f483aa8
READ of size 8 at 0x000103e304d8 thread T-1
    <empty stack>

0x000103e304d8 is located 8 bytes to the left of global variable 'trace' defined in 'vm_dump.c:754:18' (0x103e304e0) of size 8192
0x000103e304d8 is located 16 bytes to the right of global variable 'rb_cBacktraceLocation' defined in 'vm_backtrace.c:21:14' (0x103e304c0) of size 8
SUMMARY: AddressSanitizer: global-buffer-overflow
Shadow bytes around the buggy address:
  0x1000207c6040: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x1000207c6050: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x1000207c6060: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x1000207c6070: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x1000207c6080: f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
=>0x1000207c6090: f9 f9 f9 f9 00 f9 f9 f9 00 f9 f9[f9]00 00 00 00
  0x1000207c60a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000207c60b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000207c60c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000207c60d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000207c60e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==14576==ABORTING

[larskanis]

I also once tried GCC ASAN with ffi with similar results like you. There seems to be some incompatibility between libffi and ASAN. However I successfully used valgrind several times to debug memory issues with ffi.