Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

id_table.c: simple hash with quadratic probing #1

Merged
merged 1 commit into from
Aug 6, 2015
Merged

Conversation

funny-falcon
Copy link

  • one unusual modification: tombstone only collided items instead of all
    deleted.
    (invented by Yuriy Vostrikov aka delamonpansie)

- one unusual modification: tombstone only collided items instead of all
  deleted.
  (invented by Yuriy Vostrikov aka delamonpansie)
ko1 added a commit that referenced this pull request Aug 6, 2015
id_table.c: simple hash with quadratic probing
@ko1 ko1 merged commit 97c6293 into ko1:mtbl Aug 6, 2015
ko1 pushed a commit that referenced this pull request Mar 7, 2016
Activating the gem puts the gem on the load path, where simply requiring
the file will search every gem that's installed until it can find a gem
that contains the `did_you_mean` file.

Calling RubyGems' `require` will search each installed gem until it can
find one that contains the file it should require.  This means that the
more gems you have installed, the longer it can take to require that
gem.

To see this in action, lets compare the number of `stat` calls for a
"bare require" vs the number of `stat` calls for a require that follows
a gem activation by using these two programs:

```
[aaron@TC rubygems (master)]$ cat req_dym.rb
begin
  require 'did_you_mean'
rescue LoadError
end
[aaron@TC rubygems (master)]$ cat gem_dym.rb
begin
  gem 'did_you_mean'
  require 'did_you_mean'
rescue Gem::LoadError, LoadError
end
```

The first program just requires the `did_you_mean` gem, where the second
one activates the gem, then requires it.  We can count the number of
`stat` calls using `dtrace`:

```
[aaron@TC rubygems (master)]$ ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]
[aaron@TC rubygems (master)]$ sudo dtrace -q -n 'syscall::stat*:entry { printf("%s\n", copyinstr(arg0)); }' -c`rbenv which ruby`" --disable-did_you_mean req_dym.rb" | wc -l

dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
     283
[aaron@TC rubygems (master)]$ sudo dtrace -q -n 'syscall::stat*:entry { printf("%s\n", copyinstr(arg0)); }' -c`rbenv which ruby`" --disable-did_you_mean gem_dym.rb" | wc -l

dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
      13
```

The "bare require" version does over 10x the number of stat calls
compared to the "gem, then require" version.  Of course the number for
the first one depends on the number of gems you have installed that sort
before the `did_you_mean` gem.

Lets also look at trunk Ruby:

```
[aaron@TC rubygems (master)]$ ruby -v
ruby 2.4.0dev (2016-02-25 trunk 53940) [x86_64-darwin15]
[aaron@TC rubygems (master)]$ sudo dtrace -q -n 'syscall::stat*:entry { printf("%s\n", copyinstr(arg0)); }' -c`rbenv which ruby`" --disable-did_you_mean req_dym.rb" | wc -l

dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
    2325
[aaron@TC rubygems (master)]$ sudo dtrace -q -n 'syscall::stat*:entry { printf("%s\n", copyinstr(arg0)); }' -c`rbenv which ruby`" --disable-did_you_mean gem_dym.rb" | wc -l

dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
dtrace: error on enabled probe ID 3 (ID 826: syscall::stat64:entry): invalid user access in action #1 at DIF offset 24
     685
```

This change will reduce the number of `stat` calls on trunk Ruby too,
but since this installation doesn't have the `did_you_mean` gem,
RubyGems is still reading every gem spec file so that it can raise a
`Gem::LoadError` exception with a nice error message.  If we can modify
RubyGems a little, it may be possible to drop the number of stat calls
even on a Ruby installation that doesn't have the `did_you_mean` gem.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ko1 pushed a commit that referenced this pull request Oct 30, 2018
This funcion can be called from inside of rb_ast_new().
Should add appropriate case branches.

(lldb) run
Process 9135 launched: './miniruby' (x86_64)
Process 9135 stopped
* thread #1: tid = 0xdf36b, 0x00000001000ca4f9 miniruby`rb_raw_obj_info(buff="0x000000010205d158 [0    ] T_IMEMO", buff_size=256, obj=4328903000) + 2361 at gc.c:9617, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x00000001000ca4f9 miniruby`rb_raw_obj_info(buff="0x000000010205d158 [0    ] T_IMEMO", buff_size=256, obj=4328903000) + 2361 at gc.c:9617
   9614                 IMEMO_NAME(iseq);
   9615                 IMEMO_NAME(tmpbuf);
   9616 #undef IMEMO_NAME
-> 9617               default: UNREACHABLE;
   9618             }
   9619             snprintf(buff, buff_size, "%s %s", buff, imemo_name);
   9620
(lldb) bt
* thread #1: tid = 0xdf36b, 0x00000001000ca4f9 miniruby`rb_raw_obj_info(buff="0x000000010205d158 [0    ] T_IMEMO", buff_size=256, obj=4328903000) + 2361 at gc.c:9617, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
  * frame #0: 0x00000001000ca4f9 miniruby`rb_raw_obj_info(buff="0x000000010205d158 [0    ] T_IMEMO", buff_size=256, obj=4328903000) + 2361 at gc.c:9617
    frame #1: 0x00000001000c433f miniruby`obj_info(obj=4328903000) + 95 at gc.c:9671
    frame #2: 0x00000001000ce2ac miniruby`newobj_init(klass=4302478608, flags=36890, v1=0, v2=0, v3=0, wb_protected=1, objspace=0x0000000101800410, obj=4328903000) + 444 at gc.c:1882
    frame #3: 0x00000001000c0a49 miniruby`newobj_of(klass=4302478608, flags=36890, v1=0, v2=0, v3=0, wb_protected=1) + 217 at gc.c:1968
    frame #4: 0x00000001000c0bcb miniruby`rb_imemo_new(type=imemo_ast, v1=0, v2=0, v3=0, v0=4302478608) + 75 at gc.c:2017
    frame #5: 0x0000000100148f2a miniruby`rb_ast_new + 58 at node.c:1118
    frame #6: 0x000000010018d9e2 miniruby`yycompile(vparser=4328903720, p=0x0000000100729670, fname=4328903160, line=1) + 98 at parse.y:4925
    frame #7: 0x000000010018d66f miniruby`parser_compile_string(vparser=4328903720, fname=4328903160, s=4328904440, line=1) + 143 at parse.y:4995
    frame #8: 0x000000010018d768 miniruby`rb_parser_compile_string_path(vparser=4328903720, f=4328903160, s=4328904440, line=1) + 56 at parse.y:5015
    frame #9: 0x000000010018d71e miniruby`rb_parser_compile_string(vparser=4328903720, f="-e", s=4328904440, line=1) + 62 at parse.y:5008
    frame #10: 0x00000001002130d5 miniruby`process_options(argc=0, argv=0x00007fff5fbfdc08, opt=0x00007fff5fbfd9e8) + 3477 at ruby.c:1754
    frame #11: 0x00000001002122dd miniruby`ruby_process_options(argc=2, argv=0x00007fff5fbfdbf8) + 285 at ruby.c:2332
    frame #12: 0x00000001000aa966 miniruby`ruby_options(argc=2, argv=0x00007fff5fbfdbf8) + 262 at eval.c:118
    frame #13: 0x0000000100000ed4 miniruby`main(argc=2, argv=0x00007fff5fbfdbf8) + 116 at main.c:42
    frame #14: 0x00007fff933845ad libdyld.dylib`start + 1
    frame #15: 0x00007fff933845ad libdyld.dylib`start + 1
(lldb)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ko1 pushed a commit that referenced this pull request Jan 4, 2019
(lldb) target create "./miniruby"
Current executable set to './miniruby' (x86_64).
(lldb) settings set -- target.run-args  "-e0"
(lldb) run
Process 97005 launched: './miniruby' (x86_64)
./miniruby(rb_print_backtrace+0x15) [0x10024f7d5] vm_dump.c:715
./miniruby(rb_vm_get_sourceline+0x85) [0x10024c4f5] vm_backtrace.c:43
./miniruby(rb_vm_make_binding+0x146) [0x100236976] vm.c:941
./miniruby(Init_VM+0x592) [0x100249f02] vm.c:3091
./miniruby(rb_call_inits+0xc2) [0x1000c5a72] inits.c:58
./miniruby(ruby_setup+0xcb) [0x100098c6b] eval.c:74
./miniruby(ruby_init+0x9) [0x100098c99] eval.c:91
./miniruby(main+0x4d) [0x10025ddbd] addr2line.c:246
Process 97005 stopped
* thread #1: tid = 0x639bb, 0x000000010024c4f5 miniruby`rb_vm_get_sourceline(cfp=<unavailable>) + 133 at vm_backtrace.c:44, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x000000010024c4f5 miniruby`rb_vm_get_sourceline(cfp=<unavailable>) + 133 at vm_backtrace.c:44
   41       else {
   42           /* SDR() is not possible; that causes infinite loop. */
   43           rb_print_backtrace();
-> 44           __builtin_trap();
   45       }
   46   #endif
   47       return rb_iseq_line_no(iseq, pos);
(lldb)



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65598 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ko1 pushed a commit that referenced this pull request Jan 4, 2019
This is rather nitpicking but at the moment the process terminates,
heap_pages_final_slots overflows.

(lldb) bt
* thread #1: tid = 0xc0903, 0x00000001002b3bf7 miniruby`finalize_list(objspace=0x0000000101c09240, zombie=4329149840) + 999 at gc.c:2946, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
  * frame #0: 0x00000001002b3bf7 miniruby`finalize_list(objspace=0x0000000101c09240, zombie=4329149840) + 999 at gc.c:2946
    frame #1: 0x000000010026a69e miniruby`rb_objspace_call_finalizer(objspace=0x0000000101c09240) + 7118 at gc.c:3092
    frame #2: 0x0000000100268ac5 miniruby`rb_gc_call_finalizer_at_exit + 149 at gc.c:3015
    frame #3: 0x00000001002272bc miniruby`ruby_finalize_1 + 156 at eval.c:146
    frame #4: 0x00000001002282b6 miniruby`ruby_cleanup(ex=0) + 4070 at eval.c:238
    frame #5: 0x0000000100228785 miniruby`ruby_run_node(n=0x0000000102060ad8) + 85 at eval.c:317
    frame #6: 0x0000000100000b9c miniruby`main(argc=2, argv=0x00007fff5fbfdc38) + 124 at main.c:42
    frame #7: 0x00007fff9966a5ad libdyld.dylib`start + 1
    frame #8: 0x00007fff9966a5ad libdyld.dylib`start + 1
(lldb) p objspace->heap_pages
(rb_objspace::(anonymous struct)) $0 = {
  sorted = 0x0000000101c070b0
  allocated_pages = 24
  allocatable_pages = 0
  sorted_length = 24
  range = {
    [0] = 0x0000000102020028
    [1] = 0x00000001020dbfd0
  }
  freeable_pages = 0
  final_slots = 0
  deferred_final = 4329149840
}
(lldb)




git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65633 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
ko1 pushed a commit that referenced this pull request Jun 6, 2019
autoload_reset() can read this state.result.  Because autoload_reset
is a function passed to rb_ensure, there is a chance when an
execption raises before actually filling this memory region.

test/ruby/test_defined.rb:test_autoload_noload is one of such case.

Found using memory sanitizer.

==54014==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x557a683f3e5a in autoload_reset variable.c:2372:9
    #1 0x557a6707a93b in rb_ensure eval.c:1084:5
    #2 0x557a683efbf5 in rb_autoload_load variable.c:2475:14
    #3 0x557a685fc460 in vm_get_ev_const vm_insnhelper.c:938:4
    #4 0x557a68448e0a in vm_exec_core insns.def:267:11
ko1 pushed a commit that referenced this pull request Jun 6, 2019
memo.float_value might change inside of hash_sum.  In case it
flipped from false to true there, and the calculated sum is Inf,
memo.c might not be initialized at all.  This is bad.

Found using memory sanitizer:

==55293==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55dfb8d6c529 in rb_float_new_inline internal.h:1814:53
    #1 0x55dfb8d1b30c in enum_sum enum.c:4017:18
    #2 0x55dfb86d75ad in call_cfunc_m1 vm_insnhelper.c:2041:12
    #3 0x55dfb864b141 in vm_call_cfunc_with_frame vm_insnhelper.c:2207:11
    #4 0x55dfb85e843d in vm_call_cfunc vm_insnhelper.c:2225:12
    #5 0x55dfb85e08f3 in vm_call_method_each_type vm_insnhelper.c:2560:9
    #6 0x55dfb85de9c7 in vm_call_method vm_insnhelper.c:2686:13
    #7 0x55dfb849eac6 in vm_call_general vm_insnhelper.c:2730:12
    #8 0x55dfb8686103 in vm_sendish vm_insnhelper.c:3623:11
    #9 0x55dfb84dc29e in vm_exec_core insns.def:789:11
ko1 pushed a commit that referenced this pull request Jun 6, 2019
ec->cfp->iseq might not exist at the very beginning of a thread.

=================================================================
==82954==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fc86f334810 at pc 0x55ceaf013125 bp 0x7ffe2eddbbf0 sp 0x7ffe2eddbbe8
READ of size 8 at 0x7fc86f334810 thread T0
    #0 0x55ceaf013124 in vm_check_canary vm_insnhelper.c:217:24
    #1 0x55ceaefb4796 in vm_push_frame vm_insnhelper.c:276:5
    #2 0x55ceaf0124bd in th_init vm.c:2661:5
    #3 0x55ceaf00d5eb in ruby_thread_init vm.c:2690:5
    #4 0x55ceaf00d4b1 in rb_thread_alloc vm.c:2703:5
    #5 0x55ceaef0038b in thread_s_new thread.c:872:20
    #6 0x55ceaf04d8c1 in call_cfunc_m1 vm_insnhelper.c:2041:12
    #7 0x55ceaf03118d in vm_call_cfunc_with_frame vm_insnhelper.c:2207:11
    #8 0x55ceaf017985 in vm_call_cfunc vm_insnhelper.c:2225:12
    #9 0x55ceaf01548b in vm_call_method_each_type vm_insnhelper.c:2560:9
    #10 0x55ceaf014c96 in vm_call_method vm_insnhelper.c:2686:13
    #11 0x55ceaefb5de4 in vm_call_general vm_insnhelper.c:2730:12
    #12 0x55ceaf03c868 in vm_sendish vm_insnhelper.c:3623:11
    #13 0x55ceaefc95bb in vm_exec_core insns.def:771:11
    #14 0x55ceaf006700 in rb_vm_exec vm.c:1892:22
    #15 0x55ceaf00acbf in rb_iseq_eval_main vm.c:2151:11
    #16 0x55ceaea250ca in ruby_exec_internal eval.c:262:2
    ruby#17 0x55ceaea2498b in ruby_exec_node eval.c:326:12
    ruby#18 0x55ceaea247d0 in ruby_run_node eval.c:318:25
    ruby#19 0x55ceae88c486 in main main.c:42:9
    #20 0x7fc874330b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    ruby#21 0x55ceae7e5289 in _start (miniruby+0x15f289)

0x7fc86f334810 is located 16 bytes to the right of 1048576-byte region [0x7fc86f234800,0x7fc86f334800)
allocated by thread T0 here:
    #0 0x55ceae85d56d in malloc (miniruby+0x1d756d)
    #1 0x55ceaea71d12 in objspace_xmalloc0 gc.c:9416:5
    #2 0x55ceaea71cd2 in ruby_xmalloc2_body gc.c:9623:12
    #3 0x55ceaea7d09c in ruby_xmalloc2 gc.c:11479:12
    #4 0x55ceaf00c3b7 in rb_thread_recycle_stack vm.c:2462:12
    #5 0x55ceaf012256 in th_init vm.c:2656:29
    #6 0x55ceaf00d5eb in ruby_thread_init vm.c:2690:5
    #7 0x55ceaf00d4b1 in rb_thread_alloc vm.c:2703:5
    #8 0x55ceaef0038b in thread_s_new thread.c:872:20
    #9 0x55ceaf04d8c1 in call_cfunc_m1 vm_insnhelper.c:2041:12
    #10 0x55ceaf03118d in vm_call_cfunc_with_frame vm_insnhelper.c:2207:11
    #11 0x55ceaf017985 in vm_call_cfunc vm_insnhelper.c:2225:12
    #12 0x55ceaf01548b in vm_call_method_each_type vm_insnhelper.c:2560:9
    #13 0x55ceaf014c96 in vm_call_method vm_insnhelper.c:2686:13
    #14 0x55ceaefb5de4 in vm_call_general vm_insnhelper.c:2730:12
    #15 0x55ceaf03c868 in vm_sendish vm_insnhelper.c:3623:11
    #16 0x55ceaefc95bb in vm_exec_core insns.def:771:11
    ruby#17 0x55ceaf006700 in rb_vm_exec vm.c:1892:22
    ruby#18 0x55ceaf00acbf in rb_iseq_eval_main vm.c:2151:11
    ruby#19 0x55ceaea250ca in ruby_exec_internal eval.c:262:2
    #20 0x55ceaea2498b in ruby_exec_node eval.c:326:12
    ruby#21 0x55ceaea247d0 in ruby_run_node eval.c:318:25
    ruby#22 0x55ceae88c486 in main main.c:42:9
    ruby#23 0x7fc874330b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: heap-buffer-overflow vm_insnhelper.c:217:24 in vm_check_canary
Shadow bytes around the buggy address:
  0x0ff98de5e8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff98de5e8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff98de5e8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff98de5e8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff98de5e8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ff98de5e900: fa fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff98de5e910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff98de5e920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff98de5e930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff98de5e940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0ff98de5e950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==82954==ABORTING
ko1 pushed a commit that referenced this pull request Jun 6, 2019
rb_io_fptr_finalize_internal frees the memory region.

=================================================================
==85264==ERROR: AddressSanitizer: heap-use-after-free on address 0x610000000d8c at pc 0x5608e38077f7 bp 0x7ffee12d5440 sp 0x7ffee12d5438
READ of size 4 at 0x610000000d8c thread T0
    #0 0x5608e38077f6 in rb_io_memsize io.c:4749:24
    #1 0x5608e37a0481 in obj_memsize_of gc.c:3547:14
    #2 0x5608e37a4f30 in check_rvalue_consistency gc.c:1107:2
    #3 0x5608e37a2624 in RVALUE_OLD_P gc.c:1218:5
    #4 0x5608e37a5bae in rb_gc_force_recycle gc.c:6652:18
    #5 0x5608e38191f9 in rb_f_backquote io.c:9021:5
    #6 0x5608e3d8aa14 in call_cfunc_1 vm_insnhelper.c:2058:12
    #7 0x5608e3d6e23d in vm_call_cfunc_with_frame vm_insnhelper.c:2211:11
    #8 0x5608e3d54a35 in vm_call_cfunc vm_insnhelper.c:2229:12
    #9 0x5608e3d5253b in vm_call_method_each_type vm_insnhelper.c:2564:9
    #10 0x5608e3d51f50 in vm_call_method vm_insnhelper.c:2701:13
    #11 0x5608e3cf2de4 in vm_call_general vm_insnhelper.c:2734:12
    #12 0x5608e3d79918 in vm_sendish vm_insnhelper.c:3627:11
    #13 0x5608e3d06cf5 in vm_exec_core insns.def:789:11
    #14 0x5608e3d43700 in rb_vm_exec vm.c:1892:22
    #15 0x5608e3d47cbf in rb_iseq_eval_main vm.c:2151:11
    #16 0x5608e37620ca in ruby_exec_internal eval.c:262:2
    ruby#17 0x5608e376198b in ruby_exec_node eval.c:326:12
    ruby#18 0x5608e37617d0 in ruby_run_node eval.c:318:25
    ruby#19 0x5608e35c9486 in main main.c:42:9
    #20 0x7f62e9421b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    ruby#21 0x5608e3522289 in _start (miniruby+0x15f289)

0x610000000d8c is located 76 bytes inside of 192-byte region [0x610000000d40,0x610000000e00)
freed by thread T0 here:
    #0 0x5608e359a2ed in free (miniruby+0x1d72ed)
    #1 0x5608e37af421 in objspace_xfree gc.c:9591:5
    #2 0x5608e37af3da in ruby_sized_xfree gc.c:9687:2
    #3 0x5608e3799ac8 in ruby_xfree gc.c:9694:5
    #4 0x5608e380746d in rb_io_fptr_finalize_internal io.c:4728:5
    #5 0x5608e38191ed in rb_f_backquote io.c:9020:5
    #6 0x5608e3d8aa14 in call_cfunc_1 vm_insnhelper.c:2058:12
    #7 0x5608e3d6e23d in vm_call_cfunc_with_frame vm_insnhelper.c:2211:11
    #8 0x5608e3d54a35 in vm_call_cfunc vm_insnhelper.c:2229:12
    #9 0x5608e3d5253b in vm_call_method_each_type vm_insnhelper.c:2564:9
    #10 0x5608e3d51f50 in vm_call_method vm_insnhelper.c:2701:13
    #11 0x5608e3cf2de4 in vm_call_general vm_insnhelper.c:2734:12
    #12 0x5608e3d79918 in vm_sendish vm_insnhelper.c:3627:11
    #13 0x5608e3d06cf5 in vm_exec_core insns.def:789:11
    #14 0x5608e3d43700 in rb_vm_exec vm.c:1892:22
    #15 0x5608e3d47cbf in rb_iseq_eval_main vm.c:2151:11
    #16 0x5608e37620ca in ruby_exec_internal eval.c:262:2
    ruby#17 0x5608e376198b in ruby_exec_node eval.c:326:12
    ruby#18 0x5608e37617d0 in ruby_run_node eval.c:318:25
    ruby#19 0x5608e35c9486 in main main.c:42:9
    #20 0x7f62e9421b96 in __libc_start_main
/build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

previously allocated by thread T0 here:
    #0 0x5608e359a56d in malloc (miniruby+0x1d756d)
    #1 0x5608e37aed12 in objspace_xmalloc0 gc.c:9416:5
    #2 0x5608e37aebe7 in ruby_xmalloc0 gc.c:9600:12
    #3 0x5608e37aea8b in ruby_xmalloc_body gc.c:9609:12
    #4 0x5608e37a6d64 in ruby_xmalloc gc.c:11469:12
    #5 0x5608e380e4b4 in rb_io_fptr_new io.c:8040:19
    #6 0x5608e380e446 in rb_io_make_open_file io.c:8077:10
    #7 0x5608e3850ea0 in pipe_open io.c:6707:5
    #8 0x5608e384edb4 in pipe_open_s io.c:6772:12
    #9 0x5608e381910b in rb_f_backquote io.c:9014:12
    #10 0x5608e3d8aa14 in call_cfunc_1 vm_insnhelper.c:2058:12
    #11 0x5608e3d6e23d in vm_call_cfunc_with_frame vm_insnhelper.c:2211:11
    #12 0x5608e3d54a35 in vm_call_cfunc vm_insnhelper.c:2229:12
    #13 0x5608e3d5253b in vm_call_method_each_type vm_insnhelper.c:2564:9
    #14 0x5608e3d51f50 in vm_call_method vm_insnhelper.c:2701:13
    #15 0x5608e3cf2de4 in vm_call_general vm_insnhelper.c:2734:12
    #16 0x5608e3d79918 in vm_sendish vm_insnhelper.c:3627:11
    ruby#17 0x5608e3d06cf5 in vm_exec_core insns.def:789:11
    ruby#18 0x5608e3d43700 in rb_vm_exec vm.c:1892:22
    ruby#19 0x5608e3d47cbf in rb_iseq_eval_main vm.c:2151:11
    #20 0x5608e37620ca in ruby_exec_internal eval.c:262:2
    ruby#21 0x5608e376198b in ruby_exec_node eval.c:326:12
    ruby#22 0x5608e37617d0 in ruby_run_node eval.c:318:25
    ruby#23 0x5608e35c9486 in main main.c:42:9
    ruby#24 0x7f62e9421b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: heap-use-after-free io.c:4749:24 in
rb_io_memsize
Shadow bytes around the buggy address:
  0x0c207fff8160: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c207fff8170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c207fff8180: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c207fff8190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c207fff81a0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c207fff81b0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c207fff81c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c207fff81d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c207fff81e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c207fff81f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c207fff8200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==85264==ABORTING
ko1 pushed a commit that referenced this pull request Jan 28, 2020
This changeset makes no difference unless GC_DEBUG is on.  When that flag is
set, struct RVALUE is bigger than struct RObject.  We have to take care of the
additional fields.  Otherwise we get a SIGSEGV like shown below.

The way obj is initialized in this patch works for both GC_DEBUG is on and off.
See also ISO/IEC 9899:1999 section 6.7.8 paragraph ruby#21.

```
Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
62      ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:62
#1  0x00005555557dd9a7 in BSD_vfprintf (fp=0x7fffffff6be0, fmt0=0x5555558f3059 "@%s:%d", ap=0x7fffffff6dd0) at vsnprintf.c:1027
#2  0x00005555557db6f5 in ruby_do_vsnprintf (str=0x555555bfc58d <obj_info_buffers+1325> "", n=211, fmt=0x5555558f3059 "@%s:%d", ap=0x7fffffff6dd0) at sprintf.c:1022
#3  0x00005555557db909 in ruby_snprintf (str=0x555555bfc58d <obj_info_buffers+1325> "", n=211, fmt=0x5555558f3059 "@%s:%d") at sprintf.c:1040
#4  0x0000555555661ef4 in rb_raw_obj_info (buff=0x555555bfc560 <obj_info_buffers+1280> "0x0000555555d2bfa0 [0     ] T_STRING (String)", buff_size=256, obj=93825000456096) at gc.c:11449
#5  0x000055555565baaf in obj_info (obj=93825000456096) at gc.c:11612
#6  0x000055555565bae1 in rgengc_remembered (objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:6618
#7  0x0000555555666987 in newobj_init (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1, objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:2134
#8  0x0000555555666e49 in newobj_slowpath (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, objspace=0x555555c0a1c0, wb_protected=1) at gc.c:2209
#9  0x0000555555666b94 in newobj_slowpath_wb_protected (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, objspace=0x555555c0a1c0) at gc.c:2220
#10 0x000055555565751b in newobj_of (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1) at gc.c:2256
#11 0x00005555556575ca in rb_wb_protected_newobj_of (klass=93824999964192, flags=5) at gc.c:2272
#12 0x00005555557f36ea in str_alloc (klass=93824999964192) at string.c:728
#13 0x00005555557f2128 in rb_str_buf_new (capa=0) at string.c:1317
#14 0x000055555578c66d in rb_reg_preprocess (p=0x555555cc8148 "^-(.)(.+)?", end=0x555555cc8152 "", enc=0x555555cc7c80, fixed_enc=0x7fffffff74e8, err=0x7fffffff75f0 "") at re.c:2682
#15 0x000055555578ea13 in rb_reg_initialize (obj=93825000046736, s=0x555555cc8148 "^-(.)(.+)?", len=10, enc=0x555555cc7c80, options=0, err=0x7fffffff75f0 "", sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2808
#16 0x000055555578e285 in rb_reg_initialize_str (obj=93825000046736, str=93825000046904, options=0, err=0x7fffffff75f0 "", sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2869
ruby#17 0x000055555578ee02 in rb_reg_compile (str=93825000046904, options=0, sourcefile=0x555555d1a5c0 "lib/optparse.rb", sourceline=1460) at re.c:2958
ruby#18 0x0000555555748dfb in rb_parser_reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12157
ruby#19 0x00005555557581c3 in parser_reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12151
#20 0x00005555557580ac in reg_compile (p=0x555555d1f760, str=93825000046904, options=0) at parse.y:12167
ruby#21 0x0000555555746ebb in new_regexp (p=0x555555d1f760, node=0x555555dece68, options=0, loc=0x7fffffff89e8) at parse.y:10072
ruby#22 0x000055555573d1f5 in ruby_yyparse (p=0x555555d1f760) at parse.y:4395
ruby#23 0x000055555574a582 in yycompile0 (arg=93825000404832) at parse.y:5945
ruby#24 0x00005555558c6898 in rb_suppress_tracing (func=0x55555574a470 <yycompile0>, arg=93825000404832) at vm_trace.c:427
ruby#25 0x0000555555748290 in yycompile (vparser=93824999283456, p=0x555555d1f760, fname=93824999283624, line=1) at parse.y:5994
ruby#26 0x00005555557481ae in rb_parser_compile_file_path (vparser=93824999283456, fname=93824999283624, file=93824999283400, start=1) at parse.y:6098
ruby#27 0x00005555557cdd35 in load_file_internal (argp_v=140737488331760) at ruby.c:2023
ruby#28 0x00005555556438c5 in rb_ensure (b_proc=0x5555557cd610 <load_file_internal>, data1=140737488331760, e_proc=0x5555557cddd0 <restore_load_file>, data2=140737488331760) at eval.c:1128
ruby#29 0x00005555557cb68b in load_file (parser=93824999283456, fname=93824999283624, f=93824999283400, script=0, opt=0x7fffffffa468) at ruby.c:2142
ruby#30 0x00005555557cb339 in rb_parser_load_file (parser=93824999283456, fname_v=93824999283624) at ruby.c:2164
ruby#31 0x00005555556ba3e1 in load_iseq_eval (ec=0x555555c0a650, fname=93824999283624) at load.c:579
ruby#32 0x00005555556b857a in require_internal (ec=0x555555c0a650, fname=93824999284352, exception=1) at load.c:1016
ruby#33 0x00005555556b7967 in rb_require_string (fname=93824999284464) at load.c:1105
ruby#34 0x00005555556b7939 in rb_f_require (obj=93824999994824, fname=93824999284464) at load.c:811
ruby#35 0x00005555558b7ae0 in call_cfunc_1 (recv=93824999994824, argc=1, argv=0x7ffff7ecd0a8, func=0x5555556b7920 <rb_f_require>) at vm_insnhelper.c:2348
ruby#36 0x00005555558a8889 in vm_call_cfunc_with_frame (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10, empty_kw_splat=0) at vm_insnhelper.c:2513
ruby#37 0x000055555589fb5c in vm_call_cfunc (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:2538
ruby#38 0x000055555589f22e in vm_call_method_each_type (ec=0x555555c0a650, cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:2924
ruby#39 0x000055555589ef47 in vm_call_method (ec=0x555555c0a650, cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:3038
ruby#40 0x0000555555866dbd in vm_call_general (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, calling=0x7fffffffaab0, cd=0x555555d76a10) at vm_insnhelper.c:3075
ruby#41 0x00005555558ae557 in vm_sendish (ec=0x555555c0a650, reg_cfp=0x7ffff7fccfa0, cd=0x555555d76a10, block_handler=0, method_explorer=0x5555558ae5d0 <vm_search_method_wrap>) at vm_insnhelper.c:4021
ruby#42 0x000055555587745b in vm_exec_core (ec=0x555555c0a650, initial=0) at insns.def:801
ruby#43 0x0000555555899b9c in rb_vm_exec (ec=0x555555c0a650, mjit_enable_p=1) at vm.c:1907
ruby#44 0x000055555589aaf0 in rb_iseq_eval_main (iseq=0x555555c1da80) at vm.c:2166
ruby#45 0x0000555555641f0b in rb_ec_exec_node (ec=0x555555c0a650, n=0x555555c1da80) at eval.c:277
ruby#46 0x0000555555641d62 in ruby_run_node (n=0x555555c1da80) at eval.c:335
ruby#47 0x000055555557a188 in main (argc=11, argv=0x7fffffffc848) at main.c:50
(gdb) fr 7
#7  0x0000555555666987 in newobj_init (klass=93824999964192, flags=5, v1=0, v2=0, v3=0, wb_protected=1, objspace=0x555555c0a1c0, obj=93825000456096) at gc.c:2134
2134        if (rgengc_remembered(objspace, (VALUE)obj)) rb_bug("newobj: %s is remembered.", obj_info(obj));
(gdb) p ((struct RVALUE*)obj)->file
$1 = 0x65a5992b0fb25ce7 <error: Cannot access memory at address 0x65a5992b0fb25ce7>
(gdb)
```
ko1 pushed a commit that referenced this pull request Jun 25, 2020
This reverts commit 02b216e.
This reverts commit 9b8825b.

I found that combining sweep and move is not safe.  I don't think that
we can do compaction concurrently with _anything_ unless there is a read
barrier installed.

Here is a simple example.  A class object is freed, and during it's free
step, it tries to remove itself from its parent's subclass list.
However, during the sweep step, the parent class was moved and the
"currently being freed" class didn't have references updated yet.  So we
get a segv like this:

```
(lldb) bt
* thread #1, name = 'ruby', stop reason = signal SIGSEGV
  * frame #0: 0x0000560763e344cb ruby`rb_st_lookup at st.c:320:43
    frame #1: 0x0000560763e344cb ruby`rb_st_lookup(tab=0x2f7469672f6e6f72, key=3809, value=0x0000560765bf2270) at st.c:1010
    frame #2: 0x0000560763e8f16a ruby`rb_search_class_path at variable.c:99:9
    frame #3: 0x0000560763e8f141 ruby`rb_search_class_path at variable.c:145
    frame #4: 0x0000560763e8f141 ruby`rb_search_class_path(klass=94589785585880) at variable.c:191
    frame #5: 0x0000560763ec744e ruby`rb_vm_bugreport at vm_dump.c:996:17
    frame #6: 0x0000560763f5b958 ruby`rb_bug_for_fatal_signal at error.c:675:5
    frame #7: 0x0000560763e27dad ruby`sigsegv(sig=<unavailable>, info=<unavailable>, ctx=<unavailable>) at signal.c:955:5
    frame #8: 0x00007f8b891d33c0 libpthread.so.0`___lldb_unnamed_symbol1$$libpthread.so.0 + 1
    frame #9: 0x0000560763efa8bb ruby`rb_class_remove_from_super_subclasses(klass=94589790314280) at class.c:93:56
    frame #10: 0x0000560763d10cb7 ruby`gc_sweep_step at gc.c:2674:2
    frame #11: 0x0000560763d1187b ruby`gc_sweep at gc.c:4540:2
    frame #12: 0x0000560763d101f0 ruby`gc_start at gc.c:6797:6
    frame #13: 0x0000560763d15153 ruby`rb_gc_compact at gc.c:7479:12
    frame #14: 0x0000560763eb4eb8 ruby`vm_exec_core at vm_insnhelper.c:5183:13
    frame #15: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame #16: 0x0000560763eac08d ruby`rb_yield at vm.c:1132:9
    frame ruby#17: 0x0000560763edb4f2 ruby`rb_ary_collect at array.c:3186:9
    frame ruby#18: 0x0000560763e9ee15 ruby`vm_call_cfunc_with_frame at vm_insnhelper.c:2575:12
    frame ruby#19: 0x0000560763eb2e66 ruby`vm_exec_core at vm_insnhelper.c:4177:11
    frame #20: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame ruby#21: 0x0000560763eac08d ruby`rb_yield at vm.c:1132:9
    frame ruby#22: 0x0000560763edb4f2 ruby`rb_ary_collect at array.c:3186:9
    frame ruby#23: 0x0000560763e9ee15 ruby`vm_call_cfunc_with_frame at vm_insnhelper.c:2575:12
    frame ruby#24: 0x0000560763eb2e66 ruby`vm_exec_core at vm_insnhelper.c:4177:11
    frame ruby#25: 0x0000560763ea9bae ruby`rb_vm_exec at vm.c:1953:22
    frame ruby#26: 0x0000560763ceee01 ruby`rb_ec_exec_node(ec=0x0000560765afa530, n=0x0000560765b088e0) at eval.c:296:2
    frame ruby#27: 0x0000560763cf3b7b ruby`ruby_run_node(n=0x0000560765b088e0) at eval.c:354:12
    frame ruby#28: 0x0000560763cee4a3 ruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:50:9
    frame ruby#29: 0x00007f8b88e560b3 libc.so.6`__libc_start_main + 243
    frame ruby#30: 0x0000560763cee4ee ruby`_start + 46
(lldb) f 9
frame #9: 0x0000560763efa8bb ruby`rb_class_remove_from_super_subclasses(klass=94589790314280) at class.c:93:56
   90
   91  		*RCLASS_EXT(klass)->parent_subclasses = entry->next;
   92  		if (entry->next) {
-> 93  		    RCLASS_EXT(entry->next->klass)->parent_subclasses = RCLASS_EXT(klass)->parent_subclasses;
   94  		}
   95  		xfree(entry);
   96  	    }
(lldb) command script import -r misc/lldb_cruby.py
lldb scripts for ruby has been installed.
(lldb) rp entry->next->klass
(struct RMoved) $1 = (flags = 30, destination = 94589792806680, next = 94589784369160)
(lldb)
```
ko1 pushed a commit that referenced this pull request Apr 22, 2021
This commit adds a check on the ep just like in the mark function.  The
env can contain null bytes if allocation tracing is enabled.

We're seeing errors during autocompaction like this:

```
(lldb) bt 40
* thread #1, name = 'ruby', stop reason = signal SIGABRT
    frame #0: 0x00007f7d64b6018b libc.so.6`raise + 203
    frame #1: 0x00007f7d64b3f859 libc.so.6`abort + 299
    frame #2: 0x000055af5f2fefc9 ruby`die at error.c:764:5
    frame #3: 0x000055af5f2ff1ac ruby`rb_bug_for_fatal_signal(default_sighandler=0x0000000000000000, sig=11, ctx=0x000055af60bc3340, fmt="") at error.c:804:5
    frame #4: 0x000055af5f4bd08f ruby`sigsegv(sig=11, info=0x000055af60bc3470, ctx=0x000055af60bc3340) at signal.c:960:5
    frame #5: 0x00007f7d64ebe3c0 libpthread.so.0`__restore_rt
    frame #6: 0x000055af5f339b0a ruby`gc_ref_update_imemo(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9046:13
    frame #7: 0x000055af5f339172 ruby`gc_update_object_references(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9307:9
    frame #8: 0x000055af5f338e79 ruby`gc_ref_update(vstart=0x00007f7d5b510010, vend=0x00007f7d5b513ff8, stride=40, objspace=0x000055af60b2b040, page=0x000055af62577aa0) at gc.c:9452:21
    frame #9: 0x000055af5f337846 ruby`gc_update_references(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:9481:9
    frame #10: 0x000055af5f336569 ruby`gc_compact_finish(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:4840:5
    frame #11: 0x000055af5f335efb ruby`gc_page_sweep(objspace=0x000055af60b2b040, heap=0x000055af60b2b068, sweep_page=0x000055af63a1eb30) at gc.c:5046:13
    frame #12: 0x000055af5f3355c5 ruby`gc_sweep_step(objspace=0x000055af60b2b040, heap=0x000055af60b2b068) at gc.c:5214:19
    frame #13: 0x000055af5f33daf6 ruby`gc_sweep_rest(objspace=0x000055af60b2b040) at gc.c:5271:2
    frame #14: 0x000055af5f33cacd ruby`gc_sweep(objspace=0x000055af60b2b040) at gc.c:5389:2
    frame #15: 0x000055af5f33c21d ruby`gc_marks_rest(objspace=0x000055af60b2b040) at gc.c:7555:5
    frame #16: 0x000055af5f324d41 ruby`gc_rest(objspace=0x000055af60b2b040) at gc.c:8457:13
    frame ruby#17: 0x000055af5f3297d8 ruby`garbage_collect(objspace=0x000055af60b2b040, reason=45568) at gc.c:8318:9
    frame ruby#18: 0x000055af5f344ece ruby`garbage_collect_with_gvl(objspace=0x000055af60b2b040, reason=45568) at gc.c:8632:9
    frame ruby#19: 0x000055af5f344e61 ruby`objspace_malloc_gc_stress(objspace=0x000055af60b2b040) at gc.c:10592:9
    frame #20: 0x000055af5f32ced1 ruby`objspace_xmalloc0(objspace=0x000055af60b2b040, size=64) at gc.c:10767:5
    frame ruby#21: 0x000055af5f32ce11 ruby`ruby_xmalloc0(size=64) at gc.c:10988:12
    frame ruby#22: 0x000055af5f32cdac ruby`ruby_xmalloc_body(size=64) at gc.c:10997:12
    frame ruby#23: 0x000055af5f329415 ruby`ruby_xmalloc(size=64) at gc.c:12942:12
    frame ruby#24: 0x00007f7d611c4fe5 objspace.so`newobj_i(tpval=0x00007f7d5b553770, data=0x000055af639031a0) at object_tracing.c:101:35
    frame ruby#25: 0x000055af5f5b283f ruby`tp_call_trace(tpval=0x00007f7d5b553770, trace_arg=0x00007fff1016d398) at vm_trace.c:1115:2
    frame ruby#26: 0x000055af5f5b50ec ruby`exec_hooks_body(ec=0x000055af60b2b700, list=0x000055af60b2b920, trace_arg=0x00007fff1016d398) at vm_trace.c:304:3
    frame ruby#27: 0x000055af5f5b0f24 ruby`exec_hooks_unprotected(ec=0x000055af60b2b700, list=0x000055af60b2b920, trace_arg=0x00007fff1016d398) at vm_trace.c:333:5
    frame ruby#28: 0x000055af5f5b0da8 ruby`rb_exec_event_hooks(trace_arg=0x00007fff1016d398, hooks=0x000055af60b2b920, pop_p=0) at vm_trace.c:378:13
    frame ruby#29: 0x000055af5f33f8e2 ruby`rb_exec_event_hook_orig(ec=0x000055af60b2b700, hooks=0x000055af60b2b920, flag=1048576, self=0x00007f7d5b5c08c0, id=0, called_id=0, klass=0x0000000000000000, data=0x00007f7d5b513fd0, pop_p=0) at vm_core.h:1989:5
    frame ruby#30: 0x000055af5f334975 ruby`gc_event_hook_body(ec=0x000055af60b2b700, objspace=0x000055af60b2b040, event=1048576, data=0x00007f7d5b513fd0) at gc.c:2083:5
  * frame ruby#31: 0x000055af5f3342df ruby`newobj_slowpath_wb_protected [inlined] newobj_slowpath(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910, wb_protected=1) at gc.c:2284:9
    frame ruby#32: 0x000055af5f33410f ruby`newobj_slowpath_wb_protected(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910) at gc.c:2299
    frame ruby#33: 0x000055af5f333de9 ruby`newobj_of0(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, wb_protected=1, cr=0x000055af60b2b910) at gc.c:2338:11
    frame ruby#34: 0x000055af5f3227ae ruby`newobj_of(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, v1=0x000055af657d88a0, v2=0x000055af657d8890, v3=0x0000000000000000, wb_protected=1) at gc.c:2348:17
    frame ruby#35: 0x000055af5f322c5b ruby`rb_imemo_new(type=imemo_env, v1=0x000055af657d88a0, v2=0x000055af657d8890, v3=0x0000000000000000, v0=0x00007f7d5b9d19c8) at gc.c:2434:12
    frame ruby#36: 0x000055af5f5a3925 ruby`vm_env_new(env_ep=0x000055af657d88a0, env_body=0x000055af657d8890, env_size=4, iseq=0x00007f7d5b9d19c8) at vm_core.h:1363:33
    frame ruby#37: 0x000055af5f5a3808 ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fc90) at vm.c:801:11
    frame ruby#38: 0x000055af5f5a368d ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fc20) at vm.c:752:13
    frame ruby#39: 0x000055af5f5a368d ruby`vm_make_env_each(ec=0x000055af60b2b700, cfp=0x00007f7d6482fbb0) at vm.c:752:13
(lldb) f 31
frame ruby#31: 0x000055af5f3342df ruby`newobj_slowpath_wb_protected [inlined] newobj_slowpath(klass=0x00007f7d5b9d19c8, flags=0x000000000000001a, objspace=0x000055af60b2b040, cr=0x000055af60b2b910, wb_protected=1) at gc.c:2284:9
   2281	        }
   2282	        GC_ASSERT(obj != 0);
   2283	        newobj_init(klass, flags, wb_protected, objspace, obj);
-> 2284	        gc_event_hook_prep(objspace, RUBY_INTERNAL_EVENT_NEWOBJ, obj, newobj_fill(obj, 0, 0, 0));
   2285	    }
   2286	    RB_VM_LOCK_LEAVE_CR_LEV(cr, &lev);
   2287
(lldb) p obj
(VALUE) $3 = 0x00007f7d5b513fd0
(lldb) f 6
frame #6: 0x000055af5f339b0a ruby`gc_ref_update_imemo(objspace=0x000055af60b2b040, obj=0x00007f7d5b513fd0) at gc.c:9046:13
   9043	        {
   9044	            rb_env_t *env = (rb_env_t *)obj;
   9045	            TYPED_UPDATE_IF_MOVED(objspace, rb_iseq_t *, env->iseq);
-> 9046	            UPDATE_IF_MOVED(objspace, env->ep[VM_ENV_DATA_INDEX_ENV]);
   9047	            gc_update_values(objspace, (long)env->env_size, (VALUE *)env->env);
   9048	        }
   9049	        break;
(lldb) p obj
(VALUE) $4 = 0x00007f7d5b513fd0
(lldb)
```
ko1 pushed a commit that referenced this pull request Oct 21, 2021
This script is an lldb helper that just loops through all the comments
stored and prints out the comment along with the address corresponding
to the comment.

For example, I'm crashing in JIT code at address 0x0000000110000168.
Using the `lc` helper I can see that it's probably crashing inside the
exit back to the interpreter

```
(lldb) bt 5
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x22220021)
    frame #0: 0x0000000110000168
  * frame #1: 0x00000001002b5ff5 miniruby`invoke_block_from_c_bh [inlined] invoke_block(ec=0x0000000100e05350, iseq=0x0000000100c1ff10, self=0x0000000100c76cc0, captured=<unavailable>, cref=0x0000000000000000, type=<unavailable>, opt_pc=<unavailable>) at vm.c:1268:12
    frame #2: 0x00000001002b5f7d miniruby`invoke_block_from_c_bh [inlined] invoke_iseq_block_from_c(ec=<unavailable>, captured=<unavailable>, self=0x0000000100c76cc0, argc=2, argv=<unavailable>, kw_splat=0, passed_block_handler=0x0000000000000000, cref=0x0000000000000000, is_lambda=<unavailable>, me=0x0000000000000000) at vm.c:1340
    frame #3: 0x00000001002b5e14 miniruby`invoke_block_from_c_bh(ec=<unavailable>, block_handler=<unavailable>, argc=<unavailable>, argv=<unavailable>, kw_splat=0, passed_block_handler=0x0000000000000000, cref=0x0000000000000000, is_lambda=<unavailable>, force_blockarg=0) at vm.c:1358
    frame #4: 0x000000010029860b miniruby`rb_yield_values(n=<unavailable>) at vm_eval.c:0
(lldb) lc
0x11000006d "putobject_INT2FIX_1_"
0x110000083 "leave"
0x110000087 "check for interrupts"
0x110000087 "RUBY_VM_CHECK_INTS(ec)"
0x110000098 "check for finish frame"
0x1100000ed "getlocal_WC_0"
0x110000107 "getlocal_WC_1"
0x11000012a "opt_send_without_block"
0x110000139 "opt_send_without_block"
0x11000013c "exit to interpreter"
```
ko1 pushed a commit that referenced this pull request Oct 17, 2022
GnuWin32 sed strips only double quotes, but not single quotes, and
dies:

```
sed: -e expression #1, char 1: unknown command: `''
```
ko1 pushed a commit that referenced this pull request Nov 25, 2022
Always look up instance variable buffers when iterating.  It is possible
for the instance variable buffer to change out from under the object
during iteration, so we cannot cache the buffer on the stack.

In the case of Bug #19095, the transient heap moved the buffer during
iteration:

```
Watchpoint 1 hit:
old value: 0x0000000107c00df8
new value: 0x00000001032743c0
Process 31720 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
    frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
   1358	        }
   1359	        MEMCPY(new_ptr, old_ptr, VALUE, len);
   1360	        ROBJECT(obj)->as.heap.ivptr = new_ptr;
-> 1361	    }
   1362	}
   1363	#endif
   1364
miniruby`rb_obj_transient_heap_evacuate:
->  0x1006e5178 <+328>: b      0x1006e517c               ; <+332> at variable.c:1362:1
    0x1006e517c <+332>: ldp    x29, x30, [sp, #0x50]
    0x1006e5180 <+336>: add    sp, sp, #0x60
    0x1006e5184 <+340>: ret
Target 0: (miniruby) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = watchpoint 1
  * frame #0: 0x00000001006e5178 miniruby`rb_obj_transient_heap_evacuate(obj=0x000000010d6b94b0, promote=1) at variable.c:1361:5
    frame #1: 0x00000001006cb150 miniruby`transient_heap_block_evacuate(theap=0x0000000100b196c0, block=0x0000000107c00000) at transient_heap.c:734:17
    frame #2: 0x00000001006c854c miniruby`transient_heap_evacuate(dmy=0x0000000000000000) at transient_heap.c:808:17
    frame #3: 0x00000001007fe6c0 miniruby`rb_postponed_job_flush(vm=0x0000000104402900) at vm_trace.c:1773:21
    frame #4: 0x0000000100637a84 miniruby`rb_threadptr_execute_interrupts(th=0x0000000103803bc0, blocking_timing=0) at thread.c:2316:13
    frame #5: 0x000000010078b730 miniruby`rb_vm_check_ints(ec=0x00000001048038d0) at vm_core.h:2025:9
    frame #6: 0x00000001006fbd10 miniruby`vm_pop_frame(ec=0x00000001048038d0, cfp=0x0000000104a04440, ep=0x0000000104904a28) at vm_insnhelper.c:422:5
    frame #7: 0x00000001006fbca0 miniruby`rb_vm_pop_frame(ec=0x00000001048038d0) at vm_insnhelper.c:431:5
    frame #8: 0x00000001007d6420 miniruby`vm_call0_cfunc_with_frame(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:153:9
    frame #9: 0x00000001007d44cc miniruby`vm_call0_cfunc(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:164:12
    frame #10: 0x0000000100766e80 miniruby`vm_call0_body(ec=0x00000001048038d0, calling=0x000000016fdcc6a0, argv=0x0000000000000000) at vm_eval.c:210:15
    frame #11: 0x00000001007d76f0 miniruby`vm_call0_cc(ec=0x00000001048038d0, recv=0x000000010d6b49d8, id=2769, argc=0, argv=0x0000000000000000, cc=0x000000010d6b2e58, kw_splat=0) at vm_eval.c:87:12
    frame #12: 0x0000000100769e48 miniruby`rb_funcallv_scope(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000, scope=CALL_FCALL) at vm_eval.c:1051:16
    frame #13: 0x0000000100760a54 miniruby`rb_funcallv(recv=0x000000010d6b49d8, mid=2769, argc=0, argv=0x0000000000000000) at vm_eval.c:1066:12
    frame #14: 0x000000010037513c miniruby`rb_inspect(obj=0x000000010d6b49d8) at object.c:633:34
    frame #15: 0x000000010002c950 miniruby`inspect_ary(ary=0x000000010d6b4938, dummy=0x0000000000000000, recur=0) at array.c:3091:13
    frame #16: 0x0000000100642020 miniruby`exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, pairid=0x0000000000000000, arg=0x0000000000000000, outer=0, mid=2769) at thread.c:5177:23
    frame ruby#17: 0x00000001006412fc miniruby`rb_exec_recursive(func=(miniruby`inspect_ary at array.c:3084), obj=0x000000010d6b4938, arg=0x0000000000000000) at thread.c:5205:12
    frame ruby#18: 0x00000001000127f0 miniruby`rb_ary_inspect(ary=0x000000010d6b4938) at array.c:3117:12
```

In general though, any calls back out to the interpreter could change
the IV buffer, so it's not safe to cache.

[Bug #19095]
ko1 pushed a commit that referenced this pull request Feb 10, 2023
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.

Something similar came up in rubygems/rubygems#6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with ruby#6219).

Details:
---

Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:

```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```

Bundler 2.3.26 gave a helpful warning:

```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```

But then continued on and crashed while trying to report the unmet
dependency:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'

...
```

Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:

```
--- ERROR REPORT TEMPLATE -------------------------------------------------------

fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
   rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0

* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0

* #<Thread:0x0000000108dea4a0@Parallel Installer Worker #1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
   rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0

<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'

...
```

Changes
---

This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.

That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.

rubygems/rubygems@d73001a21d
ko1 pushed a commit that referenced this pull request Aug 3, 2023
[Bug #19793]

Dummy frames are created at the top level when requiring another file.
While requiring a file, it will try to convert using encodings. Some of
these encodings will not respond to to_str. If method_missing is
redefined on Object, then it will call method_missing and attempt raise
an error. However, the iseq is invalid as it's a dummy frame so it will
write an invalid iseq to the created NoMethodError.

The following script crashes:

```
GC.stress = true

class Object
  public :method_missing
end

File.write("/tmp/empty.rb", "")
require "/tmp/empty.rb"
```

With the following backtrace:

```
frame #0: 0x00000001000fa8b8 miniruby`RVALUE_MARKED(obj=4308637824) at gc.c:1638:12
frame #1: 0x00000001000fb440 miniruby`RVALUE_BLACK_P(obj=4308637824) at gc.c:1763:12
frame #2: 0x00000001000facdc miniruby`gc_writebarrier_incremental(a=4308637824, b=4308332208, objspace=0x000000010180b000) at gc.c:8822:9
frame #3: 0x00000001000faad8 miniruby`rb_gc_writebarrier(a=4308637824, b=4308332208) at gc.c:8864:17
frame #4: 0x000000010016aff0 miniruby`rb_obj_written(a=4308637824, oldv=36, b=4308332208, filename="../iseq.c", line=1279) at gc.h:804:9
frame #5: 0x0000000100162a60 miniruby`rb_obj_write(a=4308637824, slot=0x0000000100d09888, b=4308332208, filename="../iseq.c", line=1279) at gc.h:837:5
frame #6: 0x0000000100165b0c miniruby`iseqw_new(iseq=0x0000000100d09880) at iseq.c:1279:9
frame #7: 0x0000000100165a64 miniruby`rb_iseqw_new(iseq=0x0000000100d09880) at iseq.c:1289:12
frame #8: 0x00000001000d8324 miniruby`name_err_init_attr(exc=4309777920, recv=4304780496, method=827660) at error.c:1830:35
frame #9: 0x00000001000d1b80 miniruby`name_err_init(exc=4309777920, mesg=4308332496, recv=4304780496, method=827660) at error.c:1869:12
frame #10: 0x00000001000d1bd4 miniruby`rb_nomethod_err_new(mesg=4308332496, recv=4304780496, method=827660, args=4308332448, priv=0) at error.c:1957:5
frame #11: 0x000000010039049c miniruby`rb_make_no_method_exception(exc=4304914512, format=4308332496, obj=4304780496, argc=1, argv=0x000000016fdfab00, priv=0) at vm_eval.c:959:16
frame #12: 0x00000001003b3274 miniruby`raise_method_missing(ec=0x0000000100b06f40, argc=1, argv=0x000000016fdfab00, obj=4304780496, last_call_status=MISSING_NOENTRY) at vm_eval.c:999:15
frame #13: 0x00000001003945d4 miniruby`rb_method_missing(argc=1, argv=0x000000016fdfab00, obj=4304780496) at vm_eval.c:944:5
...
frame ruby#23: 0x000000010038f5e4 miniruby`rb_vm_call_kw(ec=0x0000000100b06f40, recv=4304780496, id=2865, argc=1, argv=0x000000016fdfab00, me=0x0000000100cbfcf0, kw_splat=0) at vm_eval.c:326:12
frame ruby#24: 0x00000001003c18e4 miniruby`call_method_entry(ec=0x0000000100b06f40, defined_class=4304927952, obj=4304780496, id=2865, cme=0x0000000100cbfcf0, argc=1, argv=0x000000016fdfab00, kw_splat=0) at vm_method.c:2720:20
frame ruby#25: 0x00000001003c440c miniruby`check_funcall_exec(v=6171896792) at vm_eval.c:589:12
frame ruby#26: 0x00000001000dec00 miniruby`rb_vrescue2(b_proc=(miniruby`check_funcall_exec at vm_eval.c:587), data1=6171896792, r_proc=(miniruby`check_funcall_failed at vm_eval.c:596), data2=6171896792, args="Pȗ") at eval.c:919:18
frame ruby#27: 0x00000001000deab0 miniruby`rb_rescue2(b_proc=(miniruby`check_funcall_exec at vm_eval.c:587), data1=6171896792, r_proc=(miniruby`check_funcall_failed at vm_eval.c:596), data2=6171896792) at eval.c:900:17
frame ruby#28: 0x000000010039008c miniruby`check_funcall_missing(ec=0x0000000100b06f40, klass=4304923536, recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000, respond=-1, def=36, kw_splat=0) at vm_eval.c:666:15
frame ruby#29: 0x000000010038fa60 miniruby`rb_check_funcall_default_kw(recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000, def=36, kw_splat=0) at vm_eval.c:703:21
frame ruby#30: 0x000000010038fb04 miniruby`rb_check_funcall(recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000) at vm_eval.c:685:12
frame ruby#31: 0x00000001001c469c miniruby`convert_type_with_id(val=4304780496, tname="String", method=3233, raise=0, index=-1) at object.c:3061:15
frame ruby#32: 0x00000001001c4a4c miniruby`rb_check_convert_type_with_id(val=4304780496, type=5, tname="String", method=3233) at object.c:3153:9
frame ruby#33: 0x00000001002d59f8 miniruby`rb_check_string_type(str=4304780496) at string.c:2571:11
frame ruby#34: 0x000000010014b7b0 miniruby`io_encoding_set(fptr=0x0000000100d09ca0, v1=4304780496, v2=4, opt=4) at io.c:11655:19
frame ruby#35: 0x0000000100139a58 miniruby`rb_io_set_encoding(argc=1, argv=0x000000016fdfb450, io=4308334032) at io.c:13497:5
frame ruby#36: 0x00000001003c0004 miniruby`ractor_safe_call_cfunc_m1(recv=4308334032, argc=1, argv=0x000000016fdfb450, func=(miniruby`rb_io_set_encoding at io.c:13487)) at vm_insnhelper.c:3271:12
...
frame ruby#43: 0x0000000100390b08 miniruby`rb_funcall(recv=4308334032, mid=16593, n=1) at vm_eval.c:1137:12
frame ruby#44: 0x00000001002a43d8 miniruby`load_file_internal(argp_v=6171899936) at ruby.c:2500:5
...
```
ko1 added a commit that referenced this pull request Oct 13, 2023
to avoid deadlock

```ruby
r = Ractor.new do
  obj = Thread.new{}
  Ractor.yield obj
rescue => e
  e.message
end
p r.take
```

```
(lldb) bt
* thread #1, name = 'miniruby', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff44881410 libpthread.so.0`__lll_lock_wait + 88
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab617c1d58 miniruby`ubf_event_waiting [inlined] thread_sched_lock_(th=0x0000aaab9df82980, file=<unavailable>, line=46, sched=0x0000aaab9dec79b8) at thread_pthread.c:351:5
    frame #4: 0x0000aaab617c1d50 miniruby`ubf_event_waiting(ptr=0x0000aaab9df82980) at thread_pthread_mn.c:46:5
    frame #5: 0x0000aaab617c6020 miniruby`rb_threadptr_interrupt [inlined] rb_threadptr_interrupt_common(trap=0, th=0x0000aaab9df82980) at thread.c:352:25
    frame #6: 0x0000aaab617c5fec miniruby`rb_threadptr_interrupt(th=0x0000aaab9df82980) at thread.c:365:5
    frame #7: 0x0000aaab617379b0 miniruby`rb_ractor_terminate_all at ractor.c:2364:13
    frame #8: 0x0000aaab6173797c miniruby`rb_ractor_terminate_all at ractor.c:2383:17
    frame #9: 0x0000aaab61737958 miniruby`rb_ractor_terminate_all [inlined] ractor_terminal_interrupt_all(vm=0x0000aaab9dea3320) at ractor.c:2375:1
    frame #10: 0x0000aaab61737950 miniruby`rb_ractor_terminate_all at ractor.c:2424:13
    frame #11: 0x0000aaab6164f108 miniruby`rb_ec_cleanup(ec=0x0000aaab9dea5900, ex=RUBY_TAG_NONE) at eval.c:239:9
    frame #12: 0x0000aaab6164fa3c miniruby`ruby_run_node(n=0x0000ffff417ed178) at eval.c:328:12
    frame #13: 0x0000aaab615a5ab0 miniruby`main at main.c:39:12
    frame #14: 0x0000aaab615a5a98 miniruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:58:12
    frame #15: 0x0000ffff44714b2c libc.so.6`__libc_start_main + 228
    frame #16: 0x0000aaab615a5b0c miniruby`_start + 52
(lldb) thread select 3
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
    frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
libpthread.so.0`__lll_lock_wait:
->  0xffff448813ec <+52>: svc    #0
    0xffff448813f0 <+56>: eor    w20, w20, #0x80
    0xffff448813f4 <+60>: sxtw   x20, w20
    0xffff448813f8 <+64>: b      0xffff44881414            ; <+92>
(lldb) bt
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab61823d68 miniruby`rb_vm_lock_enter_body [inlined] vm_lock_enter(no_barrier=false, lev=0x0000ffff215bfbe4, locked=false, vm=0x0000aaab9dea3320, cr=0x0000aaab9dec7890) at vm_sync.c:57:9
    frame #4: 0x0000aaab61823d60 miniruby`rb_vm_lock_enter_body(lev=0x0000ffff215bfbe4) at vm_sync.c:119:9
    frame #5: 0x0000aaab617c1b30 miniruby`thread_sched_setup_running_threads [inlined] rb_vm_lock_enter(file=<unavailable>, line=597, lev=0x0000ffff215bfbe4) at vm_sync.h:75:9
    frame #6: 0x0000aaab617c1b14 miniruby`thread_sched_setup_running_threads(vm=0x0000aaab9dea3320, add_th=0x0000aaab9df82980, del_th=<unavailable>, add_timeslice_th=0x0000000000000000, cr=<unavailable>, sched=<unavailable>, sched=<unavailable>) at thread_pthread.c:597:9
    frame #7: 0x0000aaab617c29b4 miniruby`thread_sched_wait_running_turn at thread_pthread.c:614:5
    frame #8: 0x0000aaab617c298c miniruby`thread_sched_wait_running_turn(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, can_direct_transfer=true) at thread_pthread.c:868:9
    frame #9: 0x0000aaab617c6f0c miniruby`thread_sched_wait_events(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, fd=<unavailable>, events=<unavailable>, rel=<unavailable>) at thread_pthread_mn.c:90:17
    frame #10: 0x0000aaab617c7354 miniruby`rb_thread_terminate_all at thread_pthread.c:3248:13
    frame #11: 0x0000aaab617c733c miniruby`rb_thread_terminate_all(th=0x0000aaab9df82980) at thread.c:466:13
    frame #12: 0x0000aaab617c7a64 miniruby`thread_start_func_2(th=0x0000aaab9df82980, stack_start=<unavailable>) at thread.c:713:9
    frame #13: 0x0000aaab617c7d1c miniruby`co_start [inlined] call_thread_start_func_2(th=0x0000aaab9df82980) at thread_pthread.c:2165:5
    frame #14: 0x0000aaab617c7cd0 miniruby`co_start(from=<unavailable>, self=0x0000aaab9df0f760) at thread_pthread_mn.c:421:9
```
ko1 added a commit that referenced this pull request Oct 14, 2023
to avoid deadlock

```ruby
r = Ractor.new do
  obj = Thread.new{}
  Ractor.yield obj
rescue => e
  e.message
end
p r.take
```

```
(lldb) bt
* thread #1, name = 'miniruby', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff44881410 libpthread.so.0`__lll_lock_wait + 88
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab617c1d58 miniruby`ubf_event_waiting [inlined] thread_sched_lock_(th=0x0000aaab9df82980, file=<unavailable>, line=46, sched=0x0000aaab9dec79b8) at thread_pthread.c:351:5
    frame #4: 0x0000aaab617c1d50 miniruby`ubf_event_waiting(ptr=0x0000aaab9df82980) at thread_pthread_mn.c:46:5
    frame #5: 0x0000aaab617c6020 miniruby`rb_threadptr_interrupt [inlined] rb_threadptr_interrupt_common(trap=0, th=0x0000aaab9df82980) at thread.c:352:25
    frame #6: 0x0000aaab617c5fec miniruby`rb_threadptr_interrupt(th=0x0000aaab9df82980) at thread.c:365:5
    frame #7: 0x0000aaab617379b0 miniruby`rb_ractor_terminate_all at ractor.c:2364:13
    frame #8: 0x0000aaab6173797c miniruby`rb_ractor_terminate_all at ractor.c:2383:17
    frame #9: 0x0000aaab61737958 miniruby`rb_ractor_terminate_all [inlined] ractor_terminal_interrupt_all(vm=0x0000aaab9dea3320) at ractor.c:2375:1
    frame #10: 0x0000aaab61737950 miniruby`rb_ractor_terminate_all at ractor.c:2424:13
    frame #11: 0x0000aaab6164f108 miniruby`rb_ec_cleanup(ec=0x0000aaab9dea5900, ex=RUBY_TAG_NONE) at eval.c:239:9
    frame #12: 0x0000aaab6164fa3c miniruby`ruby_run_node(n=0x0000ffff417ed178) at eval.c:328:12
    frame #13: 0x0000aaab615a5ab0 miniruby`main at main.c:39:12
    frame #14: 0x0000aaab615a5a98 miniruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:58:12
    frame #15: 0x0000ffff44714b2c libc.so.6`__libc_start_main + 228
    frame #16: 0x0000aaab615a5b0c miniruby`_start + 52
(lldb) thread select 3
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
    frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
libpthread.so.0`__lll_lock_wait:
->  0xffff448813ec <+52>: svc    #0
    0xffff448813f0 <+56>: eor    w20, w20, #0x80
    0xffff448813f4 <+60>: sxtw   x20, w20
    0xffff448813f8 <+64>: b      0xffff44881414            ; <+92>
(lldb) bt
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab61823d68 miniruby`rb_vm_lock_enter_body [inlined] vm_lock_enter(no_barrier=false, lev=0x0000ffff215bfbe4, locked=false, vm=0x0000aaab9dea3320, cr=0x0000aaab9dec7890) at vm_sync.c:57:9
    frame #4: 0x0000aaab61823d60 miniruby`rb_vm_lock_enter_body(lev=0x0000ffff215bfbe4) at vm_sync.c:119:9
    frame #5: 0x0000aaab617c1b30 miniruby`thread_sched_setup_running_threads [inlined] rb_vm_lock_enter(file=<unavailable>, line=597, lev=0x0000ffff215bfbe4) at vm_sync.h:75:9
    frame #6: 0x0000aaab617c1b14 miniruby`thread_sched_setup_running_threads(vm=0x0000aaab9dea3320, add_th=0x0000aaab9df82980, del_th=<unavailable>, add_timeslice_th=0x0000000000000000, cr=<unavailable>, sched=<unavailable>, sched=<unavailable>) at thread_pthread.c:597:9
    frame #7: 0x0000aaab617c29b4 miniruby`thread_sched_wait_running_turn at thread_pthread.c:614:5
    frame #8: 0x0000aaab617c298c miniruby`thread_sched_wait_running_turn(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, can_direct_transfer=true) at thread_pthread.c:868:9
    frame #9: 0x0000aaab617c6f0c miniruby`thread_sched_wait_events(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, fd=<unavailable>, events=<unavailable>, rel=<unavailable>) at thread_pthread_mn.c:90:17
    frame #10: 0x0000aaab617c7354 miniruby`rb_thread_terminate_all at thread_pthread.c:3248:13
    frame #11: 0x0000aaab617c733c miniruby`rb_thread_terminate_all(th=0x0000aaab9df82980) at thread.c:466:13
    frame #12: 0x0000aaab617c7a64 miniruby`thread_start_func_2(th=0x0000aaab9df82980, stack_start=<unavailable>) at thread.c:713:9
    frame #13: 0x0000aaab617c7d1c miniruby`co_start [inlined] call_thread_start_func_2(th=0x0000aaab9df82980) at thread_pthread.c:2165:5
    frame #14: 0x0000aaab617c7cd0 miniruby`co_start(from=<unavailable>, self=0x0000aaab9df0f760) at thread_pthread_mn.c:421:9
```
ko1 added a commit that referenced this pull request Oct 14, 2023
to avoid deadlock

```ruby
r = Ractor.new do
  obj = Thread.new{}
  Ractor.yield obj
rescue => e
  e.message
end
p r.take
```

```
(lldb) bt
* thread #1, name = 'miniruby', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff44881410 libpthread.so.0`__lll_lock_wait + 88
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab617c1d58 miniruby`ubf_event_waiting [inlined] thread_sched_lock_(th=0x0000aaab9df82980, file=<unavailable>, line=46, sched=0x0000aaab9dec79b8) at thread_pthread.c:351:5
    frame #4: 0x0000aaab617c1d50 miniruby`ubf_event_waiting(ptr=0x0000aaab9df82980) at thread_pthread_mn.c:46:5
    frame #5: 0x0000aaab617c6020 miniruby`rb_threadptr_interrupt [inlined] rb_threadptr_interrupt_common(trap=0, th=0x0000aaab9df82980) at thread.c:352:25
    frame #6: 0x0000aaab617c5fec miniruby`rb_threadptr_interrupt(th=0x0000aaab9df82980) at thread.c:365:5
    frame #7: 0x0000aaab617379b0 miniruby`rb_ractor_terminate_all at ractor.c:2364:13
    frame #8: 0x0000aaab6173797c miniruby`rb_ractor_terminate_all at ractor.c:2383:17
    frame #9: 0x0000aaab61737958 miniruby`rb_ractor_terminate_all [inlined] ractor_terminal_interrupt_all(vm=0x0000aaab9dea3320) at ractor.c:2375:1
    frame #10: 0x0000aaab61737950 miniruby`rb_ractor_terminate_all at ractor.c:2424:13
    frame #11: 0x0000aaab6164f108 miniruby`rb_ec_cleanup(ec=0x0000aaab9dea5900, ex=RUBY_TAG_NONE) at eval.c:239:9
    frame #12: 0x0000aaab6164fa3c miniruby`ruby_run_node(n=0x0000ffff417ed178) at eval.c:328:12
    frame #13: 0x0000aaab615a5ab0 miniruby`main at main.c:39:12
    frame #14: 0x0000aaab615a5a98 miniruby`main(argc=<unavailable>, argv=<unavailable>) at main.c:58:12
    frame #15: 0x0000ffff44714b2c libc.so.6`__libc_start_main + 228
    frame #16: 0x0000aaab615a5b0c miniruby`_start + 52
(lldb) thread select 3
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
    frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
libpthread.so.0`__lll_lock_wait:
->  0xffff448813ec <+52>: svc    #0
    0xffff448813f0 <+56>: eor    w20, w20, #0x80
    0xffff448813f4 <+60>: sxtw   x20, w20
    0xffff448813f8 <+64>: b      0xffff44881414            ; <+92>
(lldb) bt
* thread #3, name = 'bootstraptest.*', stop reason = signal SIGSTOP
  * frame #0: 0x0000ffff448813ec libpthread.so.0`__lll_lock_wait + 52
    frame #1: 0x0000ffff4487a078 libpthread.so.0`__pthread_mutex_lock + 232
    frame #2: 0x0000aaab617c0980 miniruby`rb_native_mutex_lock(lock=<unavailable>) at thread_pthread.c:109:14
    frame #3: 0x0000aaab61823d68 miniruby`rb_vm_lock_enter_body [inlined] vm_lock_enter(no_barrier=false, lev=0x0000ffff215bfbe4, locked=false, vm=0x0000aaab9dea3320, cr=0x0000aaab9dec7890) at vm_sync.c:57:9
    frame #4: 0x0000aaab61823d60 miniruby`rb_vm_lock_enter_body(lev=0x0000ffff215bfbe4) at vm_sync.c:119:9
    frame #5: 0x0000aaab617c1b30 miniruby`thread_sched_setup_running_threads [inlined] rb_vm_lock_enter(file=<unavailable>, line=597, lev=0x0000ffff215bfbe4) at vm_sync.h:75:9
    frame #6: 0x0000aaab617c1b14 miniruby`thread_sched_setup_running_threads(vm=0x0000aaab9dea3320, add_th=0x0000aaab9df82980, del_th=<unavailable>, add_timeslice_th=0x0000000000000000, cr=<unavailable>, sched=<unavailable>, sched=<unavailable>) at thread_pthread.c:597:9
    frame #7: 0x0000aaab617c29b4 miniruby`thread_sched_wait_running_turn at thread_pthread.c:614:5
    frame #8: 0x0000aaab617c298c miniruby`thread_sched_wait_running_turn(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, can_direct_transfer=true) at thread_pthread.c:868:9
    frame #9: 0x0000aaab617c6f0c miniruby`thread_sched_wait_events(sched=0x0000aaab9dec79b8, th=0x0000aaab9df82980, fd=<unavailable>, events=<unavailable>, rel=<unavailable>) at thread_pthread_mn.c:90:17
    frame #10: 0x0000aaab617c7354 miniruby`rb_thread_terminate_all at thread_pthread.c:3248:13
    frame #11: 0x0000aaab617c733c miniruby`rb_thread_terminate_all(th=0x0000aaab9df82980) at thread.c:466:13
    frame #12: 0x0000aaab617c7a64 miniruby`thread_start_func_2(th=0x0000aaab9df82980, stack_start=<unavailable>) at thread.c:713:9
    frame #13: 0x0000aaab617c7d1c miniruby`co_start [inlined] call_thread_start_func_2(th=0x0000aaab9df82980) at thread_pthread.c:2165:5
    frame #14: 0x0000aaab617c7cd0 miniruby`co_start(from=<unavailable>, self=0x0000aaab9df0f760) at thread_pthread_mn.c:421:9
```
ko1 pushed a commit that referenced this pull request Apr 5, 2024
* YJIT: A64: Use ADDS/SUBS/CMP (immediate) when possible

We were loading 1 into a register and then doing ADDS/SUBS previously.
That was particularly bad since those come up in fixnum operations.

```diff
   # integer left shift with rhs=1
-  mov x11, #1
-  subs x11, x1, x11
+  subs x11, x1, #1
   lsl x12, x11, #1
   asr x13, x12, #1
   cmp x13, x11
-  b.ne #0x106ab60f8
-  mov x11, #1
-  adds x12, x12, x11
+  b.ne #0x10903a0f8
+  adds x12, x12, #1
   mov x1, x12
```

Note that it's fine to cast between i64 and u64 since the bit pattern is
preserved, and the add/sub themselves don't care about the signedness of
the operands.

CMP is just another mnemonic for SUBS.

* YJIT: A64: Split asm.mul() with immediates properly

There is in fact no MUL on A64 that takes an immediate, so this
instruction was using the wrong split method. No current usages of this
form in YJIT.

---------

Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants