YJIT crash at !cb.has_dropped_bytes() assertion

[intrip]

We are running our app with Ruby 3.3.0-preview2 and Rails on Edge (main branch) and we noticed multiple YJIT crashes on some of our web hosts. We are using Unicorn as web server.

I've attached the full trace extracted from the unicorn crash.

error.log

Thanks for support!

[intrip]

We have been using YJIT for a while with no issue, and noticed that the start of this issue matches with the upgrade of Rails from 7.0.4 to the main branch.

[maximecb]

Hi Jacopo,

Thank you for reporting this issue, we very much appreciate it. I see from your log that you're on x86-64. Is this running on Ubuntu machines? AWS?

Do you happen to have an offline reproduction? Is this an issue that only happens once deployed, or are you able to trigger it on your CI as well?

Best regards,

• Maxime

[XrXr]

Please give preview3 a try and see if you have the same issue. We've landed many fixes since preview2 and the issue you're running into might have been fixed already.

[intrip]

Thank you for reporting this issue, we very much appreciate it. I see from your log that you're on x86-64. Is this running on Ubuntu machines? AWS?

Those are Ubuntu machines running on Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz. No AWS, we run the app in our own servers.

Do you happen to have an offline reproduction? Is this an issue that only happens once deployed, or are you able to trigger it on your CI as well?

Unfortunately, we don't have a reproduction script for it right now. The CI works fine with YJIT enabled and the issue is triggered only in the production servers.

Please let me know if I can do anything more to help you in troubleshooting it. I really appreciate your help!

[intrip]

Please give preview3 a try and see if you have the same issue. We've landed many fixes since preview2 and the issue you're running into might have been fixed already.

Sure! I'll try the upgrade path as the first step and will let you know. Thanks!

[maximecb]

Hello @intrip ! Any update on whether this bug is present in preview3 as well?

[intrip]

Hello @intrip ! Any update on whether this bug is present in preview3 as well?

We are not live with the upgrade: We'll deploy it in a couple of days. I'll let you know if that fixes it as soon as possible 🙇‍♂️

[intrip]

@maximecb we just upgraded to preview3 and enabled YJIT on one of the hosts that were affected by this issue, no error so far which is a good sign but not fail-proof. Tomorrow I'll enable YJIT on more hosts, if the error won't manifest anymore we can consider it as fixed.

[maximecb]

Hi @intrip, thanks for taking the time to test this and report back. I'm glad to hear that it seems to be fixed so far 🤞

[intrip]

@maximecb after enabling YJIT on more hosts we encountered again the error 😞.
We are running multiple Unicorn processes in the same host and when the YJIT error occurs all the process crashes at the same time, which causes overlapping error logs. This is the best error I could manage to extract so far:
error.log. Is it enough for your troubleshooting? Hopefully, I'll manage to extract a cleaner trace tomorrow. Please let me know If I can do anything else to help troubleshoot this.

[maximecb]

Hi again @intrip,

Sorry to hear that this bug is still present in preview3. On the positive side, my colleagues have just made several fixes that seem like they are very likely related to this, which are available on Ruby/master.

Would you be able to build/deploy Ruby master? It should be 100% compatible with preview3 and compiled in the same way. If you do, I would recommend picking a commit sha that passed all the tests, e.g. 11d7c75

[intrip]

Thanks, @maximecb, I'm not sure if we can deploy Ruby master with our current setup, but I'll check.

I've also found a cleaner log: error.log

[k0kubun]

The backtrace was a useful information, thanks.

However, I can't figure out why it happened with that version, so we might need you to run an arbitrary Ruby sha to get more information out of the crash message (since we currently have no crash in production pods that run Ruby master). In any case, it'd be nice to first run the Ruby sha Maxime mentioned, which has some YJIT bugfixes after preview3.

My analysis (for YJIT team to read): This is a has_dropped_bytes situation in ocb for generating a side exit for cb. In set_page (for ocb), we first set self.dropped_bytes = true, assert there's a 6-byte capacity (5 in master), write 5 bytes with jmp_ptr, and assert !self.dropped_bytes (which failed). This is very weird. write_jcc_ptr could set dropped_bytes = true when an offset is very large, but this shouldn't happen for a jump between code pages (hence the assertion). I don't think write_byte would fail either. So 🤔