Eliminate all magic builtin libraries

[headius]

JRuby has for a long time had special "built-in" libraries that did not live on the filesystem but appeared to do so. We replaced "so" with "jar" for extensions and registered about a dozen special libraries this way, mostly equivalent to MRI's native extensions.

However, bootsnap hooks into require, and expects that the files it sees loading will actually exist somewhere on the filesystem. This may be a bit fragile, if any impl has more than filesystem-based resources (as in JRuby, where we support sources from several different formats of URL), but by moving these builtin libraries to real files it allows bootsnap to work without failing to load those libraries.

I have taken two approaches to making these files no longer "built-in":

• For libraries that are loaded at boot or where we always want to load our internal extension, I replaced the builtin and and requires of it with the boot of the actual library.
• For libraries that have other code, I have incorporated the direct load of our internal extension into those files.

This relates to Shopify/bootsnap#162 and (probably) eliminates the need for Shopify/bootsnap#200.

TODO:

• Green build
• Get stdlib tweaks back into our stdlib fork
• Use a better mechanism for loading these libraries? (@kares?)
• Test with bootsnap

[headius]

There's a commit in this PR that works around an apparent problem with the "loaded features" index internally. After a modification is made directly to $LOADED_FEATURES, manually-added "provides" no longer appear to work. See #5590.

[kares]

could this one be post-poned for at least 9.2.7, please? (looking from the phone here)

would like to look into this deeper ... JRuby::Util had a new ext loading mechanism. which we started using recently as a prefered way.

we should try to avoid JI self-reflection (JRuby.runtime etc) during boot, preferably also during JRuby's std libraries loading since it leaves an inconsistent internal state (due UNDEF and similar). the 'design' idea was to expose the minimal iface from JRuby::Util ... which now, in part, seems to be thrown away.

[headius]

@kares Yeah the new mechanism is what I had in mind. I think almost all the changes I made should work ok with that. Java, JRuby, and JRubyUtil libraries will continue to be initialized from Ruby.java since they need to be there for other pieces, but the remaining libraries should be able to use your ext method. Sound ok?

I don't think there's any reason this has to be in 9.2.6 if we've got a 9.2.7 coming before end of month. Clock's ticking on getting a stable release for Rails 6.

JRuby::Util actually didn't change much at all. The only thing that is now deprecated is the listing of internal libraries that bootsnap was going to use to work around our buitins. They could still release that patch (Shopify/bootsnap#162) or we could just say JRuby 9.2.x is needed for bootsnap to work.

[headius]

I updated all external libs to use load_ext except nkf, because the library did nothing but initialize the class and I deleted it. It could be restored.

[headius]

@kares Need you to review ae9d88d for sure; I made this change just to get better logging of the exception that was breaking various ext loads, but once I made it the breaking loads started to work again. I suspect something about your re-raising as NameError was screwing up some extension load/fallback logic.

[headius]

I have tested with bootsnap in Rails 5.2.0, and all seems to be working properly. I added some logging to prove that bootsnap is properly caching paths to all required libraries:

$ JRUBY_OPTS=-w jruby -S rails s
used bootsnap cache for active_support/dependencies
used bootsnap cache for thread
used bootsnap cache for concurrent/map
used bootsnap cache for concurrent/constants
used bootsnap cache for concurrent/synchronization
used bootsnap cache for concurrent/utility/engine
used bootsnap cache for concurrent/synchronization/abstract_object
used bootsnap cache for concurrent/utility/native_extension_loader
used bootsnap cache for concurrent/concurrent_ruby.jar
used bootsnap cache for concurrent/synchronization/mri_object
...

I also see our former built-in libraries in this list, like cgi/escape. There are probably tests in bootsnap we could run but it works now where it didn't work for me before.

[kares]

looking much better,
great to see some old unused code gone and and explicit CatchThrow class

[kares]

you could add a static RubyNKF.load(org.jruby.Ruby) which delegates to createNKF
its not loaded on boot and (guessing NKF) is rarely used thus not really blocker

[kares]

not all comments where addressed esp. JRuby.runtime.posix.native? will cause self-reflection again.
which we kind of tried avoiding during normal boot ... as a resolution for another reported issue ...

[enebo]

@kares I am happy to revert this unless @headius or you can just ammend by removing the reflection.

[headius]

@kares You're right, I'll restore those uses of JRuby::Util. I did not realize the purpose was to avoid extra reflection at boot, which I agree can get problematic (not to mention impacting startup time).

[kares]

sounds good, I'll try to come up with a way of testing that the usual boot does not cause much Java proxy-ing

[headius]

Everything has been addressed in d64c07f.