Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change AOT class generation for speedier requires #2234

Closed
nirvdrum opened this issue Nov 24, 2014 · 2 comments
Closed

Change AOT class generation for speedier requires #2234

nirvdrum opened this issue Nov 24, 2014 · 2 comments
Assignees
Labels

Comments

@nirvdrum
Copy link
Contributor

@nirvdrum nirvdrum commented Nov 24, 2014

Currently, AOT compiled classes are searched by require by adding the .class suffix to the search strategy. In the very common case of just using JRuby to execute Ruby (i.e., non-AOT) this means twice as many filesystem lookups and they always fail. Changing the way AOT file generation works would allow us to speed up requiring files in the dominant use case.

Relevant IRC discussion from Nov. 24, 2014:

<headius> or just move to the flag and change nothing else
<nirvdrum> If the .class is on the classpath, I think it'll be found anyway.  I'd need to trace each of those phases to be sure.
<enebo> yeah so explicit source change requirement for users would not work if AOT files depend on requiring AOT files
<headius> nirvdrum: it doesn't quite work that way because the package in AOT .class did not match any dir structure
<headius> they had to be loaded specially
<nirvdrum> Ahh.
<nirvdrum> Well, fix that :-)
<headius> it's not really possible to pick a package for them at AOT since load path can find them from anywhere
<nirvdrum> Gotcha.
<enebo> default package!
<headius> I say we just add the flag for now and think more about how to reenable AOT in a logical way
<nirvdrum> And as a tertiary step, it'd be nice to get rid of that service loader.
<headius> the old way was a big of a hack anyway
<headius> bit
<nirvdrum> Since as best as I can tell, that's just a CNFE factory.
<headius> the service loader can't go away without breaking existing exts, but perhaps we could add the requirement that the .class has to be a resource
<nirvdrum> I think it's used for one or two internal JRuby services and the rest of the time it's just overhead.
<headius> then it's just an existance check
<headius> there are some third-party gems that still use it
<headius> because we haven't come up with a better alternative that's automatic
<nirvdrum> Ahh.  What was it for?
<headius> it was to make those .jar act like .so and be directly requirable
* travis-ci (~travis-ci@ec2-54-227-165-79.compute-1.amazonaws.com) has joined
<travis-ci> jruby/jruby (master:f9c709a by Charles Oliver Nutter): The build is still failing. (http://travis-ci.org/jruby/jruby/builds/41966578)
* travis-ci (~travis-ci@ec2-54-227-165-79.compute-1.amazonaws.com) has left
* havenwood has quit (Remote host closed the connection)
* iamjarvo has quit (Quit: My MacBook has gone to sleep. ZZZzzz…)
<headius> in retrospect I should have put somethin in the Manifest
<headius> my bad
<nirvdrum> But there's already a .jar lookup for require.
<headius> right, it does double duty
<nirvdrum> This seems to looking for a class with base name + Service
<headius> originally it was just for magic ext load, but because it has to put the jar in classpath to load the service, it also acted like adding jar to classpath
<headius> the latter behavior became widely used while the former is only known by a handful of people
<nirvdrum> Okay.
<nirvdrum> Well, it might be worth just documenting when, if ever, that can be excised.
<headius> I propose this...
<nirvdrum> There were already ext. breaking changes.  Some were put back in.
<headius> 1. add flag for .class search and default it to off (don't include .class in search)
<headius> 2. modify service lookup to try to find a .class resource rather than loading class and failing
<headius> and probably add a warning that this mechanism is deprecated
<headius> and implement a better mechanism like Manifest entry :-)
<enebo> #2 with warning seems ok to me
* paulswilliamsesq (~textual@host81-138-27-254.in-addr.btopenworld.com) has joined
<enebo> #1 is probably also ok but I want to discuss this one more
* mister_solo (~mister_so@89.107.183.4) has joined
* e_dub has quit (Quit: ZZZzzz…)
<enebo> #1 can be a punt for now I guess but I feel like we can solve this one if we make AOT authors change their pattern of packaging
<nirvdrum> I have no idea how popular AOT is.  Is there a large group of users out there?
<headius> we don't have a good pattern to follow
<headius> nirvdrum: there are commercial packages that do it
<headius> I don't think there's many
<enebo> If we make a) require_class ‘foo’ b) make wrapper foo.rb: require_class ‘foo’ c) require AOT to compile to a different dir than source
<headius> we generally discourage it because it takes a lot longer to boot
<nirvdrum> And this stuff done with jrubyc, right?  Not the force JIT?
* pchalupa has quit (Quit: Leaving)
<headius> nirvdrum: that's right
<enebo> I would argue this change is acceptable because AOT is not done for performance reasons and people do not actually want their original source distributed
<headius> you have to actively jrubyc to get AOT output
<enebo> It also follows the same pattern/conventions of Java for compilation
<nirvdrum> Generally, I'm not a fan of breaking things on people.  But 9k represents an opportunity for a bit of breakage because the language level is already going to cause issues.
<nirvdrum> At least, IMHO.
<headius> nirvdrum: indeed, and we better make those choices now :-)
<headius> enebo: I dunno...it bugs me
<enebo> why is that?
<headius> it's like javac rewriting your java sources
<enebo> why?
<enebo> WHY?
<enebo> :)
<headius> well who makes the wrapper?
<enebo> jrubyc
<enebo> It makes .class and .rb stub into different dir
<headius> so you have to output to a separate dir or it wipes out original sources
<enebo> your source is untouched
<enebo> headius: well I would not allow them to compile to same dir
<headius> that might be ok
<enebo> headius: I would output an error saying it would overwrite their sources and bomb out
<headius> harumph
<enebo> The beauty of this solution is it removes even needing a flag for .class resources
<headius> so if src_to_compile == src_stub_from_compile it errors
<enebo> yeah it would need to stat to make sure you are not overrwring the same source file you are reading from
<headius> looking to see how warbler does this now
<headius> if it uses stubs I'm on board
<enebo> but this is possible so I would not be worried about someone picking ‘./../../foo/src’
<nirvdrum> Well, I leave that to you guys.  I'd be okay with -Xrequire_search.include_class_files=true and just add the .class suffix back in.
<enebo> I guess my only reservation on the idea is that the process for generating AOT will need to change but if people nearly universally use Warbler to accomplish this I think it can be made transparent to almost all AOT users by changing warbler
* havenwood (~Havenn@gateway/tor-sasl/havenn) has joined
<enebo> That argument would also apply to a flag but I hate the idea of another environmental property
* iamjarvo (~textual@pool-98-115-181-52.phlapa.fios.verizon.net) has joined
<enebo> I would much rather only spec out ‘require_class’ method
<enebo> for that matter perhaps we could eliminate require_class as well
<enebo> just make a require of classpath url?
<enebo> but that means package too perhaps
<headius> files[apply_pathmaps(config, ruby_source, :application)] = StringIO.new("load __FILE__.sub(/\.rb$/, '.class')")
<headius> warbler rewrites them into stubs while building the jar/war
<headius> that also fixes issues with .rb files being read and evaluated directly, as some loaders/reloaders might do
* havenwood has quit (Remote host closed the connection)
<headius> I think we should still add the property for old behavior and warn
<headius> I'd rather get an idea who's using the feature before we nuke it
* tenderlove (~tenderlov@pdpc/supporter/active/tenderlove) has joined
<enebo> headius: yeah with PHONE HOME
<nirvdrum> Next up is who expects "require 'foo'" to find foo.jar?  I had no idea JRuby even did that.  Whenever I need to require a jar, I use "require 'foo.jar',"
<headius> nirvdrum: ext authors using *Service :-D
<enebo> So just to recap we will output to new dir for jrubyc and make .class and .rb
<enebo> We will abort if they try and use their source dir
<enebo> And we will allow a flag for old behavior
<enebo> with warning
<headius> that sounds right...we need an issue to track the change
<enebo> impl of .rb stub still a TODO
<headius> AOT is still a TODO :-D
<enebo> well I have soln for AOT I think as well
<enebo> which is just IR persistence unbitrott'd
<enebo> which is pretty cool since as we were saying the other day it works for 100% of all code to AOT since it starts interpd
<headius> ahh yes
<headius> so not even .class anymore
<headius> .rbj files or something :-D
<nirvdrum> headius: Hmm . . . it'd be good to drop that search, too.  I figured exts wrapped in gems would handle this more gracefully.
<enebo> headius: It could also be .class if we want too
* havenwood (~Havenn@gateway/tor-sasl/havenn) has joined
<enebo> static initializer could process compiled byte[] of persisted code
<headius> nirvdrum: that's tougher because MRI require searching does try to load ext for the bare name
<headius> so we'd diverse
<headius> diverge
<nirvdrum> I didn't know that.  So, it'll look up a .so?
* noop has quit (Ping timeout: 240 seconds)
<headius> yes
<headius> e.g. require 'openssl'
<nirvdrum> Does it also have a .so.rb lookup?
<headius> I don't think so but I'm not sure
<nirvdrum> I guess I always assumed that had an openssl.rb that loaded the ext.
<headius> now you know!
<nirvdrum> So much performance to be gained if the world could be bothered to type upwards of 4 extra chars.
<headius> and that's why require searches do twice as many stats as necessary
* anaeem1_ has quit (Remote host closed the connection)
<nirvdrum> I hope donV pops on.  I'd love to get to the bottom of the .jar.rb.
<headius> he was around earlier
<headius> I missed him too
<headius> nirvdrum: wanna throw these items into a couple issues please?
<nirvdrum> Sure.
@mkristian
Copy link
Member

@mkristian mkristian commented Nov 24, 2014

"too long" but I want to add the following

2014-11-24T16:06:37.134Z: LoadService: trying builtinLib: bla.rb
2014-11-24T16:06:37.134Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/gems/shared/gems/jruby-openssl-0.9.6.dev-java/lib/bla.rb
2014-11-24T16:06:37.135Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/2.2/site_ruby/bla.rb
2014-11-24T16:06:37.135Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/shared/bla.rb
2014-11-24T16:06:37.135Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/stdlib/bla.rb
2014-11-24T16:06:37.135Z: LoadService: trying jarExtension: bla
2014-11-24T16:06:37.135Z: LoadService: trying builtinLib: bla.class
2014-11-24T16:06:37.135Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/gems/shared/gems/jruby-openssl-0.9.6.dev-java/lib/bla.class
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/2.2/site_ruby/bla.class
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/shared/bla.class
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/stdlib/bla.class
2014-11-24T16:06:37.136Z: LoadService: trying builtinLib: bla.jar
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/gems/shared/gems/jruby-openssl-0.9.6.dev-java/lib/bla.jar
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/2.2/site_ruby/bla.jar
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/shared/bla.jar
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/stdlib/bla.jar
2014-11-24T16:06:37.136Z: LoadService: trying builtinLib: bla.jar.rb
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/gems/shared/gems/jruby-openssl-0.9.6.dev-java/lib/bla.jar.rb
2014-11-24T16:06:37.136Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/2.2/site_ruby/bla.jar.rb
2014-11-24T16:06:37.137Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/shared/bla.jar.rb
2014-11-24T16:06:37.137Z: LoadService: trying fileResource: /home/christian/projects/active/maven/jruby/lib/ruby/stdlib/bla.jar.rb
2014-11-24T16:06:37.137Z: LoadService: trying fileInClasspath: bla.rb
2014-11-24T16:06:37.137Z: LoadService: trying fileInClasspath: bla.class
2014-11-24T16:06:37.137Z: LoadService: trying fileInClasspath: bla.jar
2014-11-24T16:06:37.137Z: LoadService: trying fileInClasspath: bla.jar.rb
LoadError: no such file to load -- bla

@headius
Copy link
Member

@headius headius commented Nov 24, 2014

Added flag for old behavior and removed .class from default search logic in f5c2edf.

@headius headius closed this as completed Dec 3, 2014
@headius headius added this to the JRuby 9.0.0.0 milestone Dec 3, 2014
@headius headius self-assigned this Dec 3, 2014
@headius headius added the core label Dec 3, 2014
@headius headius removed this from the JRuby 9.0.0.0 milestone Jan 16, 2015
@headius headius added this to the JRuby 9.0.0.0-pre1 milestone Jan 16, 2015
@headius headius added this to the JRuby 9.0.0.0-pre1 milestone Jan 16, 2015
@headius headius removed this from the JRuby 9.0.0.0 milestone Jan 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants