Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Include bundler in JRuby distribution #1146

Open
mdekstrand opened this Issue · 46 comments
@mdekstrand

The JRuby distribution includes a few tools already, such as rake. It'd be very useful for at least some use cases for JRuby to also include bundler.

I'm using JRuby to bootstrap some projects (generally nanoc website builds) so that they can be fully run with nothing but a JDK. Right now, that bootstrap is:

  1. Download jruby (either from jruby.org, or via Maven)
  2. Use JRuby's gem to install bundler somewhere in the local directory
  3. Use the freshly-installed bundle to install the rest of the project's dependencies from the Gemfile

If JRuby's binary distribution and jruby-complete contained Bundler, I could entirely eliminate step 2.

@enebo
Owner

I actually think this is a good idea. We used to ship some other gems like rspec and removed them because they were not universal. Bundler is nearly universal in usage on a project of any size. So +1 from me.

@enebo
Owner

Of any substantial size...

@headius
Owner

We have actually moved toward shipping fewer libraries as built in. Currently, I don't think our normal distribution ships anything but jruby-launcher.

However, I could see adding more utilities to jruby-launcher. Rake and Bundler would both be good candidates, as would RSpec. But there's a lot of different possible combinations here and I don't want us to get mired down deciding what should be included...

@enebo
Owner

jruby-bin-* does actually install rake already. This might be another error to be fixed perhaps with out Maven changes? The main reason for adding bundler (and I think rake, gem) is that jruby-complete still does not have a nice mechanism for adding specific gems to it past manually re-packing or having a ENV plan. It would make the above scenario very simple. To me it is a balance between which one will generate more complaints :)

@mkristian
Owner
@mdekstrand

@mkristian If jruby-complete contained Bundler, then Bundler would be immediately available in a JRuby downloaded via Maven (my build scripts can reference JRuby via Maven, and use Bundler from the classpath).

@mkristian
Owner
@mdekstrand

@mkristian Perhaps I should have said 'from Maven' instead of 'via Maven'.

In my particular case, I am using Gradle, but getting JRuby from Maven. Gradle is also self-bootstrapping with gradlew. Or, in older versions of this script, using make and wget.

@timuckun

+1 Pretty much impossible to deploy a ruby app without bundler these days.

@mkristian
Owner
@mdekstrand

1 word reason for including bundler: bootstrapping. Specifically, using JRuby to get a working Ruby in an arbitrary/unknown environment with minimal effort.

My use case is bootstrapping projects that use Ruby dependencies. Typically nanoc sites. These sites are used, modified, and deployed in a managed computing environment where we may or may not have Ruby, we may or may not have a Ruby that has adequate facilities to let us build extensions, etc. Users also have crazy-limited home directory quotas, and who knows what gems installed. I don't think the standard Ruby our system administrators give us includes bundler, either.

We do have a working JVM reliably.

One of the beauties of JRuby is that it is effectively self-contained. I can download the JRuby binary zip file and it just works on any Linux, Mac, or even Windows system, so long as it can find a JVM. If my build tool (Gradle, in my case), can talk to Maven repositories, then I can just have it pull in jruby-all and not even have to write that download logic myself.

Once I have Bundler, it takes care of installing nanoc and all the other required gems locally into the project directory. Again, almost zero hassle.

The one hassle in this chain is getting Bundler installed. I have to install it somewhere; if I unpack the JRuby zip file, I can install it there, but it's still another piece. If I am launching JRuby from Maven via Gradle's JavaExec task, I don't really have anywhere to install it except a Bundler-only gem directory in the project or build directory. In which case I have to set up environment variables to tell JRuby where to find Bundler for all subsequent tasks.

If JRuby included Bundler, then boostrapping an app would be crazy-simple:

  1. Download and unpack JRuby (or just require it to be on the classpath, fetched from Maven)
  2. Run bundle install
  3. Enjoy
@mkristian
Owner
@timuckun
@qerub

+1

I'm currently "manually re-packing" (as @enebo mentioned) to include Bundler, like this: https://gist.github.com/qerub/7675016

so what is the rational why jruby needs bundler and all the other rubies do NOT.

All rubies should include Bundler, IMHO. :)

But I think it's more important for JRuby to include it because of jruby-complete where it is not installable out-of-the-box:

$ java -jar jruby-complete-1.7.8.jar -S gem install bundler
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the file:/Users/qerub/local/sw/jruby-complete/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/gems/shared directory.

(One has to fiddle with GEM_HOME to make it work as @elehack already said.)

@soveran

Bundler is easy, specially if you have it already installed. Most rubyists already use it daily, so having it as a default looks like a good idea. But on the other hand, you can solve the same problem with simpler tools. The lines of code count for Bundler yields a hard to explain number of 9800. I don't have Bundler installed, I use a combination of two gems (gs and dep) and I recommend people to do the same. People that go that route are happy with the result, but as I'm not very influential in the Ruby community, that group is rather small.

Tools like npm do a better job at managing dependencies (and fighting complexity) than what RubyGems and Bundler accomplish. In my opinion, we should learn from better tools and strive for a radical change. Instead, we have slowly but steadily perpetrated the idea that RubyGems and Bundler is all we deserve. I'm against including Bundler in any Ruby distribution, because I can't find a reasonable justification to add 9800 lines of code when I can obtain the same result with ~150.

@mkristian
Owner
@drbrain

I am the RubyGems maintainer.

The RubyGems team is working with the Bundler team to merge all of its features and integration tests into RubyGems. Currently RubyGems 2.2.0 is scheduled to ship along with Ruby 2.1.0 with the basic features of bundler and some compatibility. I expect to be largely complete around this time next year with bundler being a small shim around the imported features in RubyGems sometime in the following year.

Since Bundler is scheduled to die in under two years time, would it be prudent to include Bundler in JRuby?

@mdekstrand

@mkristian That assumes you are OK with stomping on whatever the user has in ~/.gem. I'm going for something more self-contained. JRuby and Bundler get me 95% there; my interest in this bug is making the last 5% easy to obtain.

@drbrain That is very good news. 1–2 years feels like a long way off, but it may well not make sense to put bundler in and then take it out again.

@regularfry

All we need is for Rubygems (or another tool) to gain the ability to turn a Gemfile into a Gemfile.lock. Given a Gemfile.lock, installing the locked dependencies is (as far as I've experienced) trivial. Distributing bundler with jruby, or any ruby distribution, would perpetuate needless complexity. Please, no.

@drbrain

@soveran RubyGems 2.1.x and earlier + dep + gs cannot do what Bundler does today as none of the three contain a proper dependency resolver.

@regularfry RubyGems 2.2 is shipping with Gemfile and Gemfile.lock support with Ruby 2.1.0 this Christmas, but full compatibility with Bundler will not (yet) be guaranteed.

@soveran

@drbrain It's a different approach, where a dependency resolver in runtime is no longer needed. I've been using that approach for the past six years, and during all that time I have been the happiest rubyist in town while my friends struggle with Bundler. The idea is to trade space for complexity. Once the gems are installed in isolation, there's no need for dependency resolution in runtime. The node guys have been enjoying that for a long time, and they are very proud of npm, not only because it solves that problem, but also because it is modular: http://andrew.ghost.io/emulating-node-js-modules-in-ruby/

@regularfry

@drbrain As long as I can continue to avoid runtime dependency resolution, I'll be happy. If I can simplify my stack, I'll be happier still.

@timuckun
@fxposter

@soveran Are gs + dep guarantee that only one gem version for every gem is installed? What happens if different gems have the same dependency but with different versions? As long as you have multiple versions of one gem installed - you have to have a runtime setup just like Bundler does to force specific gem versions for all the dependencies. /cc @drbrain

@drbrain

@fxposter for RubyGems 2.1.x and older dep + gs will fail to install a correct set of dependencies because RubyGems does not resolve correctly and neither tool provide a resolver.

@jeltz

@timuckun bundler does not solve the problem of platform specific gems. It solves the problem of having a consistent set of libraries when deploying your application. I do not see how that problem would be different between MRI and Jruby.

@indirect

For the record, I don't think the plan to eventually merge Bundler into Rubygems should impact the decision to include it in JRuby. The version of Rubygems that eventually ships with full Bundler support integrated will fully support Bundler. There shouldn't be big costs associated with the transition from Bundler gem to Bundler in Rubygems — if there are, people will just keep using the gem, and the goals of the merge will have failed. Assuming that we're able to successfully merge everything, it will simply mean upgrading Rubygems versions and not needing the Bundler gem after that.

I'm not sure where the "boo runtime dependency resolution" stuff is coming from, but Bundler doesn't do that (by design) in deployment mode. One of the reasons Bundler exists is that runtime dependency resolution cannot solve some dependency graphs.

In the meantime, if JRuby wants to ship universally used gems anytime sooner than the next two years, I'm happy to assist with anything I can in order to add Bundler to the distribution.

@soveran

@fxposter gs + dep don't guarantee that only one version for every gem is installed, but as you isolate the gemsets, having two versions of the same gem is not likely to happen. If it does happen, you only need to specify a strict version dependency in the .gems manifest. But I think it will be good for people dealing with this problem to try that approach, and even try npm if they want to investigate, as first hand experience is way better than having to trust my words.

@soveran soveran referenced this issue in soveran/cuba
Closed

Add Gemfile. #26

@DavidEGrayson

It seems to me that Bundler should not be included in JRuby because their test suite doesn't work in JRuby. Just yesterday I checked out the Bundler git repository and tried to run their specs with JRuby (under Windows) and it didn't work because it depends on the gems rdiscount and ronn that have C extensions. The C extension support in JRuby is not turned on by default and my understanding is that it will be going away.

It seems to me that the maintainers of JRuby would want to have a way to test every component of the JRuby distribution to make sure it actually works. Bundler is not there yet and I am not sure how much work it will take to get it there (it could be easy).

@drbrain

I believe that those two dependencies are used for building documentation.

RDoc is included in CRuby but cannot be fully built from inside CRuby. Markdown support has separate regular expressions for 1.8 and 1.9 unicode characters which require two versions of ruby to build.

If the parts of Bundler requiring ronn and rdiscount can be built before import will this be a barrier to inclusion?

@timuckun
@indirect

@DavidEGrayson why would the ability to generate the man pages be a blocker for including Bundler in JRuby? JRuby would be including releases, which have the man pages pre-generated, not random source tree commits.

@mkristian as has been extensively pointed out elsewhere, we are happy to add better JRuby support if anyone who actually uses JRuby is willing to supply it. The Bundler team is made up entirely of volunteers with full-time jobs doing things other than working on Bundler, and none of us use JRuby for anything. If you (or the author of that post) have a problem with the current state of things, please do something about it.

@timuckun no. That is years off at the earliest. See also http://andre.arko.net/2013/12/07/the-rumors-of-bundlers-death-have-been-greatly-exaggerated/.

@DavidEGrayson

@indirect You can't run "rake spec" for bundler if you don't have rdiscount and ronn installed, and I don't know how difficult it will be to fix that. The spec task depends on "spec:deps" which tries to install those, and if I remember correctly the "spec" task also tries to load those gems.

Do the official releases of bundler have a different Rakefile?

@indirect

@DavidEGrayson yes, I am perfectly aware of that. You didn't answer my question, though.

Releases of Bundler do not have a different Rakefile.

@DavidEGrayson

I can't speak for the JRuby developers but it seems to me that they should want to run specs in JRuby of any gem they include with JRuby. Being unable to build a manual doesn't seem too important.

@indirect
@DavidEGrayson

OK, sounds good. I was just making a comment about the current state of the Bundler code and why it currently isn't suitable for JRuby. Nobody else had brought up this issue so I thought it was worth saying. I wasn't really aware that you are a Bundler developer and wanting to help. If I had known that, I might have softened the first sentence of my comment to be something like, "It seems to me that Bundler should not be included JRuby until we can get the test suite to run in JRuby."

On another note: I never claimed that generating the manual was important, so it is strange you were asking me about that.

@indirect
@DavidEGrayson

My experience with Bundler's specs were different from yours. I figured I had to run them with "rake spec" so I was unable to run any specs in JRuby, but it sounds like you probably have another way of running them that gets around that, perhaps just running "rspec" in the top directory?

@mkristian
Owner

@indirect I am very very sorry that the link got transported the wrong message. the essence of the article was, if a (linux) distribution wants to use jruby as drop in replacements for ruby it want to be able to compile bundle and run their tests with it. as far I understood (I might mix up info from somewhere else) that is not possible with gentoo. and other distribution as might have similar issues - but I do not know. so my intention was not clear and I SHOULD have added some comment along with the link. again all I can say now that have will be more careful the next time.

about including bundler - you have to look at the jruby_1_7 branch and at the master branch. the master is ruby-2.1 only and will probably follow the respective rubygems version as well and that will have bundler built in. whether it really makes a lot of sense to include it into that branch - IMO there are lots of little pros and cons and no clear yes let's do it.

@drbrain

Remember, there is not yet a RubyGems version that contains all the Bundler features, and won't be for at least two years. I'm pretty sure JRuby will have a release long before RubyGems is a viable replacement for Bundler.

@headius
Owner

Related: bundler/bundler#2895

If we knew bundler specs were green on JRuby it would go a long way toward considering including it as a default gem.

@headius
Owner

@indirect You say the specs are run on JRuby on every commit, but if you are referring to travis it appears that the specs just fail immediately trying to install the aforementioned C extensions: https://travis-ci.org/bundler/bundler/jobs/19072055

My PR would go a long way toward getting specs to run – and run green – on JRuby.

@indirect

@headius When I wrote that, JRuby was newly added to the build, and all the specs were failing because of git. I didn't realize JRuby had additional issues. Thank you for the pull request, and we'll keep working on getting the specs green on JRuby!

@juniorz juniorz referenced this issue from a commit in juniorz/gocd
@juniorz juniorz Vendorize Bundler to jruby-cache's default bundled gem repository (GE…
…M_PATH)

It is necessary because jruby-complete doesn't ship with bundler, and its GEM_HOME/GEM_PATH are inside the jar. More info: jruby/jruby#1146
ced6f58
@juniorz juniorz referenced this issue from a commit in juniorz/gocd
@juniorz juniorz Vendorize Bundler to jruby-cache's default bundled gem repository (GE…
…M_PATH)

It is necessary because jruby-complete doesn't ship with bundler, and its GEM_HOME/GEM_PATH are inside the jar. More info: jruby/jruby#1146
cfd9c08
@Asmod4n

Application Servers without access to the Internet would benefit from having bundler as part of any Ruby Distribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.