Include bundler in JRuby distribution #1146

Open
mdekstrand opened this Issue Oct 21, 2013 · 60 comments

Projects

None yet
@mdekstrand

The JRuby distribution includes a few tools already, such as rake. It'd be very useful for at least some use cases for JRuby to also include bundler.

I'm using JRuby to bootstrap some projects (generally nanoc website builds) so that they can be fully run with nothing but a JDK. Right now, that bootstrap is:

  1. Download jruby (either from jruby.org, or via Maven)
  2. Use JRuby's gem to install bundler somewhere in the local directory
  3. Use the freshly-installed bundle to install the rest of the project's dependencies from the Gemfile

If JRuby's binary distribution and jruby-complete contained Bundler, I could entirely eliminate step 2.

@enebo
Member
enebo commented Oct 21, 2013

I actually think this is a good idea. We used to ship some other gems like rspec and removed them because they were not universal. Bundler is nearly universal in usage on a project of any size. So +1 from me.

@enebo
Member
enebo commented Oct 21, 2013

Of any substantial size...

@headius
Member
headius commented Oct 22, 2013

We have actually moved toward shipping fewer libraries as built in. Currently, I don't think our normal distribution ships anything but jruby-launcher.

However, I could see adding more utilities to jruby-launcher. Rake and Bundler would both be good candidates, as would RSpec. But there's a lot of different possible combinations here and I don't want us to get mired down deciding what should be included...

@enebo
Member
enebo commented Oct 23, 2013

jruby-bin-* does actually install rake already. This might be another error to be fixed perhaps with out Maven changes? The main reason for adding bundler (and I think rake, gem) is that jruby-complete still does not have a nice mechanism for adding specific gems to it past manually re-packing or having a ENV plan. It would make the above scenario very simple. To me it is a balance between which one will generate more complaints :)

@mkristian
Member

well, reading about the bundler idea - it reminded me of
soveran/cuba#20 (comment)

about jruby-complete - how can bundler help there ? (sorry for not seeing
something maybe obvious)

@mdekstrand

@mkristian If jruby-complete contained Bundler, then Bundler would be immediately available in a JRuby downloaded via Maven (my build scripts can reference JRuby via Maven, and use Bundler from the classpath).

@mkristian
Member

I removed the 'automatic install of a bundler version' with the
bundler-maven-plugin and force the pom.xml to reference the bundler version
which is needed. for me it feels less is more ;) i.e. less magic is more
control. guess personal preference.

@mdekstrand

@mkristian Perhaps I should have said 'from Maven' instead of 'via Maven'.

In my particular case, I am using Gradle, but getting JRuby from Maven. Gradle is also self-bootstrapping with gradlew. Or, in older versions of this script, using make and wget.

@timuckun
timuckun commented Dec 2, 2013

+1 Pretty much impossible to deploy a ruby app without bundler these days.

@mkristian
Member

I would like to understand why jruby should include bundler.

if I install MRI on my OS I do not get bundler, I still need to
$ gem install bundler
before I can use bundler.

same holds true if I install any ruby version via RVM or rbenv

even on travis they install ruby and then they install bundler the usual
way, they even need rake to be part of the Gemfile !!!

so what is the rational why jruby needs bundler and all the other rubies do
NOT.

just try to understand that demand ;)

  • christian
@mdekstrand

1 word reason for including bundler: bootstrapping. Specifically, using JRuby to get a working Ruby in an arbitrary/unknown environment with minimal effort.

My use case is bootstrapping projects that use Ruby dependencies. Typically nanoc sites. These sites are used, modified, and deployed in a managed computing environment where we may or may not have Ruby, we may or may not have a Ruby that has adequate facilities to let us build extensions, etc. Users also have crazy-limited home directory quotas, and who knows what gems installed. I don't think the standard Ruby our system administrators give us includes bundler, either.

We do have a working JVM reliably.

One of the beauties of JRuby is that it is effectively self-contained. I can download the JRuby binary zip file and it just works on any Linux, Mac, or even Windows system, so long as it can find a JVM. If my build tool (Gradle, in my case), can talk to Maven repositories, then I can just have it pull in jruby-all and not even have to write that download logic myself.

Once I have Bundler, it takes care of installing nanoc and all the other required gems locally into the project directory. Again, almost zero hassle.

The one hassle in this chain is getting Bundler installed. I have to install it somewhere; if I unpack the JRuby zip file, I can install it there, but it's still another piece. If I am launching JRuby from Maven via Gradle's JavaExec task, I don't really have anywhere to install it except a Bundler-only gem directory in the project or build directory. In which case I have to set up environment variables to tell JRuby where to find Bundler for all subsequent tasks.

If JRuby included Bundler, then boostrapping an app would be crazy-simple:

  1. Download and unpack JRuby (or just require it to be on the classpath, fetched from Maven)
  2. Run bundle install
  3. Enjoy
@mkristian
Member

thank you very much indeed for the detailed explanation. I have to admit
that I do not agree with you fully, if gradle can run

bundle install

then gradle can also run

gem install bundler
bundle install

and I think jruby knows where to install bundler so that the second call
finds it to install the remaining gems.

OK the is my personal option and let's see how the jruby core teams think
about this ;)

regards,
christian

@timuckun
timuckun commented Dec 2, 2013

In my opinion bundler should be a part of the core rails distribution in
the MRI too along with rake and gems. The fact is that it's just not
possible to write and deploy a ruby app without gem, rake and bundler.

In this case I am fooling around with jruby standalone because it allows me
to not have ruby installed on the server at all and I can "bundle" my app
with the ruby it needs.

On Tue, Dec 3, 2013 at 4:48 AM, Christian Meier notifications@github.comwrote:

I would like to understand why jruby should include bundler.

if I install MRI on my OS I do not get bundler, I still need to
$ gem install bundler
before I can use bundler.

same holds true if I install any ruby version via RVM or rbenv

even on travis they install ruby and then they install bundler the usual
way, they even need rake to be part of the Gemfile !!!

so what is the rational why jruby needs bundler and all the other rubies do
NOT.

just try to understand that demand ;)

  • christian


Reply to this email directly or view it on GitHubhttps://github.com/jruby/jruby/issues/1146#issuecomment-29628300
.

@qerub
Contributor
qerub commented Dec 2, 2013

+1

I'm currently "manually re-packing" (as @enebo mentioned) to include Bundler, like this: https://gist.github.com/qerub/7675016

so what is the rational why jruby needs bundler and all the other rubies do NOT.

All rubies should include Bundler, IMHO. :)

But I think it's more important for JRuby to include it because of jruby-complete where it is not installable out-of-the-box:

$ java -jar jruby-complete-1.7.8.jar -S gem install bundler
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the file:/Users/qerub/local/sw/jruby-complete/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/gems/shared directory.

(One has to fiddle with GEM_HOME to make it work as @elehack already said.)

@soveran
soveran commented Dec 3, 2013

Bundler is easy, specially if you have it already installed. Most rubyists already use it daily, so having it as a default looks like a good idea. But on the other hand, you can solve the same problem with simpler tools. The lines of code count for Bundler yields a hard to explain number of 9800. I don't have Bundler installed, I use a combination of two gems (gs and dep) and I recommend people to do the same. People that go that route are happy with the result, but as I'm not very influential in the Ruby community, that group is rather small.

Tools like npm do a better job at managing dependencies (and fighting complexity) than what RubyGems and Bundler accomplish. In my opinion, we should learn from better tools and strive for a radical change. Instead, we have slowly but steadily perpetrated the idea that RubyGems and Bundler is all we deserve. I'm against including Bundler in any Ruby distribution, because I can't find a reasonable justification to add 9800 lines of code when I can obtain the same result with ~150.

@mkristian
Member

will just nicely install it in default location of your home directory. no
need for GEM_HOME and GEM_PATH

On Mon, Dec 2, 2013 at 10:15 PM, Christoffer Sawicki <
notifications@github.com> wrote:

$ java -jar jruby-complete-1.7.8.jar -S gem install bundler
ERROR: While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the file:/Users/qerub/local/sw/jruby-complete/jruby-complete-1.7.8.jar!/META-INF/jruby.home/lib/ruby/gems/shared directory.

(One has to fiddle with GEM_HOME to make it work as @elehackhttps://github.com/elehackalready said.)

$ java -jar jruby-complete-1.7.8.jar -S gem install bundler --user-install

will just nicely install it in default location of your home directory. no
need for GEM_HOME and GEM_PATH.

@drbrain
drbrain commented Dec 3, 2013

I am the RubyGems maintainer.

The RubyGems team is working with the Bundler team to merge all of its features and integration tests into RubyGems. Currently RubyGems 2.2.0 is scheduled to ship along with Ruby 2.1.0 with the basic features of bundler and some compatibility. I expect to be largely complete around this time next year with bundler being a small shim around the imported features in RubyGems sometime in the following year.

Since Bundler is scheduled to die in under two years time, would it be prudent to include Bundler in JRuby?

@mdekstrand

@mkristian That assumes you are OK with stomping on whatever the user has in ~/.gem. I'm going for something more self-contained. JRuby and Bundler get me 95% there; my interest in this bug is making the last 5% easy to obtain.

@drbrain That is very good news. 1–2 years feels like a long way off, but it may well not make sense to put bundler in and then take it out again.

@regularfry

All we need is for Rubygems (or another tool) to gain the ability to turn a Gemfile into a Gemfile.lock. Given a Gemfile.lock, installing the locked dependencies is (as far as I've experienced) trivial. Distributing bundler with jruby, or any ruby distribution, would perpetuate needless complexity. Please, no.

@drbrain
drbrain commented Dec 3, 2013

@soveran RubyGems 2.1.x and earlier + dep + gs cannot do what Bundler does today as none of the three contain a proper dependency resolver.

@regularfry RubyGems 2.2 is shipping with Gemfile and Gemfile.lock support with Ruby 2.1.0 this Christmas, but full compatibility with Bundler will not (yet) be guaranteed.

@soveran
soveran commented Dec 3, 2013

@drbrain It's a different approach, where a dependency resolver in runtime is no longer needed. I've been using that approach for the past six years, and during all that time I have been the happiest rubyist in town while my friends struggle with Bundler. The idea is to trade space for complexity. Once the gems are installed in isolation, there's no need for dependency resolution in runtime. The node guys have been enjoying that for a long time, and they are very proud of npm, not only because it solves that problem, but also because it is modular: http://andrew.ghost.io/emulating-node-js-modules-in-ruby/

@regularfry

@drbrain As long as I can continue to avoid runtime dependency resolution, I'll be happy. If I can simplify my stack, I'll be happier still.

@timuckun
timuckun commented Dec 3, 2013

One reason bundler exists in the first place is because the MRI is not
Jruby. With jruby I know that every gem will work on every platform but
with the MRI any gem that requires a C lib has to be compiled on the target
platform.

In a way the real answer to what I am trying to do is warbler but I
recently watched a talk at a ruby conference where the presenter made the
point that rails doesn't want to live in a jar, it really needs to be in a
directory structure for it work properly.

Maybe what is really needed is a jruby specific framework which takes
advantage of all the strengths of Jruby.

@fxposter
fxposter commented Dec 3, 2013

@soveran Are gs + dep guarantee that only one gem version for every gem is installed? What happens if different gems have the same dependency but with different versions? As long as you have multiple versions of one gem installed - you have to have a runtime setup just like Bundler does to force specific gem versions for all the dependencies. /cc @drbrain

@drbrain
drbrain commented Dec 3, 2013

@fxposter for RubyGems 2.1.x and older dep + gs will fail to install a correct set of dependencies because RubyGems does not resolve correctly and neither tool provide a resolver.

@jeltz
jeltz commented Dec 3, 2013

@timuckun bundler does not solve the problem of platform specific gems. It solves the problem of having a consistent set of libraries when deploying your application. I do not see how that problem would be different between MRI and Jruby.

@indirect
indirect commented Dec 3, 2013

For the record, I don't think the plan to eventually merge Bundler into Rubygems should impact the decision to include it in JRuby. The version of Rubygems that eventually ships with full Bundler support integrated will fully support Bundler. There shouldn't be big costs associated with the transition from Bundler gem to Bundler in Rubygems — if there are, people will just keep using the gem, and the goals of the merge will have failed. Assuming that we're able to successfully merge everything, it will simply mean upgrading Rubygems versions and not needing the Bundler gem after that.

I'm not sure where the "boo runtime dependency resolution" stuff is coming from, but Bundler doesn't do that (by design) in deployment mode. One of the reasons Bundler exists is that runtime dependency resolution cannot solve some dependency graphs.

In the meantime, if JRuby wants to ship universally used gems anytime sooner than the next two years, I'm happy to assist with anything I can in order to add Bundler to the distribution.

@soveran
soveran commented Dec 4, 2013

@fxposter gs + dep don't guarantee that only one version for every gem is installed, but as you isolate the gemsets, having two versions of the same gem is not likely to happen. If it does happen, you only need to specify a strict version dependency in the .gems manifest. But I think it will be good for people dealing with this problem to try that approach, and even try npm if they want to investigate, as first hand experience is way better than having to trust my words.

@soveran soveran referenced this issue in soveran/cuba Dec 5, 2013
Closed

Add Gemfile. #26

@DavidEGrayson
Contributor

It seems to me that Bundler should not be included in JRuby because their test suite doesn't work in JRuby. Just yesterday I checked out the Bundler git repository and tried to run their specs with JRuby (under Windows) and it didn't work because it depends on the gems rdiscount and ronn that have C extensions. The C extension support in JRuby is not turned on by default and my understanding is that it will be going away.

It seems to me that the maintainers of JRuby would want to have a way to test every component of the JRuby distribution to make sure it actually works. Bundler is not there yet and I am not sure how much work it will take to get it there (it could be easy).

@drbrain
drbrain commented Dec 8, 2013

I believe that those two dependencies are used for building documentation.

RDoc is included in CRuby but cannot be fully built from inside CRuby. Markdown support has separate regular expressions for 1.8 and 1.9 unicode characters which require two versions of ruby to build.

If the parts of Bundler requiring ronn and rdiscount can be built before import will this be a barrier to inclusion?

@timuckun
timuckun commented Dec 8, 2013

On Mon, Dec 9, 2013 at 12:33 PM, Christian Meier
notifications@github.comwrote:

https://blog.flameeyes.eu/2012/03/bundler-and-jruby-a-ruby-rant

He makes some good points For most people putting the tests with the gems
seems irrelevant. In the comments of that post Charles also makes a good
point about the meta gem.

Anyway it seems like a moot point now. Apparently the movement is afoot to
combine gems and bundler.

@indirect
indirect commented Dec 9, 2013

@DavidEGrayson why would the ability to generate the man pages be a blocker for including Bundler in JRuby? JRuby would be including releases, which have the man pages pre-generated, not random source tree commits.

@mkristian as has been extensively pointed out elsewhere, we are happy to add better JRuby support if anyone who actually uses JRuby is willing to supply it. The Bundler team is made up entirely of volunteers with full-time jobs doing things other than working on Bundler, and none of us use JRuby for anything. If you (or the author of that post) have a problem with the current state of things, please do something about it.

@timuckun no. That is years off at the earliest. See also http://andre.arko.net/2013/12/07/the-rumors-of-bundlers-death-have-been-greatly-exaggerated/.

@DavidEGrayson
Contributor

@indirect You can't run "rake spec" for bundler if you don't have rdiscount and ronn installed, and I don't know how difficult it will be to fix that. The spec task depends on "spec:deps" which tries to install those, and if I remember correctly the "spec" task also tries to load those gems.

Do the official releases of bundler have a different Rakefile?

@indirect

@DavidEGrayson yes, I am perfectly aware of that. You didn't answer my question, though.

Releases of Bundler do not have a different Rakefile.

@DavidEGrayson
Contributor

I can't speak for the JRuby developers but it seems to me that they should want to run specs in JRuby of any gem they include with JRuby. Being unable to build a manual doesn't seem too important.

@indirect

I've already offered to help the JRuby team with any integration changes that they need. I no longer understand what you are objecting to.

On Mon, Dec 9, 2013 at 5:52 PM, David Grayson notifications@github.com
wrote:

I can't speak for the JRuby developers but it seems to me that they should want to run specs in JRuby of any gem they include with JRuby. Being unable to build a manual doesn't seem too important.

Reply to this email directly or view it on GitHub:
#1146 (comment)

@DavidEGrayson
Contributor

OK, sounds good. I was just making a comment about the current state of the Bundler code and why it currently isn't suitable for JRuby. Nobody else had brought up this issue so I thought it was worth saying. I wasn't really aware that you are a Bundler developer and wanting to help. If I had known that, I might have softened the first sentence of my comment to be something like, "It seems to me that Bundler should not be included JRuby until we can get the test suite to run in JRuby."

On another note: I never claimed that generating the manual was important, so it is strange you were asking me about that.

@indirect

Generating the manual is the only thing in the spec suite that doesn’t work on JRuby, as far as I know. That’s why I’m so confused about your insistence that Bundler isn’t suitable for inclusion.

Incidentally, the Bundler tests are run against JRuby as part of every commit to Bundler (albeit not passing at this moment, but that’s due to a git upgrade causing problems, I think).

On Dec 9, 2013, at 6:16 PM, David Grayson notifications@github.com wrote:

OK. I was just making a comment about the current state of the Bundler code and why it currently isn't suitable for JRuby. Nobody else had brought up this issue so I thought it was worth saying. I wasn't really aware that you are a Bundler developer and wanting to help. If I had known that, I might have softened the first sentence of my comment to be something like, "It seems to me that Bundler should not be included JRuby until we can get the test suite to run in JRuby."

On another note: I never claimed that generating the manual was important, so it is strange you were asking me about that.


Reply to this email directly or view it on GitHub.

@DavidEGrayson
Contributor

My experience with Bundler's specs were different from yours. I figured I had to run them with "rake spec" so I was unable to run any specs in JRuby, but it sounds like you probably have another way of running them that gets around that, perhaps just running "rspec" in the top directory?

@mkristian
Member

@indirect I am very very sorry that the link got transported the wrong message. the essence of the article was, if a (linux) distribution wants to use jruby as drop in replacements for ruby it want to be able to compile bundle and run their tests with it. as far I understood (I might mix up info from somewhere else) that is not possible with gentoo. and other distribution as might have similar issues - but I do not know. so my intention was not clear and I SHOULD have added some comment along with the link. again all I can say now that have will be more careful the next time.

about including bundler - you have to look at the jruby_1_7 branch and at the master branch. the master is ruby-2.1 only and will probably follow the respective rubygems version as well and that will have bundler built in. whether it really makes a lot of sense to include it into that branch - IMO there are lots of little pros and cons and no clear yes let's do it.

@drbrain
drbrain commented Dec 10, 2013

Remember, there is not yet a RubyGems version that contains all the Bundler features, and won't be for at least two years. I'm pretty sure JRuby will have a release long before RubyGems is a viable replacement for Bundler.

@headius
Member
headius commented Feb 22, 2014

Related: bundler/bundler#2895

If we knew bundler specs were green on JRuby it would go a long way toward considering including it as a default gem.

@headius
Member
headius commented Feb 22, 2014

@indirect You say the specs are run on JRuby on every commit, but if you are referring to travis it appears that the specs just fail immediately trying to install the aforementioned C extensions: https://travis-ci.org/bundler/bundler/jobs/19072055

My PR would go a long way toward getting specs to run – and run green – on JRuby.

@indirect

@headius When I wrote that, JRuby was newly added to the build, and all the specs were failing because of git. I didn't realize JRuby had additional issues. Thank you for the pull request, and we'll keep working on getting the specs green on JRuby!

@juniorz juniorz added a commit to juniorz/gocd that referenced this issue May 5, 2014
@juniorz juniorz Vendorize Bundler to jruby-cache's default bundled gem repository (GE…
…M_PATH)

It is necessary because jruby-complete doesn't ship with bundler, and its GEM_HOME/GEM_PATH are inside the jar. More info: jruby/jruby#1146
ced6f58
@juniorz juniorz added a commit to juniorz/gocd that referenced this issue May 5, 2014
@juniorz juniorz Vendorize Bundler to jruby-cache's default bundled gem repository (GE…
…M_PATH)

It is necessary because jruby-complete doesn't ship with bundler, and its GEM_HOME/GEM_PATH are inside the jar. More info: jruby/jruby#1146
cfd9c08
@Asmod4n
Asmod4n commented Sep 4, 2014

Application Servers without access to the Internet would benefit from having bundler as part of any Ruby Distribution.

@kares
Member
kares commented Oct 10, 2016

@Asmod4n except what would be the point if there's no internet access and thus now way to install gems :) ?

@Asmod4n
Asmod4n commented Oct 10, 2016

@kares you could have a local repository of gems in your intranet.

@kares
Member
kares commented Oct 10, 2016 edited

... including the bundler gem since its for the best using the very same version the bundle was installed with.

@headius
Member
headius commented Oct 11, 2016

From @drbrain's assessment in late 2013:

Remember, there is not yet a RubyGems version that contains all the Bundler features, and won't be for at least two years.

Well, we're closing in on three years now and RG still does not include all of (or even most of) Bundler's features, right @indirect @drbrain?

I'd like to see what MRI folks think about including Bundler as a default gem. It has become nearly as indispensable as RubyGems itself, and yet it isn't shipped in the box.

@kares
Member
kares commented Oct 11, 2016

even if MRI folks think its fine I still feel like its not that definite for JRuby - most of the times its fine but :

  • even lately Bundler (1.13.1) was released with a JRuby specific bug (they only test against MRI?)
  • JRuby does some RGs extensions, Bundler patches RGs heavily - potential conflict is always near
  • there has been problems (on JRuby) working around default gems - believe there's still left overs

... if Bundler is included JRuby should strongly consider running their suite (at the given release tag)

@enebo
Member
enebo commented Oct 11, 2016

@kares I think your points are true but regardless whether we ship bundler or not we have to deal with these issues now. We cannot live for too long without working bundler support and retain users. I definitely agree we need to do something about having CI for bundler to know when something breaks sooner than later.

I am still somewhat for this idea since RG still does not subsume the important bits of bundler yet and we will end up shipping a version of bundler we are confident in rather than letting people install the latest (and possibly have an immediate negative response).

@kares
Member
kares commented Oct 12, 2016

I am all for it, just that I do have concerns with the current state of RG/Bundler. default gem issues might be most critical - recall there has been an issue where it prohibited using an explicitly installed gem version (of jossl) and always loaded the default - packed version.

@mkristian
Member

@kares isn't there even a test for the case which you just mentioned about the default gem. maybe not in the main test suite but should rather easy to add, that an downgraded version of bundler is really picked.

@kares
Member
kares commented Oct 12, 2016

@mkristian unsure but I know @etehtsea has been struggling lately with some default gem behavior.

@mkristian
Member
mkristian commented Oct 12, 2016 edited

@kares is there more than #4083 which is clearly a problem ?

@etehtsea
Contributor

@kares @mkristian to be honest I can't reproduce this anymore. Maybe there was some local issue.

@kares
Member
kares commented Oct 12, 2016

@mkristian @etehtsea that was the one on my mind - thank you both for pointing me into directions...
guess my concerns are no longer relevant than ... 🚸 looking forward for anyone attempting the Bundler packaging setup ... maybe for 9.2? I will check (or add) 4 a gem default test case @mkristian mentioned.

@enebo enebo modified the milestone: Won't Fix Feb 17, 2017
@enebo
Member
enebo commented Feb 17, 2017

I was going to resolve this based on not everyone agreeing we should include it but most people seem to support including it.

I guess our main issue is operationally knowing whether a particular version of bundler is working well enough for us to include it in the distribution? This obviously is an issue whether we include it or not but we have more pressure to make sure the particular version we include is working if we plan on endorsing it by putting in the distro...

@kares
Member
kares commented Feb 18, 2017 edited

still resisting a bit ... just hit a regression in their point release (1.14.4) that only seemingly broke in a .war
... to be fair its JRuby 1.7 but they claim support for 1.9.3 the work-around is not that straight-forward as you can not really gem manage bundler itself (chicken-egg prob). having it in would mean a small disaster 😜

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment