add ability to define dependencies based on ruby engine

[rkh]

Currently there is a big issue with Rubinius: The standard library is now all in gems. This means manipulating the gem path and tools like gemsets, bundler will make these unavailable. This can be fixes by adding rubysl and other gems (like json) to the Gemfile. However, this does not really work for libraries, as dependencies do not get propagated by bundler. All in all, this means the end user for the library has to revert back to manual dependency resolution. This is not only annoying for Ruby developers, but comes very painful to users not familiar with Ruby.

I could add rubysl-* as a dependency to to my gem, but first off, not all of these install cleanly on other Ruby implementations, and second even if they would, it would only cause duplication.

So it would be great if there was the ability to define dependencies for a specific Ruby engine.

I know that this is an issue mainly caused by the decisions made by Rubinius, but they keep pointing me to RubyGems for a fix.

This is especially painful for me personally for a few reasons:

• I really care about Rubinius, but as a maintainer of many gems, the only real alternative I see is dropping official Rubinius support.
• Travis CI is currently getting a very high rate of support tickets from customers that suddenly have their projects fail on Rubinius. It is hard for us to keep up and we see people simply disabling tests on rbx.

If I could have opened this issue for two repos at once, I would.

/cc @evanphx @brixen @dbussink

[headius]

This seems like the wrong approach. Almost every gem is going to need something from stdlib, which means almost every gem will need a Rubinius-specific set of dependencies just to be runnable on Rubinius. And since several rubysl gems are Rubinius-specific, if MRI and JRuby ever spin gems off the same way we would see gems with two or three impl-specific dependency sets.

This is something that should be solved in Rubinius. All the gems in the world were written with the assumption that there would be a set of always-available standard libraries they could count on. Rubinius has removed those libraries from being "standard" and forced everyone to explicitly depend on upon them. Either all the gems in the world must change solely to accommodate Rubinius, or Rubinius should make it possible for those gems to work as-is.

[headius]

I do want to say I support having impl-specific (engine-specific) dependencies for gems in the wild that do not or will not support multiple platforms (e.g. before the json gem had a -java version, users had no way to depend on it and work on JRuby even though json-jruby gem existed). There are valid cases for having engine-specific dependencies; the standard library shouldn't be one of them.

[drbrain]

I believe @indirect's new indexer work will lay the groundwork for supporting per-engine dependencies, but I don't believe it will be ready for use in a release of RubyGems for several months.

Add another six months to a year beyond for adoption of that release and this is not a problem that will quickly solve itself through the efforts of RubyGems.

Add many more months beyond that to convince authors to update their gems to add Rubinius-only dependencies (and let's not forget the work to get inter-compatible versions of the gems an application requires released).

For the short-term problems of Rubinius I see no solution beyond Rubinius providing files from the standard library the same way Ruby does.

[brixen]

This is not a Rubinius problem. It is a deficiency in RubyGems that has existed for nearly a decade (surely since 2001 when JRuby was started.) And there are tons of JRuby specific dependencies in gems that only work because JRuby claims to be a different "platform", something that (as in RUBY_PLATFORM) historically meant OS+arch.

Already json, psych, racc, and minitest are in gems and need to be used as gems because changes are made more frequently than MRI releases. Additionally, minitest 5.0 explicitly breaks compatibility with the API that test/unit uses, so the minitest version that is bundled with MRI must remain at 4.x or be hidden in test/unit or test/unit deprecated or ... These problems are not going away.

There has not been a reason to have a "standard library" in Ruby since RubyGems was created, other than the fact that MRI's core library is severely deficient and MRI refused to bundle RubyGems for years. Furthermore, the "standard library" is some of the most poorly maintained Ruby code around. There are probably a dozen gems for using HTTP in Ruby because of dissatisfaction with the "standard library" version. Instead of piling more crap and cruft and special cases on years of brokenness, we can get stuff fixed ASAP. If this requires Rubinius to fork RubyGems, so be it. If @indirect needs help getting the indexer finished and into a version of RubyGems, we can help.

[brixen]

Also, note that there is nothing that need be Rubinius-specific in RubySL. It's merely a work in progress and everything could run right now in MRI with a few FFI tweaks. So while @headius continues to complain, the only thing he offers to bring to RubySL is Java code, which is exactly the problem with the existing Ruby "standard library". It's not even Ruby in many places.

The one thing that fixing this deficiency in RubyGems would do for RubySL is allow JRuby to maintain its Java gems and these would be used on JRuby simply because they would resolve for that engine.

[drbrain]

This is not an appropriate place to post off-topic rants if they don't contain links to patches or issues to fix said complaints. Please restrict such comments to the Rubinius tracker and/or blog.

[rkh]

The one thing that fixing this deficiency in RubyGems would do for RubySL is allow JRuby to maintain its Java gems and these would be used on JRuby simply because they would resolve for that engine.

I guess the whole thing would also be solvable for me if instead of engine specific dependencies, I could do engine specific releases (ie, sinatra/travis gem for Rubinius).

[drbrain]

Rubinius declares itself as the "rubinius" platform, just like jruby declares itself as the "java" platform:

https://github.com/rubinius/rubinius/blob/master/library/rubygems/defaults/rbx.rb#L1-L4

You can check the platforms your ruby claims with gem env platforms.

[drbrain]

@rkh … you can use the platform for rubinius-specific releases:

Gem::Specification.new 'sinatra', '1.whatever' do |s|
  # …
  s.platform = Gem::Platform.new %w[universal rubinius]
end

[brixen]

@drbrain everything I wrote is relevant to the @rkh's statements in the issue or arguments against fixing this long-standing deficiency in RubyGems.

Use of platform does not fix the problem. Different versions of Rubinius (and other Ruby implementations) will depend on different versions of gems. The only viable solution is to add engine and engine version to dependency specification.

[drbrain]

@brixen until this feature is implemented I'm not going to throw my hands up in the air and say "oh well!", instead I will provide workarounds. Currently the best workaround is to use the platform that Rubinius exposes as a proxy for the lack of engine-specific dependencies.

Presently I have no priority for implementation of this feature (it's scheduled under milestone "Future"). There are many long-standing deficiencies in RubyGems, but I prioritize issues that help the most people. This makes merging bundler features my primary goal after bug fixes that are introduced through regular development of RubyGems.

If you would like to accelerate the timeline of this feature I can assist you in making a patch. Note that this feature will require support in the APIs RubyGems uses to determine the dependencies of gems, so you will also need to coordinate with @indirect's new index/API efforts.

[headius]

This is not a Rubinius problem. It is a deficiency in RubyGems that has existed for nearly a decade (surely since 2001 when JRuby was started.)

RubyGems did not exist in 2001. According to http://onestepback.org/articles/rubygemsfacets/genesisthemidnighthackers.html, RubyGems grew out of a hackfest at RubyConf 2003.

When I joined the project in 2005, we opted to use "java" platform for JRuby gems with native extensions...

And there are tons of JRuby specific dependencies in gems that only work because JRuby claims to be a different "platform", something that (as in RUBY_PLATFORM) historically meant OS+arch.

...and the vast majority of "java" platform gems do ship a native extension written in Java or some other non-Ruby JVM language. A few of them use JRuby's FFI-like JVM language integration to call out to non-Ruby libraries. In either case, "java" is exactly right for JRuby's RUBY_PLATFORM, and I don't see it as a hack or a hijack of any kind.

I do agree that platform is not something that should be overloaded to handle Ruby engine. JRuby's RUBY_ENGINE is 'jruby', but its RUBY_PLATFORM is 'java'. They're two different things.

These problems are not going away.

Indeed. I hope to work to improve the state of ruby_core stdlib for 2.2, pulling more libraries out as gems and incorporating API-compatible (within reason) libraries to replace some that are showing their age and lack of maintenance. However, I don't see how this relates to needing engine-specific dependencies; if, as you say, there are better pure-Ruby alternatives to stdlib libraries, gem authors can depend on them regardless of engine. What's different here is that any gem that requires standard libraries must now do extra work to run properly on Rubinius...work they didn't have to do just a couple months ago, and work they don't have to do on other Ruby implementations.

So while @headius continues to complain, the only thing he offers to bring to RubySL is Java code, which is exactly the problem with the existing Ruby "standard library". It's not even Ruby in many places.

For gems that are just Ruby, there's no problem. For gems that include native extensions or engine-specific code -- of which there are many under the "rubysl" meta-gem, some of which even contain Rubinius-specific code -- it seems perfectly valid to provide platform-differentiated versions for JRuby.

You argue against having 'java' versions of rubysl gems...and then you release rubysl gems that have C code that does not work on implementations without MRI's C API (JRuby, MacRuby, IronRuby, Topaz...). It seems illogical to require users to add separate dependencies per impl when a single dependency with multiple platforms (or engines) would suffice. Bottom line: gems with native code should install the right native code for the target implementation, without the user having to add a dependency per implementation.

I will again state that I support having engine as a dimension of gem versioning and/or dependency. I just don't think it's appropriate to require all gem authors to add stdlib dependencies just because one implementation unilaterally moved all the Ruby stdlib into gems.

I understand that you believe this was the right decision for Rubinius, but the infrastructure of gems obviously doesn't support this (at present) and developers are building libraries expecting an always-available stdlib. Just because Rubinius made this move doesn't mean RubyGems is broken or gem authors are wrong; it means Rubinius chose a new path unsupported by the current Ruby ecosystem, and as a result Rubinius will have to justify that move and work to help evolve the ecosystem. I'm standing by to help with that work.

[copiousfreetime]

Making a note here that doing dependencies based upon RUBY_ENGINE should probably be done in conjunction with doing dependencies based upon RUBY_PLATFORM (see #628).

[djberg96]

See #1468

[jrochkind]

I'm guessing this is not going to be happening?

[eregon]

Somewhat related, RUBY_ENGINE should be considered when installing pre-compiled gems: #2945