rubygems creates ruby wrappers around non-ruby executables, causing them to fail

[olbrich]

when providing executables with a gem, they get wrapped in a ruby wrapper that provides some support for using previous
versions of a gem. If the executable is not a ruby file (e.g., a bash script, php, etc..) then the wrapper will fail.

Suggestions:

• allow the gemspec to force the gem to be installed with '--no-wrapper' and leave it to the developer to properly
  wrap executables.
• check the shebang line of the executable and skip wrapping for anything that explicitly uses something other than
  ruby
• provide a warning at the time the gem is built that files will be wrapped (check for .bat files, or shebangs that don't include ruby... perhaps not perfect, but it will probably catch most cases).
• document the existing behavior better and provide instructions for work arounds for non-ruby binaries.

[olbrich]

Original rubyforge ticket

[zenspider]

As stated in the previous ticket:

I don't know of a good way to do this that works across platforms and makes sense.

Rubygems is for ruby. If a file is in your gem's bindir and listed as a gem executable, it should be ruby. If your gem has "special needs" then it should do it out of band, like create a Rakefile that does your special steps for you and list the Rakefile as an extension in the spec. Or just use a post-install message telling people to install with --no-wrapper.

Lemme know if I didn't understand something.

[zenspider]

I haven't heard back in months and I still think this is a bad idea. Closing.

[steveklabnik]

One use case:

Shoes 4 is written in JRuby. We need to pass options on startup to the JVM on the OSX platform only. Normally we'd do this via a bash script, but we can't, and since it's Ruby, the best we could do is spin up another Ruby, which would make the terrible JVM startup time absolutely unbearable.

[kigster]

@steveklabnik @zenspider @olbrich @hugogiraudel I just bumped into another valid reason in favor of allowing gem developers to install non-ruby executables automatically.

I am unaware of any way that a child process can affect the parent's process environment on UNIX, in particular, user's current directory. This is no different from Ruby scripts. However, were it a shell script, you can simply eval a string containing commands, and affect the parent shell process, including changing your current directory.

I am building a ruby version of the wd (warp directory) tool that's been available for zsh for quite some time now. I love the concept, and want to expand on it, but the core functionality of this tool is to be able to change the directory of the user that runs wd in the first place. It's easy to do in a shell, but impossible to do in Ruby.

I am currently shipping two executables with the gem: warp-dir – is a ruby file that runs the gem's code. To work around the limitation of not being able to change the directory of the parent process, I am having the ruby code generate shell compatible executable code (even for printing help). Talk about mete-meta programming :) So warp-dir ruby script is not meant to be run directly by humans. You are supposed to do something like this: eval $(warp-dir --help). So the wd executable is a tiny script that provides this functionality. And it would be lovely to have it be installed without ruby wrappers automatically. Gem user should really not worry about this type of thing.

I am glad that it looks like I can choose --no-wrapper flag while installing the gem, but it would be much more preferable if the gem author could explicitly tell rubygems not to create wrappers regardless.

One possible way to do this in a backwards compatible way, is by introducing a new config option for such scripts:

  gem.files         = `git ls-files`.split($\).reject{ |f| f =~ /^doc\// }
  gem.executables   << 'warp-dir'
  gem.non_ruby_executables   << 'wd'
  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})

By using non_ruby_executables we could add a new feature without breaking any prior expectations.

My 2c.

[mheffner]

If this is the case, it would be great to verify this at packaging time and reject the executable. It was somewhat unexpected surprise that it blew up later after installation.

[ndbroadbent]

Sorry for commenting on such an old issue, but would it be possible to add something like this non_ruby_executables option?

I'm working on this PR for rubocop-daemon, which makes RuboCop much faster. It's incredibly fast when I use a bash script to talk to the daemon server using netcat, so I wanted to install this bash script as an executable.

In the meantime, how I could I provide this bash script to people who install the rubocop-daemon gem? Maybe a "post-install" script? I don't want to wrap it with Ruby, because this adds some extra latency.

[deivid-rodriguez]

I think this is unlikely to be implemented at the moment, but I don't mind reopening this ticket 🤷‍♂️.

As a workaround, I suggest you ship the script in a different location, and tell users to copy it to a project binstub in a post install message.

[iboB]

@deivid-rodriguez, sure, I'm working on such a workaround and others with the same issues have implemented their own. But allowing the packaging of helper non-ruby executables has the potential to make gems much more powerful and easy to use

[kigster]

@deivid-rodriguez

Implementing support for non_ruby_executable would probably take an experienced Rubyist a couple of hours at most.

• Clearly, enough people think this is a useful feature.
• Clearly, shell wrappers represent the lion share of such executables.
• Why not embrace this, and make everyone's life easier?

[iboB]

IMO from the proposed solutions there are three which I would qualify as almost identical in terms of how useful they are:

• Force --no-wrapper from gemspec
• Check shebang for each provided executable and don't wrap if it's not ruby
• I just thought of a combination of the above: Look for a second comment line after the shebang and if it's # no-wrap, don't wrap that particular executable
• non_ruby_executables in gemspec

They can probably satisfy 99% of the needs (and the needs are in <1% of all gems). However this means that for a multi-platform gem (0.01% of all cases), we'd end-up with /some/path/bin/gem.bat or gem.cmd which are the windows versions of the executable. This is clutter and not very pleasant. Plus it would be hard to provide special different custom executables for, say, mac, linux, and unix (you need to work around this with a shell script which first identifies the system and then executes the real code)

I think a truly powerful solution will allow us to provide custom install code. A hook to be executed after gem install has been called and all "classic" behavior is complete.

Still, if one of the first list is easier, even if there is will to implement the Truly Powerful Solution ™, we could absolutely start with one of the first. I can see value in having both the powerful solution and one of the first ones

[deivid-rodriguez]

@iboB From the solutions proposed, the one most appealing to me is "Check shebang for each provided executable and don't wrap if it's not ruby", since it doesn't require us adding any new feature or changing the gemspec specification. If someone can work on that, I'm happy to review such a change.

[iboB]

If no one starts or completes this by next week, I think I'll have time to try to whip up a PR. I'll write here if I start working on this

[iboB]

So, I started here: https://github.com/iboB/rubygems/tree/issue_88

The functionality is done in the first commit. It's like 5 lines of code. Now all that's left is the remaining 99% in tests and docs 🤣

[deivid-rodriguez]

@iboB Thanks for working on this 🎉. Two early comments:

• I believe already some "ruby shebang" detection code around. It'd be great if you could reuse it somehow.
• Let's focus on the bug fix for now, and leave the feature (that "rubygems: no-wrap" magic comment) for a separate PR and discussion).

[deivid-rodriguez]

Hi @iboB, I'm trying to catch up with old issues. I'm in general in favour of this, but I would fix only the "non ruby shebang case" for now and wait for anybody to request the other feature. As far as I know, none of the use cases mentioned here are trying to ship a batch script with a gem, so I'd say we can attend that problem when it becomes a problem for someone? I'm just overwhelmed by the amount of features we already need to deal with, so I'd like to minimize the changes we make.

[iboB]

@deivid-rodriguez I am. It is a problem for me. The whole reason I got involved with this issue is because I am trying to ship batch files with my gem.

[kigster]

@deivid-rodriguez @iboB I thought I was in the same boat, with my gem warp-dir. It's a gem that's compatible with ZSH's wd. It installs a command wd (which is a BASH function that invokes gem's executable) and allows you to create and navigate straight to the directory bookmarks on the file system. I initially thought that I needed to ship wd as a shell script because I realized that on UNIX a sub-process can not change the directory for the parent process. But then I realized that if wd is a shell script, and it's executed as a shell script, it will still run in a sub-process, so the same restriction applies. Therefore the only way I solved this is to ask the user to install the shell function with warp-dir install --dotfile ~/.bashrc or whatever dotfile.

TL;DR

1. I no longer feel I have the need to install BASH files into gem's bin folder.
2. I suspect that any use-case that claims this requirement, will find an alternative elegant solution that only installs ruby into the Gems' bin folder.
3. Furthermore, I suspect that allowing arbitrary executables to be installed with gems will create all sorts of new attack vectors for malicious software. Can I build and install a C-binary together with gem as well?
4. With the recent addition of ruby rules to Bazel Build System, adding noni-ruby binaries would likely complicate the integration. The system is already very complicated since the goal is to create a completely hermetically sealed installation of a Ruby command with all of its supporting dependencies, including Ruby itself, standard library, and the gems, in the temporary build folder.

[iboB]

I still think it's ridiculous that adding a custom executable is impossible with gems and I definitely wouldn't call the workarounds and jumps through hoops people have created because of this "elegant".

I would also allow shipping binary executables, yes. In most cases it's a bad idea but certain uses may exist. More power to the developers!

As for this being an attack vector... how? You can ship absolutely anything right now. Anything except non-ruby executables in the bin folder. How is also allowing other types of executables more dangerous? It's only more convenient to do certain things, but a malicious attacker won't be stopped by inconvenience.

[deivid-rodriguez]

@iboB I think I'd rather use the rules for detecting Windows Batch scripts that you mentioned before rather than introducing a rubygems specific magic comment. The rules you mentioned exactly match Apache Tika's rules: https://github.com/apache/tika/blob/41a99cce34f90259ad8775cb8abbdbcc19e7ca27/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L80-L90. So it seems like a common criteria to use. What do you think?

[iboB]

These particular rules don't match the second-most common first line for batch files which is colon-colon. Valid ruby programs whose first line begins with rem or @echo off are also possible.

Still I don't quite understand what's wrong with the magic string. What exactly are we trying to prevent by not allowing it? I also don't understand why not allow optional opt-out of wrap even for ruby scripts

• It's future proof: Not all shells are batch or bash
• It's backwards compatible: There are no unwrapped scripts, so there's nothing to break
• It allows gems to be used for all kinds of packages instead of forcing users to jump through hoops.
• It's not more insecure: It doesn't introduce new attack vectors. You can easily ship your malicious code and call it from a by-the-book-honestly-wrapped script today. What's the difference?

I fail to see a single objective reason to be against the magic string.

The reason might be "I, the dictator of rubygems, am against it because the stars tell me so!". If this is the case, I will not whip up some broken batch detector, which will be full of false positives, or false negatives, or both, and will instead resign from this issue.

[deivid-rodriguez]

I'm just discussing and thinking what's best, there's no need for that tone.

[iboB]

Apologies for the snark reply, but I'm really exasperated here.

There is a proposed solution and there are arguments put forward for that solution. The response, after months, is "how about something else" with no actual counter arguments, or even mention how something else is better or how the original proposal is bad.

Trying to match shebangs or typical batch first liners is fine and makes the gem author's job easier at times, but it really makes no sense to prevent them from having a way to deal with false positives and false negatives. It will only lead to another level of hoop jumps and hacky workarounds.

The magic string is the best idea I have for this.

[deivid-rodriguez]

Well, this is a really really niche use case, so nobody cares much about it. Previously this was closed and there was no reply for years, now I'm trying to loop back every few weeks. Sorry but this is all I have.

One reason for auto-detection instead of a magic string feature is that it would just work by default. With the magic string the gem author will run into the weird issue, and might not have a clue on how to solve it and have trouble finding about hidden feature. She might even think: this is clearly a batch file, why didn't rubygems do the right thing, it's probably broken, and just give up.

Something else I'm thinking is: have you tried using a Rakefile extension for this? You can put your script at, say, lib/scripts/foo.cmd, and then add some ruby code as a Rakefile extension that will run at install time and put the script in the proper location.

[iboB]

I'm not familiar with that solution. I'll give it a shot.

After coming across this issue and the times it was referenced with no workarounds added here, I assumed there was no workaround.

Also non_ruby_executables in the gemspec is an alternative idea from above, which should not be forgotten. This satisfies all needs and there is no room for false positives or false negatives. (could be also be no_wrap_executables or something else)

[deivid-rodriguez]

It's an idea I just had, let me know how it goes!

[iboB]

Also, On the "niche-ness" of this use case, I honestly think it's a feedback loop of softs. Every other package manager allows shipping of arbitrary shell scripts or executables so people which have this use case just use alternative package managers. Shipping ruby software with npm is not unheard of.

[deivid-rodriguez]

You may be right, and "niche-ness" is very subjective anyways, so I retract that. What we know for fact is that it hasn't gauged much interest between the different rubygems maintainers.

[tonischilling]

Hello to you all. I think this is a good moment to encourage you to continue with that thing!!
I still feel as a beginner with ruby and all that, so I usually just follow the discussion - you would never know of my interest.
But I think there are many more out there like me - just listening silently - waiting for the "no-wrapper" or "this-should-be-installed-in-that-way-under-these-conditions" feature.
Hm, sounds like a lot of work - sorry.

[pboling]

I just wrote and shipped an internal gem for the purpose of sharing internal bash scripts among many Ruby projects.
To my surprise these normal bash scripts, with a shebang line indicating as much, were being run as Ruby, and so I ended up here. I never even considered the possibility that they would be wrapped. It seems to be a notion among some maintainers that people would assume only Ruby would be shipped, and I can confirm that such an idea never crossed my mind, and now I have no idea what alternative I can come up with that I like. Yes, I hate the rake task idea. I want these general purpose scripts to just work from anywhere on a system, and I had wanted to distribute and update them via rubygems.

I want shell scripts because GNU tools are powerful, and CIs work natively with their exit status. I haven't read the whole thread yet, but I'm surprised this has been and is a thing.