New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unpacking of gems with symlinks broken on windows #1623

Open
enebo opened this Issue May 24, 2016 · 14 comments

Comments

Projects
None yet
5 participants
@enebo
Contributor

enebo commented May 24, 2016

My current problem is that File.symlink is not supported on windows and extract_tar_gz unconditionally calls it. This problem also occurs on JRuby as well at Ruby 2.2 (I have not checked whether there happens to be symlink support in a newer version of MRI).

JRuby tracking issue on this is: jruby/jruby#3905.

This issue is related to:

  • Network problems
  • Installing a library
  • Publishing a library
  • The command line gem
  • Other

Here are my current environment details:

$ gem env version
~ 508% /c/Ruby22-x64/bin/ruby -S gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 2.6.4
  - RUBY VERSION: 2.2.2 (2015-04-13 patchlevel 95) [x64-mingw32]
  - INSTALLATION DIRECTORY: C:/Ruby22-x64/lib/ruby/gems/2.2.0
  - USER INSTALLATION DIRECTORY: C:/Users/enebo/.gem/ruby/2.2.0
  - RUBY EXECUTABLE: C:/Ruby22-x64/bin/ruby.exe
  - EXECUTABLE DIRECTORY: C:/Ruby22-x64/bin
  - SPEC CACHE DIRECTORY: C:/Users/enebo/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: C:/ProgramData
  - RUBYGEMS PLATFORMS:
    - ruby
    - x64-mingw32
  - GEM PATHS:
     - C:/Ruby22-x64/lib/ruby/gems/2.2.0
     - C:/Users/enebo/.gem/ruby/2.2.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - C:\Users\enebo\bin
     - C:\Program Files\Git\mingw64\bin
     - C:\Program Files\Git\usr\local\bin
     - C:\Program Files\Git\usr\bin
     - C:\Program Files\Git\usr\bin
     - C:\Program Files\Git\cmd
     - C:\Program Files\ConEmu\ConEmu\Scripts
     - C:\Program Files\ConEmu
     - C:\Program Files\ConEmu\ConEmu
     - C:\Program Files\Java\jdk1.7.0_79\bin
     - C:\Program Files (x86)\AMD APP\bin\x86_64
     - C:\Program Files (x86)\AMD APP\bin\x86
     - C:\Windows\system32
     - C:\Windows
     - C:\Windows\System32\Wbem
     - C:\Windows\System32\WindowsPowerShell\v1.0
     - C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static
     - C:\Program Files\Microsoft SQL Server\110\Tools\Binn
     - C:\Program Files\WIDCOMM\Bluetooth Software
     - C:\Program Files\WIDCOMM\Bluetooth Software\syswow64
     - C:\ProgramData\chocolatey\bin
     - C:\Program Files\Git\cmd
     - C:\Program Files (x86)\Skype\Phone
     - C:\jruby-9.1.1.0\bin
     - C:\opt\maven\bin
     - C:\opt\ant\bin
     - C:\Program Files\Git\usr\bin\vendor_perl
     - C:\Program Files\Git\usr\bin\core_perl
     - C:\Program\ Files\ \(x86\)\Rust\bin
     - C:\opt\rust\cargo\bin
     - C:\Program Files\Git\usr\bin
     - C
     - C:\Program Files\Java\jdk1.7.0_79\bin
     - C:\Users\enebo\Developer\bin
     - C:\Users\enebo\opt\jay\bin
     - C:\Users\enebo\opt\jflex\bin
     - C:\Program Files\Git\usr\local\bin
~ 506% /c/Ruby22-x64/bin/ruby -S gem install warbler
Fetching: rubyzip-1.2.0.gem (100%)
Successfully installed rubyzip-1.2.0
Fetching: jruby-rack-1.1.20.gem (100%)
Successfully installed jruby-rack-1.1.20
Fetching: jruby-jars-9.1.1.0.gem (100%)
Successfully installed jruby-jars-9.1.1.0
Fetching: warbler-2.0.1.gem (100%)
C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:386:in `symlink': symlink() function is unimplemented on this machine (NotImplementedError)
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:386:in `block (2 levels) in extract_tar_gz'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package/tar_reader.rb:65:in `each'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:363:in `block in extract_tar_gz'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:457:in `block in open_tar_gz'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:454:in `wrap'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:454:in `open_tar_gz'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:362:in `extract_tar_gz'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:343:in `block (2 levels) in extract_files'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package/tar_reader.rb:65:in `each'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:340:in `block in extract_files'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package/file_source.rb:30:in `open'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package/file_source.rb:30:in `with_read_io'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/package.rb:337:in `extract_files'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/installer.rb:766:in `extract_files'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/installer.rb:296:in `install'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/resolver/specification.rb:97:in `install'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/request_set.rb:166:in `block in install'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/request_set.rb:156:in `each'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/request_set.rb:156:in `install'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/commands/install_command.rb:205:in `install_gem'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/commands/install_command.rb:255:in `block in install_gems'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/commands/install_command.rb:251:in `each'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/commands/install_command.rb:251:in `install_gems'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/commands/install_command.rb:158:in `execute'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/command.rb:310:in `invoke_with_build_args'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/command_manager.rb:169:in `process_args'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/command_manager.rb:139:in `run'
        from C:/Ruby22-x64/lib/ruby/site_ruby/2.2.0/rubygems/gem_runner.rb:55:in `run'
        from C:/jruby-9.1.1.0/bin/jgem:21:in `<top (required)>'
        from C:/jruby-9.1.1.0/bin/gem:4:in `load'
        from C:/jruby-9.1.1.0/bin/gem:4:in `<main>'

I will abide by the code of conduct.

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo May 24, 2016

Contributor

Ok looked it up and Ruby 2.3 did in fact add symlink support. So a fix for this might be a little more involved.

Contributor

enebo commented May 24, 2016

Ok looked it up and Ruby 2.3 did in fact add symlink support. So a fix for this might be a little more involved.

@segiddins

This comment has been minimized.

Show comment
Hide comment
@segiddins

segiddins May 24, 2016

Member

This was added in #1418

Member

segiddins commented May 24, 2016

This was added in #1418

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo May 24, 2016

Contributor

@segiddins I only see this fixing the binstub not arbitrary symlinks while unpacking gem contents?

Contributor

enebo commented May 24, 2016

@segiddins I only see this fixing the binstub not arbitrary symlinks while unpacking gem contents?

@segiddins

This comment has been minimized.

Show comment
Hide comment
@segiddins

segiddins May 25, 2016

Member

What do you think the correct behavior here would be? Only thing I can think of is failing in a more friendly manner

Member

segiddins commented May 25, 2016

What do you think the correct behavior here would be? Only thing I can think of is failing in a more friendly manner

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo May 25, 2016

Contributor

@segiddins I can think of three paths:

  1. silently ignore symlink entries (what RG has done up until this last release added support). This may or may not break things. I have first-hand experience that warbler gem has a softlink in it but does not break Windows support (it seems to ship it's specs and it has a softlink test in it). So presence of a softlink on a gem which is intended to work on windows likely will not break anything. However most gem authors probably don't think about windows as a platform.
  2. warn on symlink entries. This will at least allow the gem to install although it could lead to reports if the gem actually depends on the symlink.
  3. fail to install gems with a friendlier message than a backtrace.

At this point I think 3 will surprise some windows users but it probably is a time-limited issue. Ruby 2.3 added softlinks and JRuby will pretty soon as well (one of our missing 2.3 features yet to be done). If you are using an earlier version of Ruby doing 3 will generate MRI and JRuby bugs when they hit a gem which has softlinks (we got 2 in the last week). Obviously if the gem depends on softlinks then it could also generate an issue report. Since this is new behavior of gems I do not think we will see any/many of these issues any time soon.

Contributor

enebo commented May 25, 2016

@segiddins I can think of three paths:

  1. silently ignore symlink entries (what RG has done up until this last release added support). This may or may not break things. I have first-hand experience that warbler gem has a softlink in it but does not break Windows support (it seems to ship it's specs and it has a softlink test in it). So presence of a softlink on a gem which is intended to work on windows likely will not break anything. However most gem authors probably don't think about windows as a platform.
  2. warn on symlink entries. This will at least allow the gem to install although it could lead to reports if the gem actually depends on the symlink.
  3. fail to install gems with a friendlier message than a backtrace.

At this point I think 3 will surprise some windows users but it probably is a time-limited issue. Ruby 2.3 added softlinks and JRuby will pretty soon as well (one of our missing 2.3 features yet to be done). If you are using an earlier version of Ruby doing 3 will generate MRI and JRuby bugs when they hit a gem which has softlinks (we got 2 in the last week). Obviously if the gem depends on softlinks then it could also generate an issue report. Since this is new behavior of gems I do not think we will see any/many of these issues any time soon.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius May 25, 2016

Contributor

I want to also chime in here and say that having a hard failure here seems really unreasonable to me. I doubt we intended to have a softlink in any of our packages, and suddenly a new version of RG comes along that can no longer install them on Windows. That's a regression.

The fallback option of simply having two copies of the same file is just fine, because an installed gem is never really expected to be modified, right? So there's practically no chance someone's going to change the softlink target expecting to see the softlink update.

At most, I think RG should warn that softlinks will be copies on Ruby impls that don't support symlinking on Windows (@enebo's 2nd path above).

Contributor

headius commented May 25, 2016

I want to also chime in here and say that having a hard failure here seems really unreasonable to me. I doubt we intended to have a softlink in any of our packages, and suddenly a new version of RG comes along that can no longer install them on Windows. That's a regression.

The fallback option of simply having two copies of the same file is just fine, because an installed gem is never really expected to be modified, right? So there's practically no chance someone's going to change the softlink target expecting to see the softlink update.

At most, I think RG should warn that softlinks will be copies on Ruby impls that don't support symlinking on Windows (@enebo's 2nd path above).

@segiddins

This comment has been minimized.

Show comment
Hide comment
@segiddins

segiddins May 25, 2016

Member

@headius it's not a regression -- previously rubygems just didnt include symlinks when packaging. It also warns when packaging a gem that includes symlinks

The fallback option of simply having two copies of the same file is just fine, because an installed gem is never really expected to be modified, right? So there's practically no chance someone's going to change the softlink target expecting to see the softlink update.

That won't work for anything that depends upon the realpath of the symlinked files

Member

segiddins commented May 25, 2016

@headius it's not a regression -- previously rubygems just didnt include symlinks when packaging. It also warns when packaging a gem that includes symlinks

The fallback option of simply having two copies of the same file is just fine, because an installed gem is never really expected to be modified, right? So there's practically no chance someone's going to change the softlink target expecting to see the softlink update.

That won't work for anything that depends upon the realpath of the symlinked files

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo May 25, 2016

Contributor

I think copy could lead to requiring the same file twice. I am a bit fuzzy but I thought softlinks resolved to real path in require semantics.

Our main challenge in the presences of a failure is that for 9k we need to add symlink support which is not an issue really. For JRuby 1.7.x we will never support symlinks as it is not part of Ruby 1.9.3. Perhaps we stop updating rubygems there or maybe port over part of @djberg96 symlink logic.

For existing users on existing Rubies once they update rubygems locally they will hit this (at least we know all JRuby users with warbler in their Gemfile.lock will) and then open up a bug report on our project. I am not sure how many that will be but it is mainly why we are going for the warning or ignore options :)

Contributor

enebo commented May 25, 2016

I think copy could lead to requiring the same file twice. I am a bit fuzzy but I thought softlinks resolved to real path in require semantics.

Our main challenge in the presences of a failure is that for 9k we need to add symlink support which is not an issue really. For JRuby 1.7.x we will never support symlinks as it is not part of Ruby 1.9.3. Perhaps we stop updating rubygems there or maybe port over part of @djberg96 symlink logic.

For existing users on existing Rubies once they update rubygems locally they will hit this (at least we know all JRuby users with warbler in their Gemfile.lock will) and then open up a bug report on our project. I am not sure how many that will be but it is mainly why we are going for the warning or ignore options :)

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius May 25, 2016

Contributor

@segiddins It is a regression, because gems that packaged and installed correctly before no longer package and install correctly. That's crystal clear to me.

You're right about the realpath. So what about just omitting the links (with a warning) if symlinking isn't supported? Then both the packager and the installer will get a warning about symlinks not being supported everywhere, but gems accidentally pushed with symlinks will still install like they did before.

If you get to the point that a gem has installed and the links were required for it to work, well, that gem has decided not to be compatible with your environment.

If this remains a hard failure, I think you're going to get more bug reports. symlinks have a way of creeping into things. A gem publisher who gets bug reports that their gem no longer installs on Windows will probably be asking why you folks broke his gem. Before, the gem published and installed fine. Now, we're requiring gem publishers to explicitly exclude symlinks that were implicitly excluded before.

Interestingly, it looks like @nobu added this and MRI's Windows symlink support around the same time. If RG intends to support older versions of Ruby, this new hard failure needs to be changed.

Contributor

headius commented May 25, 2016

@segiddins It is a regression, because gems that packaged and installed correctly before no longer package and install correctly. That's crystal clear to me.

You're right about the realpath. So what about just omitting the links (with a warning) if symlinking isn't supported? Then both the packager and the installer will get a warning about symlinks not being supported everywhere, but gems accidentally pushed with symlinks will still install like they did before.

If you get to the point that a gem has installed and the links were required for it to work, well, that gem has decided not to be compatible with your environment.

If this remains a hard failure, I think you're going to get more bug reports. symlinks have a way of creeping into things. A gem publisher who gets bug reports that their gem no longer installs on Windows will probably be asking why you folks broke his gem. Before, the gem published and installed fine. Now, we're requiring gem publishers to explicitly exclude symlinks that were implicitly excluded before.

Interestingly, it looks like @nobu added this and MRI's Windows symlink support around the same time. If RG intends to support older versions of Ruby, this new hard failure needs to be changed.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius May 25, 2016

Contributor

Another alternative: require publishers to opt in to this feature by specifying include_symlinks = true in their gemspecs. Implicitly doing this when it wasn't before, and then blowing up when installing in some environments is definitely going to surprise people.

Contributor

headius commented May 25, 2016

Another alternative: require publishers to opt in to this feature by specifying include_symlinks = true in their gemspecs. Implicitly doing this when it wasn't before, and then blowing up when installing in some environments is definitely going to surprise people.

jasonrclark added a commit to shoes/shoes4 that referenced this issue Dec 6, 2016

Symlinks have trouble with JRuby + Rubygems
Getting caught by rubygems/rubygems#1623, but
turns out our symlink isn't really necessary or desired at all. Just
getting rid of it.
@arthurnn

This comment has been minimized.

Show comment
Hide comment
@arthurnn

arthurnn Aug 15, 2017

Member

I am seeing an issue where a private gem that has some extensions that the source of the C lib is a symlink can't build + install anymore. I guess is related to this.

Gem 2.6.11 build/install: I see the issue
Gem 2.0.14.1 build + Gem 2.6.11 install: works fine

So I suspect it is something on the build process.
During building my gem on 2.6.11, I get a bunch of warnings like:

WARNING:  ext/foo-ruby/inc is a symlink, which is not supported on all platforms

I wonder when this started happening.
This thread seems specific to windows, but the issue I am seeing is happening on MacOS.

Member

arthurnn commented Aug 15, 2017

I am seeing an issue where a private gem that has some extensions that the source of the C lib is a symlink can't build + install anymore. I guess is related to this.

Gem 2.6.11 build/install: I see the issue
Gem 2.0.14.1 build + Gem 2.6.11 install: works fine

So I suspect it is something on the build process.
During building my gem on 2.6.11, I get a bunch of warnings like:

WARNING:  ext/foo-ruby/inc is a symlink, which is not supported on all platforms

I wonder when this started happening.
This thread seems specific to windows, but the issue I am seeing is happening on MacOS.

@segiddins

This comment has been minimized.

Show comment
Hide comment
@segiddins

segiddins Aug 16, 2017

Member

Keep in mind windows doesn't have symlinks

Member

segiddins commented Aug 16, 2017

Keep in mind windows doesn't have symlinks

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Aug 16, 2017

Contributor

@segiddins but if you use symlink methods and are an administrator on windows those APIs work and make an analogue (cannot remember the windows name for them). Not sure if there are even a semantic difference at this point or not from the viewpoint of a Ruby programmer?

@arthurnn really weird to see that warning come out on macos. I wonder if someone is platform checking us and seeing 'java' and then assuming it is not sof-tlinkable?

Contributor

enebo commented Aug 16, 2017

@segiddins but if you use symlink methods and are an administrator on windows those APIs work and make an analogue (cannot remember the windows name for them). Not sure if there are even a semantic difference at this point or not from the viewpoint of a Ruby programmer?

@arthurnn really weird to see that warning come out on macos. I wonder if someone is platform checking us and seeing 'java' and then assuming it is not sof-tlinkable?

@enebo

This comment has been minimized.

Show comment
Hide comment
@enebo

enebo Aug 16, 2017

Contributor

@segiddins heh we both know this was added in 2.3 so I don't know why I am replying to you about it...I just wanted to point out some Rubys will obey the API now.

Contributor

enebo commented Aug 16, 2017

@segiddins heh we both know this was added in 2.3 so I don't know why I am replying to you about it...I just wanted to point out some Rubys will obey the API now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment