Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tapioca sync fails because of nokogiri native gem having different dependencies than compiled gem #208

Closed
connorshea opened this issue Jan 18, 2021 · 9 comments · Fixed by #213
Closed

tapioca sync fails because of nokogiri native gem having different dependencies than compiled gem #208

connorshea opened this issue Jan 18, 2021 · 9 comments · Fixed by #213

Comments

@connorshea
Copy link
Contributor

Essentially, this error occurs when I run bundle exec tapioca sync on my Rails app vglist:

Connors-MacBook-Pro-2:vglist connorshea$ bundle exec tapioca sync
Removing RBI files of gems that have been removed:

bundler: failed to load command: tapioca (/Users/connorshea/.rbenv/versions/2.7.2/bin/tapioca)
Traceback (most recent call last):
	44: from /Users/connorshea/.rbenv/versions/2.7.2/bin/bundle:23:in `<main>'
	43: from /Users/connorshea/.rbenv/versions/2.7.2/bin/bundle:23:in `load'
	42: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/exe/bundle:37:in `<top (required)>'
	41: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors'
	40: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/exe/bundle:49:in `block in <top (required)>'
	39: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli.rb:24:in `start'
	38: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	37: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli.rb:30:in `dispatch'
	36: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	35: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	34: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	33: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli.rb:497:in `exec'
	32: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli/exec.rb:28:in `run'
	31: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli/exec.rb:63:in `kernel_load'
	30: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/cli/exec.rb:63:in `load'
	29: from /Users/connorshea/.rbenv/versions/2.7.2/bin/tapioca:23:in `<top (required)>'
	28: from /Users/connorshea/.rbenv/versions/2.7.2/bin/tapioca:23:in `load'
	27: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/exe/tapioca:6:in `<top (required)>'
	26: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
	25: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
	24: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
	23: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
	22: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/cli.rb:92:in `sync'
	21: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca.rb:10:in `silence_warnings'
	20: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/site_ruby/2.7.0/rubygems/user_interaction.rb:46:in `use_ui'
	19: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca.rb:11:in `block in silence_warnings'
	18: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/cli.rb:93:in `block in sync'
	17: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `block in _on_method_added'
	16: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `call'
	15: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/generator.rb:149:in `sync_rbis_with_gemfile'
	14: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `block in _on_method_added'
	13: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `call'
	12: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/generator.rb:332:in `perform_removals'
	11: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `block in _on_method_added'
	10: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `call'
	 9: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/generator.rb:298:in `removed_rbis'
	 8: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `block in _on_method_added'
	 7: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `call'
	 6: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/generator.rb:270:in `expected_rbis'
	 5: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `block in _on_method_added'
	 4: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/sorbet-runtime-0.5.6189/lib/types/private/methods/_methods.rb:233:in `call'
	 3: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/tapioca-0.4.13/lib/tapioca/gemfile.rb:35:in `dependencies'
	 2: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/spec_set.rb:81:in `materialize'
	 1: from /Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/spec_set.rb:81:in `map!'
/Users/connorshea/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/bundler-2.2.3/lib/bundler/spec_set.rb:87:in `block in materialize': Could not find mini_portile2-2.5.0 in any of the sources (Bundler::GemNotFound)

I tried upgrading RubyGems, upgrading Bundler, upgrading Sorbet, downgrading Tapioca from 0.4.13 to 0.4.10, running bundle pristine to reinstall absolutely every gem in my Gemfile, and none of it changed anything. It occurs on both Bundler 2.2.3 and 2.2.5.

The only thing that fixed it was downgrading nokogiri from 1.11.1 to 1.10.10, which removes the native gem differentiation.

I suspect it has to do with this, nokogiri 'proper' requiring mini_portile2 but its native versions not?:

    nokogiri (1.11.1)
      mini_portile2 (~> 2.5.0)
      racc (~> 1.4)
    nokogiri (1.11.1-x86_64-darwin)
      racc (~> 1.4)
    nokogiri (1.11.1-x86_64-linux)
      racc (~> 1.4)

It looks like it's an intentional choice by Nokogiri, because mini_portile2 is required to compile nokogiri, but if you have a pre-compiled version there's no reason to install it: sparklemotion/nokogiri#2078

So that'd explain why we don't have the gem locally on macOS. I suspect we need to figure out a way to filter out gems that wouldn't exist locally?
@connorshea connorshea mentioned this issue Jan 18, 2021
Merged
@connorshea
Copy link
Contributor Author

Unfortunately this is not fixed by adding a tapioca config file since Tapioca doesn't take the exclude config value into consideration when choosing what gems to require:

---
exclude: ['mini_portile2']

@connorshea
Copy link
Contributor Author

connorshea commented Jan 18, 2021 •
edited

If I change materialize(specs) to materialize(specs, ['mini_portile2']) here:

tapioca/lib/tapioca/gemfile.rb

Line 35 in 1efec9e

.materialize(specs)

Then the problem is fixed. Assuming there's no good way to determine whether a gem is platform-specific and should be excluded via Bundler, then we could pass the exclude configuration there? It doesn't seem like the materialize method cares what you pass it, though. It just checks whether there's a value for the argument and then appends it to the missing_specs variable.

connorshea added a commit to connorshea/vglist that referenced this issue Jan 18, 2021
@connorshea
Regenerate Sorbet gem signatures with Tapioca.
ed44115 
This uses a workaround I discovered for this issue:
Shopify/tapioca#208

Kind of a hack, but it works.
@connorshea connorshea mentioned this issue Jan 18, 2021
Merged
@paracycle
Copy link
Member

@connorshea Thanks for reporting this and the detailed analysis. This looks related to #28. I'll take another look at this. Maybe we can do a pre-pass on the specs list to figure out which specs might not exist locally and exclude those in materialize.

@paracycle
Copy link
Member

It doesn't seem like the materialize method cares what you pass it, though. It just checks whether there's a value for the argument and then appends it to the missing_specs variable.

Looking at the code, I just realized what you mean by this. It does seem like passing in an empty array for missing_specs parameter would just silence all the errors coming from missing specs, and we would get all the specs that are missing back as a side-effect. Good find, I'll work on this.

@paracycle
Copy link
Member

I am having a hard time reproducing this case on my local machine. Is it possible for you to link to a Gist or something that has dependency on Tapioca and Nokogiri where I could reproduce this? The code change is trivial but I would like to add a test to make sure we don't regress if we fix this.

@connorshea
Copy link
Contributor Author

connorshea commented Jan 23, 2021 •
edited

@paracycle sure, I'm putting one together right now. I think the key is that you need to use bundle lock --add-platform ruby for it to pull in the generic self-compiled nokogiri. I'm not totally sure why that was in my repo, maybe a legacy thing from before Bundler 2.2? Regardless, I think it's worth keeping support for.

@connorshea
Copy link
Contributor Author

@paracycle I've created a really basic repo by just doing these commands:

bundle init
bundle add tapioca nokogiri
bundle install
bundle lock --add-platform=ruby
bundle exec tapioca sync

So all you need to do is clone this repo and run bundle install && bundle exec tapioca sync, and it should fail: https://github.com/connorshea/tapioca-nokogiri-bug-example

If you're on Linux, I'm not 100% sure if this'll fail because it falls back to "ruby" and compiles nokogiri from scratch or if it'd just add linux as a platform in the gemfile automatically.

@paracycle
Copy link
Member

Awesome, thanks for the repro. I am able to reproduce with those steps and passing an array for missing_specs silences the errors on that and allows us to print which gems we are skipping since they are missing.

I'll PR something today or tomorrow (depending on the amount of code changes I need to make to test this cleanly).

@paracycle paracycle mentioned this issue Jan 25, 2021
Merged
@connorshea
Copy link
Contributor Author

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix problem with missing gem specs Shopify/tapioca
2 participants
@paracycle @connorshea