vendor/bundle/ruby: cleanup unneeded files #15954
Conversation
These do need to be vendored for VSCode editor support.
🎉
https://github.com/github/licensed may be an alternate option here and we could commit a single |
How does this work in practice? I use VSCode but whenever I want to lint things I just run |
Most VSCode RuboCop plugins will just run |
|
How it works in practice is we configure the path: Line 10 in b2cf50b and we have this shim: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/gems/rubocop which calls: https://github.com/Homebrew/brew/blob/master/Library/Homebrew/dev-cmd/rubocop.sh which invokes and then it invokes a special RuboCop startup script (we don't use RuboCop's own binary): https://github.com/Homebrew/brew/blob/master/Library/Homebrew/utils/rubocop.rb. |
Bo98
force-pushed
the
vendor-cleanup
branch
from
ca89262
to
e04c650
Compare
Bo98
force-pushed
the
vendor-cleanup
branch
from
e04c650
to
ea9c776
Compare
|
Hmm it seems I think the problem is the directories aren't fully removed with this commit because some readme etc files were ignored, so the gem directory is still there. Worst case scenario is we just do the |
I guess we could look at e.g. once file for each gem we need and run
Sounds good! |
This wouldn't work because it'll cause uncommitted changes if bundle removes those gems.
I did come up with something like this: diff --git a/Library/Homebrew/utils/gems.rb b/Library/Homebrew/utils/gems.rb
index 25c10b29f..aed4621d3 100644
--- a/Library/Homebrew/utils/gems.rb
+++ b/Library/Homebrew/utils/gems.rb
@@ -22,6 +22,11 @@ module Homebrew
File.join(ENV.fetch("HOMEBREW_LIBRARY"), "Homebrew", "Gemfile")
end
+ # @api private
+ def bundler_definition
+ @bundler_definition ||= Bundler::Definition.build(Bundler.default_gemfile, Bundler.default_lockfile, false)
+ end
+
# @api private
def valid_gem_groups
install_bundler!
@@ -29,7 +34,7 @@ module Homebrew
Bundler.with_unbundled_env do
ENV["BUNDLE_GEMFILE"] = gemfile
- groups = Bundler::Definition.build(Bundler.default_gemfile, Bundler.default_lockfile, false).groups
+ groups = bundler_definition.groups
groups.delete(:default)
groups.map(&:to_s)
end
@@ -229,7 +234,35 @@ module Homebrew
bundle_check_failed = !$CHILD_STATUS.success?
# for some reason sometimes the exit code lies so check the output too.
- bundle_installed = if bundle_check_failed || bundle_check_output.include?("Install missing gems")
+ bundle_install_required = bundle_check_failed || bundle_check_output.include?("Install missing gems")
+
+ # Check if the install is intact. This is useful if any gems are added to gitignore.
+ # We intentionally map over everything and then call `any?` so that we remove the spec of each bad gem.
+ bundle_install_required ||= bundler_definition.specs.map do |spec|
+ spec_file = "#{Gem.dir}/specifications/#{spec.full_name}.gemspec"
+ next false unless File.exist?(spec_file)
+
+ cache_file = "#{Gem.dir}/cache/#{spec.full_name}.gem"
+ if File.exist?(cache_file)
+ require "rubygems/package"
+ package = Gem::Package.new(cache_file)
+ def package.verify # disable verification
+ true
+ end
+ contents = package.contents
+
+ package_install_intact = contents && contents.all? do |gem_file|
+ File.exist?("#{Gem.dir}/gems/#{spec.full_name}/#{gem_file}")
+ end
+ next false if package_install_intact
+ end
+
+ # Mark gem for reinstallation
+ File.unlink(spec_file)
+ true
+ end.any?
+
+ bundle_installed = if bundle_install_required
if system bundle, "install"
true
elseWhich is effective at doing a full verification. It does add half a second to no-op So a couple paths forward we could do:
|
|
Thanks @Bo98, great work as usual!
This seems pretty reasonable to me given it makes things considerably more reliable. If that can get slimmed down at all: great, otherwise I think it's not a big deal.
This seems also reasonable but would want to ensure it's enforced by CI i.e. if you forget to bump (or bump unnecessarily): something in CI fails. This would be a blocker on doing this. Suggestion: do the above for now and someone can improve performance once it's working. |
Bo98
force-pushed
the
vendor-cleanup
branch
2 times, most recently
from
431e7fc
to
9fcfbfa
Compare
Bo98
force-pushed
the
vendor-cleanup
branch
from
9fcfbfa
to
5820494
Compare
I took a stab at it. Despite the initial complexity, 🤞 it should make these type of things trivial to deal with in the future, instead of scratching heads for a couple weeks like I have here over something I only noticed when I was switching between branches (CI was passing). |
| run: | | ||
| { | ||
| echo "vendor-version=$(cat ../.homebrew_vendor_version)" | ||
| echo "ignored<<EOS" | ||
| git check-ignore -- * | ||
| echo "EOS" | ||
| } >> "${GITHUB_OUTPUT}" | ||
|
|
||
| - name: Compare to base ref | ||
| working-directory: ${{ steps.set-up-homebrew.outputs.gems-path }}/${{ steps.ruby-abi.outputs.version }} | ||
| run: | | ||
| git checkout "origin/${GITHUB_BASE_REF}" | ||
| rm .homebrew_vendor_version | ||
| brew install-bundler-gems --groups=all | ||
| if [[ "$(cat .homebrew_vendor_version)" == "${{ steps.gem-info.outputs.vendor-version }}" ]]; then | ||
| ignored_gems="${{ steps.gem-info.outputs.ignored }}" | ||
| while IFS= read -r gem; do | ||
| gem_dir="./gems/${gem}" | ||
| [[ -d "${gem_dir}" ]] || continue | ||
| exit_code=0 | ||
| git check-ignore --quiet "${gem_dir}" || exit_code=$? | ||
| if (( exit_code != 0 )); then | ||
| if (( exit_code == 1 )); then | ||
| echo "::error::VENDOR_VERSION needs bumping in utils/gems.rb" >&2 | ||
| else | ||
| echo "::error::git check-ignore failed" >&2 | ||
| fi | ||
| exit "${exit_code}" | ||
| fi | ||
| done <<< "${ignored_gems}" | ||
| fi |
There was a problem hiding this comment.
Not blocking but might be nice to pull these into dedicated Bash scripts instead at some point.
Draft because this requires #15953.