[Experimental] Add "gem rebuild" command.

[duckinator]

What was the end-user or developer problem that led to this PR?

If a gem implements the requirements for reproducible builds, and I have a copy of the source, I want to be able to verify that the released version was built from that source.

What is your fix for the problem, implemented in this PR?

I implemented the gem rebuild command. See the discussion in #3118 for details.

$ ls -a
./  ../
$ git clone https://github.com/duckinator/okay.git
Cloning into 'okay'...
remote: Enumerating objects: 483, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (86/86), done.
remote: Total 483 (delta 87), reused 134 (delta 83), pack-reused 314
Receiving objects: 100% (483/483), 77.50 KiB | 3.52 MiB/s, done.
Resolving deltas: 100% (213/213), done.
$ gem rebuild -C ./okay okay 12.0.1
Fetching okay-12.0.1.gem
Downloaded okay version 12.0.1 as /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem.
WARNING:  open-ended dependency on cacert (>= 0) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: okay
  Version: 12.0.1
  File: okay-12.0.1.gem

Built at: 2021-09-11 20:01:16 EDT (1631404876)
Original build saved to:   /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem
Reproduced build saved to: /usr/home/puppy/dev/rebuild-test/rebuild/new/okay-12.0.1.gem
Working directory: ./okay

Hash comparison:
  558bf6a6343c984f11021aff41c1a63ec07911c0658a408af6e4308b3e7764ea      /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem
  558bf6a6343c984f11021aff41c1a63ec07911c0658a408af6e4308b3e7764ea      /usr/home/puppy/dev/rebuild-test/rebuild/new/okay-12.0.1.gem

SUCCESS - original and rebuild hashes matched
$

Failed example:

$ ls -a
./  ../
$ git clone https://github.com/duckinator/okay.git
Cloning into 'okay'...
remote: Enumerating objects: 483, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (86/86), done.
remote: Total 483 (delta 87), reused 134 (delta 83), pack-reused 314
Receiving objects: 100% (483/483), 77.50 KiB | 3.10 MiB/s, done.
Resolving deltas: 100% (213/213), done.
$ echo '#hi' >> okay/lib/okay.rb 
$ tail -n 2 okay/lib/okay.rb 
end
#hi
$ gem rebuild -C ./okay okay 12.0.1
Fetching okay-12.0.1.gem
Downloaded okay version 12.0.1 as /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem.
WARNING:  open-ended dependency on cacert (>= 0) is not recommended
  use a bounded requirement, such as '~> x.y'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: okay
  Version: 12.0.1
  File: okay-12.0.1.gem

Built at: 2021-09-11 20:01:16 EDT (1631404876)
Original build saved to:   /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem
Reproduced build saved to: /usr/home/puppy/dev/rebuild-test/rebuild/new/okay-12.0.1.gem
Working directory: ./okay

Hash comparison:
  558bf6a6343c984f11021aff41c1a63ec07911c0658a408af6e4308b3e7764ea      /usr/home/puppy/dev/rebuild-test/rebuild/old/okay-12.0.1.gem
  f803f93864366942b9aed3e5f7d806c1bffccae5d59eff74f68dbd59d6811426      /usr/home/puppy/dev/rebuild-test/rebuild/new/okay-12.0.1.gem

FAILURE - original and rebuild hashes did not match
$

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[matiaskorhonen]

Does this work with cryptographically signed gems? (see https://guides.rubygems.org/security/)

[duckinator]

@matiaskorhonen I haven't put in work to specifically support that, so I'd be surprised if it worked. I'm not sure what would be needed to support it.

[duckinator]

Sorry for the slow movement on this — I've been struggling with basically-perpetual brain fog the last two months or so, and it's making it really difficult to actually work on the RubyGems codebase.

I did finally get this to the point that it has a test which fails when expected to instead of passing unexpectedly, which is nice.

At the moment, the test fails because it tries to download something not on RubyGems.org. I tried using the --original flag to specify a local file to compare against but it just gave a different error instead (see the TODO near the end of test/rubygems/test_gem_commands_rebuild_command.rb). That's the next thing to figure out, I think.

[duckinator]

Rebased off master (c44fe8e).

[duckinator]

Resolved linter errors.

[duckinator]

I'm having trouble figuring out why the tests are failing, but the gem rebuild command itself is working:

~/test/rebuild$ ls -a
./  ../
~/test/rebuild$ git clone --branch v12.0.2 https://github.com/duckinator/okay.git
Cloning into 'okay'...
remote: Enumerating objects: 512, done.
remote: Counting objects: 100% (43/43), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 512 (delta 0), reused 0 (delta 0), pack-reused 469
Receiving objects: 100% (512/512), 73.46 KiB | 2.62 MiB/s, done.
Resolving deltas: 100% (240/240), done.
~/test/rebuild$ gem rebuild -C ./okay --gemspec okay.gemspec okay 12.0.2
Fetching okay-12.0.2.gem
Downloaded okay version 12.0.2 as /usr/home/puppy/test/rebuild/rebuild/old/okay-12.0.2.gem.
WARNING:  open-ended dependency on cacert (>= 0) is not recommended
  use a bounded requirement, such as "~> x.y"
WARNING:  See https://guides.rubygems.org/specification-reference/ for help
  Successfully built RubyGem
  Name: okay
  Version: 12.0.2
  File: okay-12.0.2.gem

Built at: 2023-08-31 21:39:02 EDT (1693532342)
Original build saved to:   /usr/home/puppy/test/rebuild/rebuild/old/okay-12.0.2.gem
Reproduced build saved to: /usr/home/puppy/test/rebuild/rebuild/new/okay-12.0.2.gem
Working directory: ./okay

Hash comparison:
  38a8bd78ce10bc19189ead0b56fa490d03788a2f926a6481b2f1f5d5fa5ab75b      /usr/home/puppy/test/rebuild/rebuild/old/okay-12.0.2.gem
  38a8bd78ce10bc19189ead0b56fa490d03788a2f926a6481b2f1f5d5fa5ab75b      /usr/home/puppy/test/rebuild/rebuild/new/okay-12.0.2.gem

SUCCESS - original and rebuild hashes matched
~/test/rebuild$

[duckinator]

Extracting common code from RebuildCommand#download_gem and FetchCommand#execute doesn't seem to make sense. There's only like 4 lines that are the same, and they're scattered throughout the two functions.

I believe everything else y'all have requested has been addressed.

[duckinator]

Rebased off master (1aa7252). @segiddins @deivid-rodriguez this is ready for a final review. If y'all are happy with it and CI passes, it's ready to merge.

[duckinator]

Rebased off master (5956bfb). Will merge if CI passes. 👍

[deivid-rodriguez]

@duckinator Opinions on releasing this with next patch version as experimental vs waiting to next minor? Normally I'd release it with next patch as experimental but since you've made a plugin version of it, not sure.

I tend towards releasing it, just for visibility but I wanted to ask your thoughts.

[duckinator]

@deivid-rodriguez I think we should release it. The plugin is basically a way to "backport" the functionality to old versions, and only gets updated by a script that updates the plugin's files with ones from the RubyGems repo.

The sooner the real version is out, the sooner the plugin becomes irrelevant.

[nobu]

Is it intentional that the downloaded and rebuilt gem files are left under gem_rebuild* temporary directory?

[deivid-rodriguez]

mmmm, I think it'd be better to clean after ourselves? Thoughts @duckinator?

[duckinator]

leaving the files after gem rebuild was intentional, for local inspection, but the test suite should be made to clean up after itself independently of that.