Automate release process #78
Comments
|
@mudge How would you like to proceed with a 2.x release? Would you like to see the automation set up first, or would you be okay with manually generating the packages and pushing them? @flavorjones Just curious if you had any thoughts on automation. Does Nokogiri manually push based on https://github.com/sparklemotion/nokogiri/blob/main/CONTRIBUTING.md#making-a-release? |
|
@stanhu I release manually. It's a pretty low lift, and using RCD containers is repeatable, so I haven't felt a strong need to automate releases. |
|
I’m happy to push releases manually too. What does the process look like at this point? |
|
@mudge Normally I'd say:
bundle exec rake clobber
bundle exec rake compile
bundle exec rake spec
bundle exec rake gem
bundle exec rake gem:native
However, I'm seeing some linker errors if I don't do a I'm not sure why that's happening, since the libraries were just compiled in the Sometimes when I run It seems to build fine when I do a Note it works fine in CI: https://github.com/mudge/re2/actions/runs/5634578432/job/15264669351?pr=76 |
|
Here's the script I use to build nokogiri: https://github.com/sparklemotion/nokogiri/blob/main/scripts/build-gems I clobber between each of the platform types which probably avoids the behavior you're seeing. (I'm not sure what the root cause is.) |
|
Thanks, I did see that script. If I try to compile the Windows gems in a sequence, I get that unknown bundle exec rake gem:x64-mingw-ucrt
bundle exec rake gem:x64-mingw32 One issue is that the Docker image $ docker run -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x64-mingw32 bash
root@035b400d58f2:/# cat ~/.rake-compiler/config.yml
---
rbconfig-x64-mingw32-2.4.0: "/usr/local/rake-compiler/ruby/x86_64-w64-mingw32/ruby-2.4.0/lib/ruby/2.4.0/x64-mingw32/rbconfig.rb"
rbconfig-x64-mingw32-2.5.0: "/usr/local/rake-compiler/ruby/x86_64-w64-mingw32/ruby-2.5.0/lib/ruby/2.5.0/x64-mingw32/rbconfig.rb"
rbconfig-x64-mingw32-2.6.0: "/usr/local/rake-compiler/ruby/x86_64-w64-mingw32/ruby-2.6.0/lib/ruby/2.6.0/x64-mingw32/rbconfig.rb"
rbconfig-x64-mingw32-2.7.0: "/usr/local/rake-compiler/ruby/x86_64-w64-mingw32/ruby-2.7.0/lib/ruby/2.7.0/x64-mingw32/rbconfig.rb"
rbconfig-x64-mingw32-3.0.0: "/usr/local/rake-compiler/ruby/x86_64-w64-mingw32/ruby-3.0.0/lib/ruby/3.0.0/x64-mingw32/rbconfig.rb"
root@035b400d58f2:/# ls -al ~/.rake-compiler/ruby/x86_64-w64-mingw32/
total 28
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 .
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ..
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ruby-2.4.0
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ruby-2.5.0
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ruby-2.6.0
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ruby-2.7.0
drwxr-xr-x 1 rvm rvm 4096 Jan 12 2023 ruby-3.0.0The latest snapshots in https://github.com/rake-compiler/rake-compiler-dock/pkgs/container/rake-compiler-dock-image have the same issue. I think that's why we see these warnings (https://github.com/rake-compiler/rake-compiler/blob/100d4241f27fbc9568846a2c1d7a63febeef4522/lib/rake/extensiontask.rb#L398) in https://github.com/mudge/re2/actions/runs/5634578432/job/15264669351?pr=76: When I provided a "fake" entry for Ruby 3.2, I did notice the Now, I'm puzzled how the |
|
Ok, I see there are two issues here:
@flavorjones I'm guessing you don't run into that with Nokogiri because libxml2 and the other libraries don't link against anything in UCRT. To avoid this collision, we might want
When In any case, CI works fine since each job is built independently. To handle releases, we could:
|
Just some data points from Nokogiri:
|
|
Ok, #81 seems to fix both issues. |
|
@stanhu I'm on vacation and can't dig in on the mini portile paths, but your diagnosis sounds about right (and I've seen something similar in grpc cross builds). I can look when I'm back in a week or so. |
|
@flavorjones Thanks! I'm pretty sure this has solved the problem. I can build all native gems with no need to @mudge With #81 and #82 (UPDATE: #83 needed too), the release process would be:
The script doesn't test the gem contents as Nokogiri's scripts do, but we can consider that later. I still wonder if it's easier to push the gems built from CI. |
|
@mudge Sorry to bother you again. Would you be able to cut a 2.x release? |
|
Hi @stanhu, Thanks for your continued patience. As you may have noticed, I'm struggling to make time to properly review and test this at the moment. However! I did just try running a build on the current I suspect the first is harmless and just means that Docker is running the images under virtualisation (see rake-compiler/rake-compiler-dock#78) but do you recognise the second? |
|
I'm wondering about how best to handle version 1 and 2 of the gem. Is it high time I gave up on older Ruby versions, matching @flavorjones' policy of the last four versions (regardless of EOL status) and only offer 2.0 with its minimum Ruby version of 2.7 (even though the gem will work with older Rubies, it's just that the packaging of it does not)? If so, we could merge this into Or do I still want to maintain the version 1 branch, potentially pushing new versions if re2 requires it? In which case, there's some branching hygiene and documentation to update across Sadly, I can't easily see what versions of Ruby current users are on but it is clear GitLab is the biggest dependent and it requires Ruby 3.0 so perhaps it is time to put version 1 to bed. Worst case, if there is something I need to backport, I can always dig it out of the git history later. |
|
My build finished with the following error: Here's the full detail from |
|
@mudge I should have mentioned that it's best right now to build native gems on a Linux x86 machine to avoid the overhead of QEMU. moby/moby#20634 (comment) suggests this |
|
I can reproduce the problem via: % docker run --platform linux/i386 -it ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x86-linux
linux32: Failed to set personality to linux32: Invalid argumentI think this is just a limitation with Docker for Mac. I tried enabling the Rosetta emulation feature (docker/for-mac#6796 (comment)), but that didn't seem to help. |
|
In that case, perhaps it would be simplest to set up GitHub Actions to run the full cross-build and save all the artifacts with the default retention period? For now, we can do it on every push but you could imagine doing this only on release and then I can manually download and publish the gems to RubyGems. |
|
@mudge The full cross-build artifacts were previously retained for 1 day, but I increased this to 7 days in #86. You can see an example list of all the artifacts here: https://github.com/mudge/re2/actions/runs/5871088933 |
|
Oh, but I just realized we may have to update the build script to use the tag instead the timestamp for the version. |
|
For now, can we hardcode the version to 2.0.0.beta1? |
|
I'm happy with the current setup: CI will build a gem with whatever is specified in |
Once #76 is merged, and the
v2.0.xbranch is released, we should probably automate the build and publish process, such as: https://github.com/y-crdt/yrb/blob/main/.github/workflows/gem.ymlThe text was updated successfully, but these errors were encountered: