Add support for Ruby 3.0.0-preview1

[JonRowe]

👋 We're in the process of switching RSpec over to using Github actions and it would be helpful to use to have have access to preview versions of Ruby 3.0.0.

(It'd also be helpful to have some legacy versions of Ruby such as 1.8.7. and 1.9.2 but I realise thats probably very out of scope, if anyone wants to help get those running on Github actions with us it would be very appreciated).

[hsbt]

FYI ruby/ruby-builder#4 (comment)

[JonRowe]

I disagree with that conclusion, preview builds are more useful to a gem library because they indicate a stable build that should work, head often contains things that are in progress and flux. We're safely able to get consistency with a preview build that we cannot with head, meaning we often have to allow failures for head for the stability of our builds, but we don't have that problem with preview.

[MSP-Greg]

@JonRowe

We're in the process of switching RSpec over to using Github actions

I did look thru some of the RSpec repos, and didn't find any GitHub Actions (GHA) workflow files. Anyway...

Two differences between GHA and popular CI services are:

1. A much higher number of concurrent jobs.
2. 'Allow failure' is still not as friendly as what people were used to when using Travis or AppVeyor.

So, ruby/ruby is now running a lot more CI jobs than were run using previous CI services.

Before GHA, I worked quite a bit with the Travis head build, and it was an automated build that was not tested.

All current Ruby MRI head builds in this action are fully tested. There are even a few CLI/bin tests run that check whether the 'install' works, as these may not be tested in ruby/ruby.

If tests do not pass, the previous passing head build is left as the build used for CI testing. Ubuntu & macOS builds are done once a day, Windows are run three times a day.

There are quite a few repos using the head builds available here, and many do not implement an 'Allow failure' setup for those jobs.

If you're basing your opinion on other CI services, maybe try the GHA builds first. If you are using the builds, and are having issues with failures, any feedback (either here or ruby/ruby) would be appreciated.

Thanks for RSpec...

[JonRowe]

It's on a branch and an issue as we experiment, its early days, you can see the latest "most rubies" setup here:

https://github.com/rspec/rspec-core/actions/runs/343419327

We're moving off Travis because Travis is limiting support for open source.

Currently we are missing support for 3.0.0-preview1 which would have to remain on Travis, or build a custom image like we may have to do for older Rubies.

[JonRowe]

We find people actually use released versions of Ruby, like preview or rc1, rc 2 etc so we need to support them by testing those specific versions.

[MSP-Greg]

Thanks for the GHA run link. I may not have been clear enough, but everything I stated was very specific to MRI builds.

We find people actually use released versions of Ruby, like preview or rc1, rc 2 etc

I can understand that, but it can also be stated that adoption of head/master builds for CI has been good, so why should the (probably smaller set of) people using rc builds be accounted for? As mentioned in the link @hsbt provided, rc builds really don't have a lot of significance. There's a good chance if someone reported a bug, the response would be 'have you tried it with current master?'

like we may have to do for older Rubies.

Old compilers, unsecure OpenSSL etc. It's odd when a group states that they don't have the resources to update their app, so they'd like other people to continue to support old tools used in their infrastructure, whether that be apps, gems, or CI tools.

It's a matter of opinion. From my perspective, given finite resources, there are other things to work on other than supporting rc builds...

Edit: I'd rather encourage people to use head/master builds all year long rather than use rc builds for a few months.

[JonRowe]

I can understand that, but it can also be stated that adoption of head/master builds for CI has been good, so why should the (probably smaller set of) people using rc builds be accounted for? As mentioned in the link @hsbt provided, rc builds really don't have a lot of significance. There's a good chance if someone reported a bug, the response would be 'have you tried it with current master?'

We don't get people reporting issues from head, where as we do with the official preview versions. They report bugs to us from RSpec on those versions, we do occasionally refer issues back to Ruby but generally its something a new version is introducing for us to handle that doesn't change.

From our perspective we use our finite resources to support fixed versions, especially as preview and release candidates lead to the support of full versions.

[eregon]

We're in the process of switching RSpec over to using Github actions

Great to hear :)

I disagree with that conclusion, preview builds are more useful to a gem library because they indicate a stable build that should work, head often contains things that are in progress and flux. We're safely able to get consistency with a preview build that we cannot with head, meaning we often have to allow failures for head for the stability of our builds, but we don't have that problem with preview.

That's kind of my point, previews are not stable builds, consider them basically a random commit from ruby/ruby master.
Of course, they are a fixed build, so they don't change and remain as good or broken as they are.

The purpose of previews (AFAIK) is to make it easier to build a development version of the upcoming version and gather early feedback. Given Ruby installers support installing ruby-head/ruby-dev I'm not sure what's the extra value nowadays, except announcing "this is a good time to try the upcoming features".

Following the idea "make it easier to try", it might make sense to have it for ruby/ruby-builder too, but OTOH ruby/ruby-builder builds are not meant for end users (only to run in GitHub Actions).
Concretely, there are several practical problems, highlighted in ruby/ruby-builder#4 (comment). Most of them apply to RCs too:

• No Windows builds of previews. I don't know anyone making reusable builds of Ruby previews on Windows (yet). Having inconsistent platform support is a source of confusion.
• It's quite a few extra builds, and these builds don't make sense to be used after the real release. Can we drop those builds after the release? Probably not so easy, we would first need to deprecate them before dropping them.
• Previews are not more stable than ruby-head in my experience. I'm happy to see counter examples, using setup-ruby (as @MSP-Greg mentioned, the builds here are significantly more stable than TravisCI head builds).
• There can be many breaking changes between a preview and the next release (e.g. Fiber scheduler & Ractor API is changing recently), so it often doesn't make too much sense to test in CI, at least not for new APIs that have not stabilized yet.

head often contains things that are in progress and flux

So do previews, several APIs have already been renamed since preview1 and many bugs fixed.

We find people actually use released versions of Ruby, like preview or rc1, rc2 etc so we need to support them by testing those specific versions.

rc1/rc2 etc do aim for progressively more stability.
Nobody should use previews or RCs in production, I think that should be considered unsupported.

I agree with @MSP-Greg. Testing ruby-head seems enough and very similar for stability.
This is not the same as TravisCI ruby-head builds.

I'll reconsider if there are several concrete cases where using ruby-head is less stable than a preview, and the error is not something that would be valuable to report to Ruby core.

like we may have to do for older Rubies.

Yes, that definitely needs Docker or a VM with a old OS. I don't think it's possible to build 1.8 or 1.9 on anything remotely recent.
My personal opinion is 1.8/1.9 support should be dropped, unless a company sponsors it and also sponsors the needed CI resources & builds for it.

[JonRowe]

3.0.0.preview1 is a fixed in time release, even if its marked as unstable, it's even on the official download page:
https://www.ruby-lang.org/en/downloads/

So do previews, several APIs have already been renamed since preview1 and many bugs fixed.

Those changes are not in preview1 though, they will be in preview2 or rc1 etc.

[eregon]

We don't get people reporting issues from head, where as we do with the official preview versions.

I would treat them exactly the same as reports from people using ruby-head.
To validate the bug, I would check it's not been fixed on ruby-head. If it is fixed on ruby-head, there is IMHO no point to change anything. That illustrates for me that it's always better to test against ruby-head than previews.
Maybe we need to educate users that previews are not stable and bugs are expected.

@JonRowe Could you try ruby-head and if that causes extra failures link them here?

To really consider this, we would need to solve the 4 items above. It seems a lot more work than what it's worth.
It's useful to test against ruby-head all year round (even if allow failures), but there are only a couple previews per year.

[eregon]

FWIW there are Docker builds for each commit of ruby/ruby at https://hub.docker.com/r/rubylang/rubyfarm
Maybe that's a good solution if someone wants to test a fixed development version.
ruby/setup-ruby cannot support a persisted build for each commit, that would be way too much.

[MSP-Greg]

@JonRowe Could you try ruby-head and if that causes extra failures link them here?

From the link above (https://github.com/rspec/rspec-core/actions/runs/343419327), it appears to have passed?

See: https://github.com/rspec/rspec-core/runs/1346985289

[eregon]

I'm thinking having RCs in ruby/setup-ruby might be nice, to catch extra bugs before the real release.
Although, testing ruby-head is likely even better, and doesn't need any CI adjustment to pick up RC-like builds.
And previews are similar from a building point of view.

Interestingly, TravisCI doesn't seem to have 3.0.0-preview1, but it has other previews and RCs: https://rubies.travis-ci.org/

No Windows builds of previews. I don't know anyone making reusable builds of Ruby previews on Windows (yet). Having inconsistent platform support is a source of confusion.

I think this is the main issue, and I have no idea how to solve it.
Maybe we just don't have them on Windows, and if someone asks "there are no Windows preview builds provided anywhere, and we don't have the resources and time to figure out how to build them on Windows".
Building previews on Ubuntu & macOS should be easy, if they do build without any patch.

It's quite a few extra builds, and these builds don't make sense to be used after the real release. Can we drop those builds after the release? Probably not so easy, we would first need to deprecate them before dropping them.

I guess we can just keep them, it's often not so many previews.

There can be many breaking changes between a preview and the next release (e.g. Fiber scheduler & Ractor API is changing recently), so it often doesn't make too much sense to test in CI, at least not for new APIs that have not stabilized yet.

Code in CI is probably unlikely to use new experiemental APIs. But in any case, it's not ruby/setup-ruby's problem, so I think it's fair to ignore this point.
This point is about communicating what a preview is, and it seems there is some confusion about them.

Previews are not more stable than ruby-head in my experience. I'm happy to see counter examples, using setup-ruby (as @MSP-Greg mentioned, the builds here are significantly more stable than TravisCI head builds).

This point still stands IMHO. But it seems in some cases people want to test against both ruby-head and previews like in the run above.

I think @JonRowe wants a good way to answer these issues reported with 3.0.0-preview1. Testing it in CI is one way to ensure it works of course. Other possibilities to address such issues:

• If it fails on preview1 but passes on head: "Try ruby-head instead, previews are unstable builds and that is a bug of the preview."
• If it fails on both preview1 and head, the question is "does RSpec support ruby-head?". If yes, then there is no point to test preview1. If no, or best effort, it still seems far more useful to check against ruby-head.
  Does that make sense?

[JonRowe]

Interestingly, TravisCI doesn't seem to have 3.0.0-preview1, but it has other previews and RCs: https://rubies.travis-ci.org/

Travis uses RVM, any version on RVM is available automatically but they don't autogenerate their docs.