Proposal for simplifying testing of pre-releases

[KristofferC]

Julia has in general three types of categories of releases. The LTS (currently 1.0.x), the latest release (currently 1.6.1), and the nightly release.

Before a new Julia release, there are typically a set of prereleases. These are betas (for when there are still known issues in the release) and release candidates (RCs) (where there are no known issues). We want as many people as possible to test these prereleases to find issues in the prereleases so that we don't get a bad final release.

It is possible to test prereleases on CI right now. For example, you can use 1.7-nightly to test the release-1.7 branch and you can use ^1.7.0-0" to test the latest tagged 1.7 prerelease.

The issue is:
- Not a lot of people know about this
- You need to update your CI script every time you want to test the prerelease of the new upcoming release.

This makes it so that not many people run CI on the prereleases and it is common to get quite a few bug reports in the week after the final release of a version.

To remedy this I think it would be useful if one could be able to test the LTS prerelease and the prerelease of the upcoming release without changing your CI script. However, we do not want to run these tests all the time (most of the time there does not exist a prerelease so running CI on the release-1.x branch is just a waste).

Therefore, I propose adding the following "versions" that one can use in CI (not sure if the names are the best):

• LTS-prerelease
• latest-prerelease

For example, latest-prerelease would currently map to 1.7.0-beta3.

What is special about the prerelease versions is that they only run if there is a prerelease "in progress". This requires us to put some mutable state that indicates that there is a prerelease ongoing for the LTS or the latest release. I propose putting that information in the https://julialang-s3.julialang.org/bin/versions.json file since the script already needs to read that file. For example, there could be an entry:

"LTS-prerelease" = {
     "active": false,
     "files": [ .... ],
    .... 
}

If active is set to false the prerelease version CI scripts should immediately return success. Otherwise, CI should run as normal with the files given. This allows people to unconditionally put the prerelease versions in their CI scripts and whenever a prerelease is going on, we flip the switch in the json file and CI will automatically start running with the prerelease and we can get a lot of feedback quickly.

[staticfloat]

Sounds reasonable to me; how do we dynamically add these jobs into CI runs though? Most CI setups (e.g. GHA) don't allow for dynamic job creation. I guess we could have jobs that just instantly pass if there is no prerelease version?

[SaschaMann]

Most CI setups (e.g. GHA) don't allow for dynamic job creation.

GHA allows it, but it complicates the workflow too much for a standard copy-pastable CI workflow, I think.

I think a no-op and updating the title of the job in the PR view to indicate whether it found a version would work. (not sure if the token scope is broad enough to update the job title)

---

I think the main problem that needs to be solved for this would be updating versions.json. The current setup (or any GHA-based setup, I think) wouldn't allow for frequent updates, and if we want checksums, it'd likely have to be integrated into the build process of those pre-release binaries. It may also make sense to split it into a separate file for mutable versions.

[KristofferC]

how do we dynamically add these jobs into CI runs though?

I wrote

If active is set to false the prerelease version CI scripts should immediately return success.

so the job is still there, it just succeeds without running anything.

[KristofferC]

I think the main problem that needs to be solved for this would be updating versions.json. The current setup (or any GHA-based setup, I think) wouldn't allow for frequent updates,

It should be able to work analogously to how nightly does, I thought.

[DilumAluthge]

I think the main problem that needs to be solved for this would be updating versions.json. The current setup (or any GHA-based setup, I think) wouldn't allow for frequent updates,

It should be able to work analogously to how nightly does, I thought.

If I understand correctly, we currently do not have nightly in the versions.json file.

[KristofferC]

Aha, okay. Maybe that info should be stored somewhere else then.

[SaschaMann]

(most of the time there does not exist a prerelease so running CI on the release-1.x branch is just a waste).

I misread this part of the original post, sorry. I thought latest-prerelease was meant to point at the binaries of the release-1.7 branch.

---

If those binaries are irrelevant and only tagged pre-releases need an alias, my suggestion would be a separate file aliases.json that only includes aliases but not file information such as file names or checksums. The latter are already included in versions.json anyway.

It would be possible to include this information in versions.json. However, I feel like it may be better to separate mutable and immutable information. It’d also be breaking and somewhat awkward because it would require adding a key/object to it that doesn’t match the existing schema, e.g.

{
  "0.7.0-rc2": {
    "files": [],
    "stable": false
  },
  "0.4.0": {},
  "1.6.0": {},
  "aliases": {
    "latest-pre-release": {},
    "lts-pre-release": {}
  }
}

would break all keys being versions.

aliases.json could look like this:

{
    "latest-prerelease": {
        "active": true,
        "version": "v1.7.0-beta3"
    },
    "LTS-prerelease": {
        "active": false,
        "version": ""
    }
}

The information whether an alias is active and what version it points to could be inferred from the list of git tags in that repo and standard semver rules.

How this could work:

• Builds can be triggered automatically by a tag push event in julialang/julia.
• Determine whether the pushed tag is a pre-release for either the LTS or latest version, or if it's a full release that deactivates a previous pre-release.
• setup-julia checks if the version the alias points to exists in versions.json. If there isn't one, treat it as if the pre-release was inactive. This avoids issues in the time period between the creation of the git tag and the availability of the binaries.
• Downside: active: true does not imply that binaries for the pre-release are available.
  • There might be situations where a build will pass silently as a no-op as if there wasn’t an active pre-release due to some kind of genuine issue with versions.json. I feel like this would be noticed rather quickly and isn't a big concern in practice.
• The active key primarily acts as a shortcut to avoid downloading versions.json in cases where we know with certainty there isn't a pre-release.

To avoid the downsides, it would be possible to run the build script manually or on a frequent schedule. This has other downsides like wasting CI time, times where a pre-release has been announced but the script hasn't run yet, and/or more manual work for the release managers.

The logic that generates the file should probably live in a small unregistered Julia package in a separate repo in julialang, to make it easier to add some tests for it. That package/script would then be called from a workflow in julialang/julia.

In setup-julia, getJuliaVersion would be extended to check whether the version given as input to the action matches an alias. If it does, treat it as the version the alias points to:

setup-julia/src/installer.ts

Lines 77 to 93 in f577b7c

	export function getJuliaVersion(availableReleases: string[], versionInput: string): string {
	if (semver.valid(versionInput) == versionInput || versionInput.endsWith('nightly')) {
	// versionInput is a valid version or a nightly version, use it directly
	return versionInput
	}
	// Use the highest available version that matches versionInput
	let version = semver.maxSatisfying(availableReleases, versionInput)
	if (version == null) {
	throw new Error(`Could not find a Julia version that matches ${versionInput}`)
	}
	// GitHub tags start with v, remove it
	version = version.replace(/^v/, '')
	return version
	}

---

Does this sound reasonable?

[SaschaMann]

As a side effect, this would allow an alias lts = 1.0.6 which has been requested before.

[ViralBShah]

Can we have something where if an active pre-release exists, it gets run, otherwise it just exits immediately with a green?

[SaschaMann]

Can we have something where if an active pre-release exists, it gets run, otherwise it just exits immediately with a green?

I'd argue given that the purpose of the action is to install Julia of a given version, failing to do so because a pre-release doesn't exist shouldn't silently exit as green. However, the workflow can be configured to handle this case and ignore the error and exit immediately anyway.

[ViralBShah]

Makes sense. I am not intimately familiar with the details of the right way to implement this, but would love to have it.

What would help me (and perhaps others) is to have a permanent rule in package CIs to test the current active pre-release (if there is one at a given time or silently exit). This will help greatly in having the package ecosystem be ready for new releases of Julia. Otherwise as a package maintainer, I tend to only look at packages after the latest stable Julia release (since that is the most convenient thing).

[SaschaMann]

I am not intimately familiar with the details of the right way to implement this, but would love to have it.

Having thought more about it, probably the simplest solution would be to introduce an input to setup-julia like continue-if-prerelease-doesnt-exist (subject to bikeshedding), so that one would have

- uses: julia-actions/setup-julia@v1
  with:
    continue-if-prerelease-doesnt-exist: true
    ...

if desired. Then internally, the action could check if the version is a prerelease and exit with 0 despite not being able to install it. We could probably argue that that should be the default setting for prerelease version ranges, too, given that the vast majority of use cases of this feature would be CI.

Exiting with a nonzero nonstandard exit code and dealing with it on the workflow level generally seems a bit cleaner but given the vast majority of Julia package devs know very little about Actions beyond copying and setting up CI, the above would probably be the more pragmatic option.

[ViralBShah]

Going with the pragmatic option makes sense. As a package dev, I can confirm I don't know CI internals, but am happy to paste stuff in and keep it green.

[topolarity]

Rather than have a version that does "nothing" sometimes, why don't we take a cue from juliaup:

beta: always points to the latest beta version if one exists. If a newer release candidate exists, it will point to that, and if there is neither a beta or rc candidate available it will point to the same version as the release channel.

This behavior would be good enough for CI, imo.

Even better would be to have all of the juliaup channels available in CI - I believe the existing alpha channel is equivalent to the latest-prerelease proposed here.

It would obviously be ideal if the CI workers ignore any duplicate version requests, but I don't think that's required. These channels are very useful, even if they waste a bit of CI time.

[SaschaMann]

It would obviously be ideal if the CI workers ignore any duplicate version requests, but I don't think that's required. These channels are very useful, even if they waste a bit of CI time.

Given that almost 21k repos use the action, presumably most of those using it for CI, it'd be best to find a solution that avoids wasting CI time. Even if it's "just" a few seconds per run, it adds up extremely quickly to become a major waste of resources.

---

Sorry I haven't really had time to spend on open source stuff at all in the recent months, so there's no progress on this from my side. (On this note, thanks a lot @ViralBShah, @DilumAluthge, @IanButterworth for taking care of the action!)

[ViralBShah]

I think a little bit of CI time wasted for now is a good immediate solution if a better one is not immediately obvious - since it would save a lot more time scrambling to fix things. Also, there are a lot of CI time saving gains possible by removing julia nightly testing in many packages if we can have pre-release testing. (I think most nightly testing exists largely because of copy-paste).

[ViralBShah]

Does #234 fully address this?