Add a nightly channel

[davidanthoff]

We'll have to think a bit how we update the version DB for that as we don't want to publish a new version in the store every night... We should probably have a separate version db for nightly builds or something like that, so that we can insulate users that don't deal with that from frequent updates of that. We also need to think about an ID scheme for nightly builds. Presumably there is one already? I'm just not familiar with it :)

[musm]

Wouldn't it be better for juliaup to read in the version db from an external source instead of baking in the db version manager with the juliaup source itself, thereby eliminating the need to update juliaup when new nightlies are uploaded?

[davidanthoff]

Having it baked in simplifies things a lot: don't have to handle the situation where a file gets corrupted, don't have to worry about the file version and the binary version drifting apart, don't have to figure out a way to download version file updates in the background somehow etc. The current approach just makes for a very stable and robust solution, with no real downside.

So I think my preference at the moment would be to come up with something special for the nightlies, and leave the "normal" case as is.

[musm]

That makes sense to me. The mechanism used for nightlies can be battle-tested and if deemed reliable enough, could then be used for other channels if so desired in the future.

[ViralBShah]

I just tried juliaup and immediately wanted a nightly channel. Also I wanted the 1.8 pre-releases. Basically I imagined a 1.8 release channel that gets me the latest beta / rc release.

[davidanthoff]

juliaup add beta will get you the 1.8 beta.

[LilithHafner]

As a start, nightlies don't even need a version database. They can just download whatever file lives at https://julialangnightlies-s3.julialang.org/bin/$os_str/$arch_dir/julia-latest-$os_append$arch_append.$extension
See the links on the download page
(This is what I did in UpdateJulia)

[StefanKarpinski]

I think the issue with that may be knowing when to download new nightlies? But perhaps for nightlies it would be fine to try one per day? Or maybe something with ETags? I believe that S3 automatically supports ETags, so this could be pretty simple to implement.

[LilithHafner]

For my use cases, yestaday's nightly is just as good as today's, so a time-based approach would be fine.

ETags would be great if they could be fetched promptly, but the queries I wrote are sometimes quite slow:

julia> @time HTTP.request("HEAD", "https://julialangnightlies-s3.julialang.org/bin/mac/x64/julia-latest-mac64.dmg")
 24.274753 seconds (632 allocations: 100.922 KiB, 0.05% compilation time)
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 118106979
x-amz-id-2: Gbv+lpFooK5eqE8rceZGyb8gqx5r/tmBXwwLM+i5EugUBqaB2DK31ebTMTM+s4NqdZpFJY7fIrY=
x-amz-request-id: BBF9C2X78WGEEH5Q
Last-Modified: Tue, 21 Jun 2022 20:10:10 GMT
x-amz-expiration: expiry-date="Sun, 21 Aug 2022 00:00:00 GMT", rule-id="Cyberduck-u1YrSIk1"
ETag: "bc45bc4361d20c7f72a92bb2481d77ab-15"
Content-Type: application/x-apple-diskimage
Server: AmazonS3
Accept-Ranges: bytes
Age: 268
Date: Wed, 22 Jun 2022 14:32:40 GMT
Via: 1.1 varnish
X-Served-By: cache-nrt-rjtf7700065-NRT
X-Cache: HIT
X-Cache-Hits: 0
X-Timer: S1655908361.691796,VS0,VE72

"""

[StefanKarpinski]

Oof, yeah, that's quite slow. What about spinning off a background process that does the ETag thing, updates if there's an update to be had and then exits. That way the nightly would be available the next time you run the command? Would require a bit of locking, of course, but I think that's already needed.

[LilithHafner]

That sounds reasonable. I think nightly release channel is one of the few cases where it is acceptable to perform automatic silent background updates (i.e. triggered when a user invokes $ julia +nightly).

Even with automatic updates in this manner, it is possible for a user to invoke $ julia +nightly and get an old version. In this case, we should prompt them in a similar way we prompt folks with an out of date release or beta channel with an additional hint that an automatic update is underway.

Ideally, we can still support $ juliaup update nightly, even with automatic updates. If an automatic update is already underway, $ juliaup update nightly blocks until it is complete.

[StefanKarpinski]

That sounds like a great plan to me.

[visr]

Since #131 got fixed https://julialang-s3.julialang.org/bin/versions.json is used to find new versions. That JSON doesn't have nightlies. Though it does have the 1.9.0-alpha1 for instance, and with be6719e that is available in the alpha channel.

So I guess we need to decide if nightlies belong in versions.json, and if not, juliaup needs to get it from julialangnightlies-s3 outside of the version database. Either way the proposed fixes of #197/#384 won't apply anymore.

[Roger-luo]

So I guess we need to decide if nightlies belong in versions.json, and if not, juliaup needs to get it from julialangnightlies-s3 outside of the version database.

I'm wondering what would be the problem to get nightly from julialangnightlies-s3? I think download it directly from there would be less changes to upstream? Nightly has a frequently changed version commit sha anyway.

[visr]

Yeah indeed, I didn't mean to imply that julialangnightlies-s3 was a problem, just that versions.json related changes caused the previously submitted PRs for nightly to no longer be an option.

[ulysses4ever]

Another case in point that this should be done: Julia GtiHub action has nightly. It'd be nice to be on par with it. E.g. if the nightly-based GitHub workflow fails, it's nice to have a quick way to try it out locally.

[Krastanov]

For what is worth, it is trivial to do this yourself using the link option in juliaup in order to link to a custom executable

# download the nightly
 wget https://julialangnightlies-s3.julialang.org/bin/linux/x86_64/julia-latest-linux-x86_64.tar.gz -O /tmp/julia.tar.gz
# remove the old nightly
 rm -rf ~/.julia/juliaup/julia-nightly
# set up the new nightly and clear up the folder name
 mkdir ~/.julia/juliaup/julia-nightly
 tar -xf /tmp/julia.tar.gz -C ~/.julia/juliaup/julia-nightly
 mv ~/.julia/juliaup/julia-nightly/julia-*/* ~/.julia/juliaup/julia-nightly/

The first time you do this you also need juliaup link nightly ~/.julia/juliaup/julia-nightly/bin/julia.

Now I have a nightly managed by juliaup, and a non-juliaup command line shortcut to update it.

[ulysses4ever]

@Krastanov this is a good workaround, maybe worth posting somewhere visible. But also this is exactly the kind of code I was hoping to avoid when getting juliaup…

[LilithHafner]

Bump, this is the last thing we need to reach feature parity with UpdateJulia.jl

[davidanthoff]

Just FYI, I'm currently taking the PR from @maleadt that we reverted and am modifying that a bit with some of the new strategies we discussed since then. Will hopefully have a new PR soon.