Add a nightly channel

[davidanthoff]

We'll have to think a bit how we update the version DB for that as we don't want to publish a new version in the store every night... We should probably have a separate version db for nightly builds or something like that, so that we can insulate users that don't deal with that from frequent updates of that. We also need to think about an ID scheme for nightly builds. Presumably there is one already? I'm just not familiar with it :)

[musm]

Wouldn't it be better for juliaup to read in the version db from an external source instead of baking in the db version manager with the juliaup source itself, thereby eliminating the need to update juliaup when new nightlies are uploaded?

[davidanthoff]

Having it baked in simplifies things a lot: don't have to handle the situation where a file gets corrupted, don't have to worry about the file version and the binary version drifting apart, don't have to figure out a way to download version file updates in the background somehow etc. The current approach just makes for a very stable and robust solution, with no real downside.

So I think my preference at the moment would be to come up with something special for the nightlies, and leave the "normal" case as is.

[musm]

That makes sense to me. The mechanism used for nightlies can be battle-tested and if deemed reliable enough, could then be used for other channels if so desired in the future.

[ViralBShah]

I just tried juliaup and immediately wanted a nightly channel. Also I wanted the 1.8 pre-releases. Basically I imagined a 1.8 release channel that gets me the latest beta / rc release.

[davidanthoff]

juliaup add beta will get you the 1.8 beta.

[LilithHafner]

As a start, nightlies don't even need a version database. They can just download whatever file lives at https://julialangnightlies-s3.julialang.org/bin/$os_str/$arch_dir/julia-latest-$os_append$arch_append.$extension
See the links on the download page
(This is what I did in UpdateJulia)

[StefanKarpinski]

I think the issue with that may be knowing when to download new nightlies? But perhaps for nightlies it would be fine to try one per day? Or maybe something with ETags? I believe that S3 automatically supports ETags, so this could be pretty simple to implement.

[LilithHafner]

For my use cases, yestaday's nightly is just as good as today's, so a time-based approach would be fine.

ETags would be great if they could be fetched promptly, but the queries I wrote are sometimes quite slow:

julia> @time HTTP.request("HEAD", "https://julialangnightlies-s3.julialang.org/bin/mac/x64/julia-latest-mac64.dmg")
 24.274753 seconds (632 allocations: 100.922 KiB, 0.05% compilation time)
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 118106979
x-amz-id-2: Gbv+lpFooK5eqE8rceZGyb8gqx5r/tmBXwwLM+i5EugUBqaB2DK31ebTMTM+s4NqdZpFJY7fIrY=
x-amz-request-id: BBF9C2X78WGEEH5Q
Last-Modified: Tue, 21 Jun 2022 20:10:10 GMT
x-amz-expiration: expiry-date="Sun, 21 Aug 2022 00:00:00 GMT", rule-id="Cyberduck-u1YrSIk1"
ETag: "bc45bc4361d20c7f72a92bb2481d77ab-15"
Content-Type: application/x-apple-diskimage
Server: AmazonS3
Accept-Ranges: bytes
Age: 268
Date: Wed, 22 Jun 2022 14:32:40 GMT
Via: 1.1 varnish
X-Served-By: cache-nrt-rjtf7700065-NRT
X-Cache: HIT
X-Cache-Hits: 0
X-Timer: S1655908361.691796,VS0,VE72

"""