Nightly builds

[xermicus]

I want to have nightly builds of the main branch (given there was at least one commit since the last nightly build).

This should be re-using the logic from the existing release process - however if possible I'd like to not create a GH release for each nightly build but just publish them to resolc-bin directly.

[alvicsam]

Out of curiosity: why don't you want to have them in releases in this repo? If not how do you see that? Pushing binaries directly into https://github.com/paritytech/resolc-bin? In this case there is a problem with git, the repo size is going to be enormous and slow in a couple of months.

We also could create nightly releases in the resolc-bin repo. We don't need to append them to the json files, right?

[xermicus]

Intuitively I just feel like having nightly releases will spam the release page. Users will end up scrolling endlessly to find any older one.

Just checked solc, they also don't do it. Instead they push to their binaries repo. I thought we could do the same?

We don't need to append them to the json files, right?

I don't see them for example here so I guess no. CC @smiasojed

[alvicsam]

Yes, but on the other hand they don't put binaries of nightly releases into the solc-bin repo. You can see that for example in the linux-amd64 folder they have about 90 binaries for the 5 years. I tried a git clone of the repo and after 6 minutes it could download only 24%. Revive has 340 MB of binaries on each release. You can image how bad it's going to be in 3 months if we put binaries into git. Also the ci workflow that generates json files is going to stop working at some point because it checks out the resolc-bin repo.

[xermicus]

Ah okay. I just saw commits mentioning "nightly". But seemingly only for the JS version.

And you are right, trying to access the solc-bin repo via the web UI doesn't really work. This doesn't seem to be ideal. I am open to alternative solution.

[alvicsam]

If the nightly builds needed only for small amount of time we can have a nightly pipeline that will build artifacts and keep them for a few days. According to github docs they can be kept for 90 days.
Another solution would be creating a revive-nightly-releases repo and do the nightly releases there.

[smiasojed]

Why do we need to push binaries to the resolc-bin repo? Don’t we already reference binaries via JSON files pointing to releases in the revive repo?
If you don't want to include nightly references in our current list.json files, maybe we should create a nightly/ subdirectory, or use the nightly suffix like list_nightly.json instead.

[xermicus]

Why do we need to push binaries to the resolc-bin repo?

The nightly binaries need to live somewhere. But maybe it can be just in the action artifacts cache?

The problem with nightly pipelines deleting artifacts is that users might deploy contracts built by those artifacts. When the compiler builds are deleted it's much harder to reproduce contract builds. However maybe I am too paranoid. We will just state that the nightly builds should never be use for production deployments.

So what seems appealing to me: Run nightly release pipelines but don't create a release. Instead set the retention period to something reasonable (like 1 month) and link those artifacts somewhere in resolc-bin (as opposed to my initial suggestion of pushing them there). @alvicsam WDYT?

[alvicsam]

I would opt for a separate repo for nightly builds json files and keeping files as artifacts for 30 days.

So let's agree on technical part:

1. Create a nightly dir in resolc-bin and the same subdirs as in the root (linux, macos, wasm, windows)
2. Adjust current Build & Release pipeline to run nightly if there were new commits and keep artifacts for 30 days
3. Create json files in nightly subfolders with links to artifacts. For a version I suggest using nightly-YYYY-MM-DD

Are there any additional suggestions/questions/objections?

[xermicus]

Sounds good to me! Except I don't understand what the separate repo is used for then? Wouldn't the pipeline run here (existing revive repo) and commit the link to the artifacts into the resolc binary repo (also already existing)?