Provide PR artifacts and nightly builds

[umarcor]

Describe the project you are working on:

Not relevant. This request is usable for anyone willing to use/test development versions of Godot.

Describe the problem or limitation you are having in your project:

If some issue/problem is fixed by a not-yet-merged PR, users willing to test if that specific version fits their needs are forced to checkout and build Godot themselves. https://github.com/godotengine/build-containers provides some helpful resources, but it is still a workflow for developers.

The same issue applies to users willing to try branch master at any time.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

In GitHub Actions workflows, artifacts can be easily made available for any push and/or PR. See https://docs.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts: "For pushes and pull requests, GitHub stores artifacts for 90 days".

Then, any user would be able to browse the runs of some PR, download the godot tarball for their platform, and try it. Those artifacts are hidden enough not to be confused with "official" releases, but easy enough to find for users/contributors to have a quick feedback. See, for instance, https://github.com/umarcor/hwstudio/actions/runs/223019994 (the repo itself is a facade for now).

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

There is an official Action maintained by GitHub: https://github.com/actions/upload-artifact

- uses: actions/upload-artifact@v2
  with:
    name: my-artifact
    path: path/to/artifact/ # or path/to/artifact

Hence, adding those four lines to the existing workflows would suffice.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

Contributors might enable it in their forks, but it would be really cumbersome.

Is there a reason why this should be core and not an add-on in the asset library?:

This question does not apply.

Nightly builds

As an optional feature, a pre-release GitHub Release can be used to provide Godot tarballs which are in sync with master. Action https://github.com/eine/tip/ allows to gather all the artifacts of a workflow and push them as assets of a pre-release. At the same time, the git tag is updated, so that the pre-release points to the commit that produced the latest available tarballs/assets. See, for example:

• https://github.com/ghdl/ghdl/blob/nightly/.github/workflows/push.yml#L156-L168
  • https://github.com/ghdl/ghdl/releases/tag/nightly
• https://github.com/umarcor/hwstudio/blob/godot/root/.github/workflows/build.yml#L92-L138
  • https://github.com/umarcor/hwstudio/releases/tag/nightly

Ref: https://hugo.pro/projects/godot-builds/

[Calinou]

We can't make use of GitHub Actions' Artifacts feature as we'd run into the 1 GB size limit very quickly.

We could use the GitHub Releases approach, but…

At the same time, the git tag is updated, so that the pre-release points to the commit that produced the latest available tarballs/assets.

Doing this will notify repository watchers every time a new nightly build is created, even if they chose the "Releases only" notification level. It'd be better to keep the tag outdated and update the release description instead to avoid notificating users endlessly.

The downside with using GitHub Releases is that we can't host PR builds this way. At this point, we may want to look into using TuxFamily (if it has enough storage) or rent a storage-focused VPS to host all the builds.

[Zireael07]

What happens when you run into the 1 GB limit? Does it only keep the latest x artifacts then?

[Calinou]

@Zireael07 I believe it prevents you from publishing new artifacts, rather than removing the old ones automatically. Either way, even if it removed the old ones automatically, we'd only be able to keep like 2 commits built (including PRs!).

[Zireael07]

Ouch. Both of those things put a fairly big damper on the idea.

[hcorion]

Just happened to come across this issue, this is something we do over at https://github.com/RPCS3/rpcs3.
For our "nightly" builds we auto-deploy to a seperate repository https://github.com/RPCS3/rpcs3-binaries-win (we have seperate repos for different OSes) which acts as a free persistent backup of all our commit builds, very helpful for regression testing. See our .ci folder in the main repo for a little more detail on how that works.

For PR builds we rely on Microsoft Azure's free for open source pipeline artifacts system https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts. See our azure-pipelines.yml.
Appveyor is also an alternative that offers free for open source (but time-limited, ie not persistent same with Azure) artifact hosting https://www.appveyor.com/docs/packaging-artifacts/.

[umarcor]

We can't make use of GitHub Actions' Artifacts feature as we'd run into the 1 GB size limit very quickly.

@Calinou, I'm not sure about that. I'm sure there is a limit, and uploading might be blocked when the limit is reached (i.e., it is NOT a FIFO). However, 1GB seems unreal to me. Might it be that there is a limit of 1GB per artifact? Can you please provide the reference?

• Size limits? actions/upload-artifact#9
• Artifacts exceeded limits actions/upload-artifact#59

Furthermore, I'm not sure if the limit applies to free/open source projects. I believe it might be for private repos/orgs. Should there be a limit, there are actions for pruning older artifacts. However, I agree that it would be undesirable to need such solution.

We could use the GitHub Releases approach, but…

At the same time, the git tag is updated, so that the pre-release points to the commit that produced the latest available tarballs/assets.

Doing this will notify users every time a new nightly build is created, even if they choose the "Releases only" notification level. Doing this is too much. It'd better to keep the tag outdated and update the release description instead to avoid notificating users endlessly.

No, users are not notified. That's a problem that existed with early versions of eine/tip, because the release was regenerated (re-published) each time. That was really annoying (eine/tip#111). Now, eine/tip force pushes the tag only, which does not trigger a "new release" notification.

The description of the pre-relase itself is kept empty. Hence, GitHub shows the commit message of the tag. That is, when the tag is updated (force-pushed), the description of the pre-release is updated automatically. Yet, if maintainers write the description manually, that one is used (and preserved).

The only caveat of this approach is that the "date" of the pre-release is the one when the tag was first released. The commit sha, the description, the assets, etc. all of them can be updated, but not the date. Actually, doing so is what would trigger notifications.

we'd only be able to keep like 2 commits built (including PRs!).

In the worst case scenario, I believe that updating 'nightly' from 'master' only would be an easy to achieve and still useful solution.

• It is possible to schedule it daily, weekly, etc. instead of running it for each push to master. So, it can be guaranteed that the limit is never reached within 90 days).
• Once the plumbing is available in the main repo, users/contributors can fork it, comment the condition and have the artifacts generated. Hence, any user can pick a PR, push it to their own fork, and have the artifacts generated without accounting for godotengine's limit.

For our "nightly" builds we auto-deploy to a seperate repository RPCS3/rpcs3-binaries-win (we have seperate repos for different OSes) which acts as a free persistent backup of all our commit builds, very helpful for regression testing. See our .ci folder in the main repo for a little more detail on how that works.

@hcorion I guess you do this because you generate a new release for each run, and you want to avoid spamming notifications, isn't it? Wouldn't it be equivalent to use a single (pre-)release, since artifacts are distinctly named already? Note that this is done in https://github.com/msys2/msys2-autobuild/releases, where a single repo is used and a pre-release exists for the "nightly" builds of each "platform".

For PR builds we rely on Microsoft Azure's free for open source pipeline artifacts system docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts. See our azure-pipelines.yml.

I thought that GitHub Actions was, indeed, Microsoft Azure (just a rebranding). Are artifact limits in Azure different from GitHub?

[Zireael07]

Looks like you're right and the storage limit is only for private repos: https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions

[KoBeWi]

users willing to test if that specific version fits their needs are forced to checkout and build Godot themselves

Only building is necessary, to test changes you can just download the patch file (by adding .patch to PR url) and use git apply. Or use this one-line command: git pull origin refs/pull/0000/head ; git reset --soft HEAD~1 (where 0000 is the PR id).

[Zireael07]

@KoBeWi That requires "users" to be familiar with git. And I assume the "users" the OP mentions are just people wanting to just try out Godot. (Also "only building is necessary" is still a fairly big hurdle - we routinely get issues related to the build tooling, someone can't run scons, someone's scons spits out errors, someone only has MinGW, etc. etc. )

[umarcor]

@KoBeWi, as @Zireael07 said, with "users" I mean "users of Godot". That is, developers who are familiar with git, and who would potentially be able to build godot from sources; but who don't want to do it. The main reason is not to install/setup all the build dependencies. This is because "users of Godot" (i.e. "game/tool developers") do not need to install Godot's build dependencies in order to use it.

More specifically, I downloaded Godot_v3.2.2-stable_win64.exe.zip on Windows three days ago. I extracted it, and was ready to go. Now, I find a bug in the engine that seems to be fixed in master or by PR0000. I want to download the equivalent zipfile for master/PR0000, extract it, and open my project with that version of Godot. It that works, I can provide very quick feedback to the contributor(s).

Of course, ideally any user willing to try development versions of Godot would also be willing to build it from sources. However, in practice, many people cannot, because they don't know how or because they are not allowed to install tools on the machine they are using. In both cases, time is a limiting factor, and that's what this issue/request tries to cut down.

[hcorion]

@hcorion I guess you do this because you generate a new release for each run, and you want to avoid spamming notifications, isn't it? Wouldn't it be equivalent to use a single (pre-)release, since artifacts are distinctly named already? Note that this is done in https://github.com/msys2/msys2-autobuild/releases, where a single repo is used and a pre-release exists for the "nightly" builds of each "platform".

The reason we do this is because we can easily query for old builds without spamming users with new tag messages, we have a build archive which ties in with this system here https://rpcs3.net/compatibility?b. If I understand the mingw system you don't have access to pre-existing builds.
Something that's really nice about storing all the builds like our approach is that if a non-technical user discovers a regression (in our case it would just be a regular user, but in Godot's case it could be say an artist), they simply go the build archive and start downloading old builds and don't need to know the ins and outs of patch files and deal with the annoyances of compiling 10 or so different builds.
Regressions happen often in RPCS3, so it works well for our use case, but if Godot doesn't have regressions very frequently it could be a non-issue.

[umarcor]

The reason we do this is because we can easily query for old builds without spamming users with new tag messages, we have a build archive which ties in with this system here rpcs3.net/compatibility?b.

@hcorion, that's what I meant: you don't need one (or multiple) additional repos to avoid spamming users. You can achieve the same result using a pre-release (or multiple) in the main repo. The trick is to create one realease for each "archive", and then manipulate the assets only.

If I understand the mingw system you don't have access to pre-existing builds.

I'm not sure to understand the question. In msys2, it is decided to keep the last attempt only, because it is a rolling environment. However, it is possible to know which run produced each asset, should that be required.

Anyway, the (single) point I am trying to draw attention to is that spamming is only an issue if you create or recreate releases, not if you just update the content.

Something that's really nice about storing all the builds like our approach is that if a non-technical user discovers a regression (in our case it would just be a regular user, but in Godot's case it could be say an artist), they simply go the build archive and start downloading old builds and don't need to know the ins and outs of patch files and deal with the annoyances of compiling 10 or so different builds.

Regressions happen often in RPCS3, so it works well for our use case, but if Godot doesn't have regressions very frequently it could be a non-issue.

I understand that your solution has longer persistence than 90 days. But I believe the resource usage (even if free) is not worth the gained functionality. In the case of godot, I am proposing a persistence of a few days. GitHub provides 90, but I think that a week would suffice.

[Arrow-x]

I have a low end CPU so Building Godot takes hours for me. I never used actions before, so can it stack PRs, like many PRs in the same Godot build??

[umarcor]

@Arrow-x, yes. You would:

• Create a new branch off master.
• Add actions/upload-artifacts as a last step in the existing workflow.
• Cherry-pick or rebase the commits from the PRs you want to stack.
• Push to GitHub.

The workflow will run, Godot will be built, and the result will be available to you as a zipfile.

In fact, you can already achieve it, regardless of the official repo picking this approach.

[umarcor]

@Arrow-x, interestingly, the feature is already in the workflow for windows (only): https://github.com/godotengine/godot/blob/master/.github/workflows/windows_builds.yml#L64-L71. It seems that @aaronfranke added it last month (godotengine/godot@e5df136) but it was commented due to the space concerns that arised above too.

Anyway, the point is that you can just uncommented that, and you are ready to go. No need to add actions/upload-artifacts yourself (unless you want artifacts/assets for other platforms).

[aaronfranke]

@umarcor No, that commit is just splitting/renaming the files. If you click "View blame prior to this change" when using blame on GitHub, you can get closer to the actual source: https://github.com/godotengine/godot/blame/60fab23262500789145b83738745fee9de58b7f0/.github/workflows/main.yml#L59

[umarcor]

@aaronfranke, thanks. Pinging @RevoluPowered as the author of godotengine/godot#40282 and, I assume, of the commit message: "I left the 'publish artefacts' disabled until we can request more storage from Microsoft, 5 GB is far to low for us and we would eat this limit very fast. (it is tested and works fine)". @RevoluPowered, where is that "5GB" coming from? Did you actually have size limit issues in this org or was it disabled for being on the safe side?

[Arrow-x]

is there a way to get the Mono version of the Godot window editor?

[umarcor]

@umarcor should I enable the actions for the templates too? and should I disable the other platforms?
(Sorry to derail this proposal )

Adding the artifact upload step for the templates is up to you. That depends on you wanting to try some fix in the editor itself or something that corresponds to templates too. Also, I don't know wether templates need to match the build commit of the editor.

Regarding disabling other platforms, again, up to you. If you are not going to use those artifacts, you might well remove those workflow files from your branch. However, if you are combining several PRs, you might want to keep building all the workflows/platforms, as that might let you know about possible conflicts.

For Mono, check the linux workflow: https://github.com/godotengine/godot/blob/master/.github/workflows/linux_builds.yml. Precisely:

• https://github.com/godotengine/godot/blob/master/.github/workflows/linux_builds.yml#L12
• https://github.com/godotengine/godot/blob/master/.github/workflows/linux_builds.yml#L134

So, you need to copy https://github.com/godotengine/godot/blob/master/.github/workflows/windows_builds.yml#L67-L71 in that workflow, and upload ./bin/godot.linuxbsd.opt.tools.64.mono.

[Arrow-x]

wouldn't that give me a Linux Mono Editor, I am on Windows? Should I change the Scons commands in the Linux_Builds.yml to compile a Windows build ?

[umarcor]

@Arrow-x, unfortunately, that I cannot tell, since I am not familiar with Mono at all. Nonetheless, it is sctrictly unrelated to CI/artifacts. That is, you should be able to read the "regular instructions" (https://docs.godotengine.org/en/latest/development/compiling/) and apply them in this context.

[Arrow-x]

@umarcor thank you so much for indulging me, and I apologize for taking so much of your time

[Xrayez]

wouldn't that give me a Linux Mono Editor, I am on Windows? Should I change the Scons commands in the Linux_Builds.yml to compile a Windows build ?

@Arrow-x there was some unnoticed bugs in the past with Mono on Windows recently: godotengine/godot#41770, so issues like this could justify compiling Windows Mono in the CI, makes sense for me. Could also be a good opportunity to actually run all those procedures which require to build Godot with Mono (generating mono_glue etc.)

[umarcor]

As discussed in godotengine/godot#42422, I'm closing this in favour of #1458.