REQUEST: Permalink to latest nightly builds — for QA/CI pipelines !

[holta]

Can Kiwix consider helping QA Automation?

As requested by @deldesir in Haiti who wants to help modernize QA with CI scripts to help with CI integration testing using kiwix-tools, libzim and others — to surface critical issues efficiently for https://Internet-in-a-Box.org field communities and others!

That's CI in every sense (Continuous Improvement not just Continuous Integration !) allowing ongoing semi-automated smoke testing & functional testing of https://download.kiwix.org/nightly and https://download.openzim.org/nightly daily (i.e. nightly) builds.

QUESTION: Could this be solved similar to the same way Kiwix's "release" channel solves this today, below...?

EXAMPLE: https://www.kiwix.org/en/downloads/kiwix-serve/ links to the latest release builds such as...

• https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-x86_64.tar.gz
• https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-armhf.tar.gz

(So that manual work digging up version numbers like 3.3.0-1 and dates like 2022-10-04 is completely avoided — allowing community QA feedback loops to become MUCH tighter & MUCH more efficient !)

SUMMARY: The request is that Kiwix consider allowing QA Test Automation — by assisting with permalink direct links to nightly builds — so the "latest nightly" channel can become much more useful for Community QA / Test Automation 🙏

 

CLARIFICATION: Nightly builds break regularly and that is perfectly normal & understandable! Anybody who takes QA seriously knows this is perfectly normal on dev branches! No Worries At All ☺️

 

ASIDE: WordPress solves this same need for permalinks in a slightly different way, with URL Redirects from things like...

https://wordpress.org/latest.zip

Which today happens to redirect to...

https://wordpress.org/wordpress-6.0.2.zip

(i.e. just like file-level symlinks / permalinks, but redirecting to another URL showing the actual version number in the URL right away! Either solution works great [filesystem-level or URL-level permalinks] if Kiwix can hopefully consider either idea!)

[rgaudin]

That's quite a long text for a simple request 😉 Answer is “sure”. I can cook something quickly if you reduce the scope by listing exactly those you'd want: not all projects follow same naming convention so we'll have to extend later.

[kelson42]

@rgaudin If we could have a convention to follow to ease that work that would be even better, so we could fix the few problematic filenames.

[rgaudin]

Yes, that would probably be for the best. I think the dated versions (majority of them) are the way to go. It's explicit. We'd need to fix the ones without suffixes and the JS ones with an hash.

[kelson42]

Is kiwix-tools_linux-i586-2022-10-05.tar.gz correct and can be take as example?

[rgaudin]

Yes, it is.

[kelson42]

@mossroy @Jaifroid Possible for you for kiwix-js and kiwix-js-windows repositories?

[Jaifroid]

@kelson42 Are you asking whether we can provide permalinks to the latest nightly build for each package, or whether we can update the filenames to include the date?

I never added the date, because it seemed redundant to me (the package is inside a folder that is named after the date), so instead each package has the SHA of the last commit of the branch being built, which seems more useful, as there could be several commits on one day.

[kelson42]

@Jaifroid I'm asking to rename the files following the norm given by @rgaudin so we can have a common format to then easily create permalinks.

[rgaudin]

Yes, renaming the files as to have a unique pattern: similar to the release files but instead of the version, it's the date. Obviously this should be same date as the folder name.

If you upload multiple times per day, then you are using the nightly folder wrong. In the nightly should go a daily snapshot of your master branch.

If you want to retrieve CI-built ones, either use github artefacts or upload to tmp.kiwix.org/ci

[Jaifroid]

@rgaudin When I wrote "there could be several commits on one day" I didn't mean there could be several uploads to nightly. There is only one build and upload per night. What I meant was that the date alone does not inform the user at which point in the repository history the snapshot was taken

However, we can easily add the date in CI, it's not a problem.

[kelson42]

@rgaudin any chance that if there is a git revid between architecture and date, it can be handled/ignored?

[Jaifroid]

@rgaudin any chance that if there is a git revid between architecture and date, it can be handled/ignored?

To give an example, I've added (but not yet committed) a regex to CI that would produce this format:

kiwix-js-electron_x86-64_e7e37a0_2022-10-06.appimage

Is this OK, or do I need to remove the revision ID? Also, should it not be an underscore between the semantic parts of the filename, reserving hyphens for joining parts within the semantic unit? Practically, this means that there should be an underscore before the date rather than a hyphen (hyphens are within the date unit only). Is that correct?

[rgaudin]

If the pattern is different from the release, there is no point in renaming.

kiwix-js-electron_x86-64_1.9.4.appimage
kiwix-js-electron_x86-64_2022-10-06.appimage

I know that the date alone doesn't guarantee a commit correlation. That's not the point and it's the same for every project. Should there be a need to identify which commit produced the build, I think this can be found in the github actions listing (using time of upload).

FYI, what we do for kiwix-hotspot (which needs renaming as well because it doesn't include the date) is that the version metadata is updated during the build process to include the commit (if it's not a release). It's additional time to setup but really comfortable afterwards

[Jaifroid]

@rgaudin OK, so I'll remove the revision ID from the filename.

FYI we already add the rev ID to the version number displayed in the apps if it's a nightly build (on both Kiwix JS and KJSW), but the filename is a quick way of identifying if it's worth downloading a particular build (e.g. after a bugrfix commit). I agree that such a user (who will be technically adept) can find the info through other means.

[Jaifroid]

Done for KJSWL (please confirm it looks OK). Now I'll make a PR for KJS.

[rgaudin]

Great ; thank you

[Jaifroid]

Sorry, I didn't think that fixing it for a different repo would close this issue! In any case, it should now be fixed for Kiwix JS as well as Kiwix JS Windows, but we won't know for sure till tonight's nightly.

[rgaudin]

Done ; due to the variety of formats, I eventually chose to be more flexible with the /nightly redirects: it strips [_-\.]2022-12-05 for instance.
We still should migrate all files to the agreed-upon format.

[kelson42]

@rgaudin I just marged the PR to have compliant name for Android nightlies. Hopefuly now we can have a permalink for them as weel?

[rgaudin]

It's automatic and should already work

❯ curl -I http://download.kiwix.org/nightly/libkiwix_android-x86.tar.gz
HTTP/1.1 301 Moved Permanently
Date: Wed, 11 Jan 2023 18:09:10 GMT
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate
Location: http://download.kiwix.org/nightly/2023-01-11/libkiwix_android-x86-2023-01-11.tar.gz