Make the ./publish script reusable instead of "vendoring" it everywhere

[generalmimon]

Right now, we have the ./publish shell script vendored (i.e. copied) in every repo that needs to publish build results to some other repository (on behalf of @kaitai-bot).

In fact, it seems that there are three versions of the script - I call them version 1, 1.1 and 2.

Version 1.1 has one extra feature above 1:

• it proactively adds the GitHub RSA public key (hardcoded in the script) to ~/.ssh/known_hosts if it's not there

Version 2 has two extra features above 1.1:

• supports specifying non-default Git branch via GH_REPO_BRANCH bash variable (used only in ci_targets repo though; in kaitai_ci_ui repo, there is just boring GH_REPO_BRANCH=master)
• if rsync is not found on PATH, it tries /c/cygwin64/bin/rsync.exe

Version 1:

• kaitai_struct_tests / publish-targets
• kaitai_struct_webide / publish
• kaitai_struct_doc / publish

Version 1.1:

• kaitai_struct_formats / .github/publish

Version 2:

• kaitai_ci_ui / publish
• ci_targets / publish

I think that it totally makes sense to have a publish script in only one place and use it from all these repos.

Of course, we would need to make it accept parameters - I think CLI args (./publish --outdir _build/html/ --repo formats-kaitai-io.github.io) are the way to go. Environment vars would be easier to implement, but less transparent, they can be set on various places.

I'm not sure where should be this generic publish script placed though - https://github.com/kaitai-io/kaitai_struct seems to be a logical place, but I'm thinking about adding the repo with the script as a Git submodule to all the repos (which is IMHO better than git clone-ing the repo, curling the file from raw.githubusercontent.com or other perversities), and it wouldn't make much sense to add kaitai_struct as a submodule to repo kaitai_struct_tests for example (it's supposed to be the other way around).

So maybe create a dedicated repository just for the publish script?

[GreyCat]

It would be even cooler if we could make a GitHub Action-compatible action. We can reuse existing shell script, wrapping it as Docker action or potentially rewrite it at all using JavaScript action.

[KOLANICH]

wrapping it as Docker action

I am not sure that GH docker-based actions are as efficient as GitLab CI pipelines. I mean GH still loads an own Docker container with Ubuntu and a lot of build-related stuff. So loading an our own container above it is just a waste of time and a bandwidth and other resources. Waste of bandwidth and resources is not our problem, if MS likes it, it is their choice, but waste of time is clearly not what we want.

I wonder if we can cache our large dependencies anyhow (there is an official action for that, but it is Node-powered, I prefer to stay away of node.js ecosystem, BTW recently some backdoors have been discovered in npm again) and then rely on the stuff already preinstalled and updated by MS.

[generalmimon]

Related: I've just deleted legacy script update-ci-status for the old CI (0068fe6).

[generalmimon]

I've created the repo https://github.com/kaitai-io/push-artifacts for the GitHub Action. I plan to run the action in a Docker container.

[generalmimon]

Hmm, I've just realized that it doesn't make much sense to package the script as a GitHub Action only, because we actually run most of the publish scripts in Travis CI, not GH Actions:

repo / publish script	CI config invoking the script	CI running the script
kaitai_struct_tests / publish-targets	kaitai_struct / .travis.yml	Travis CI
kaitai_struct_webide / publish	.travis.yml	Travis CI
kaitai_struct_doc / publish	.travis.yml	Travis CI
kaitai_struct_formats / .github/publish	.github/workflows/main.yml	GitHub Actions
kaitai_ci_ui / publish	.travis.yml	Travis CI
ci_targets / publish	.travis.yml, .appveyor.yml, .github/workflows/main.yml	Travis CI, Appveyor, GitHub Actions

[generalmimon]

A checklist of the transition from publish script with hardcoded parameters to a parameterized script push_artifacts / publish:

• kaitai_struct_tests
• kaitai_struct_webide
• kaitai_struct_doc
• kaitai_struct_formats
• kaitai_ci_ui
• ci_targets

[GreyCat]

I wonder if we can actually establish some kind of test bed for this script itself? In theory, it should be relatively straightforward: just publish a commit to some kind of scrap repo using Travis / AppVeyor / GH Actions for every PR to this script and/or every commit to master?

[GreyCat]

And, yeah, great work with extracting it, by the way! Looks really cool!

[GreyCat]

Hmm, I've just realized that it doesn't make much sense to package the script as a GitHub Action only, because we actually run most of the publish scripts in Travis CI, not GH Actions

May be in some point in time it will make sense to fully migrate off Travis, actually, especially due to the shutdown of travis-ci.org pending in ~2 months, so we'll need to migrate to either travis-ci.com or somewhere else.

[KOLANICH]

fully migrate off Travis, actually, especially due to the shutdown of travis-ci.org pending in ~2 months

They are not closing Travis, they are forcing free users to go to .com domain, where they claim they now provide free services too. But it is a bad sign that a company cannot afford to pay for a yet another domain name.

IDK any good free CI, GitLab was the best, but since they continue to do bad things, like continue eliminating server-side HTML generatikn in favour of client-side JS, using Cloudflare and enabling their "Under attack" mode, that injects obfuscated JavaScript code, I cannot advice them anymore. SourceHut looks promising (also FOSS), but not the official service (I cannot say anything special about that, since I have personally never used that, but there is some info in the Net and some personal experience of chatting with Drew on IRC channel, where he told me that he decides if the feature itself is needed only after I implemented it, the nuance is that he gives free accounts to srchut to people who have contributed to its development by code).

There also some standalone gitlab instances, like gnome, kde and freedesktop, but KS is not a part of their ecosystem.

There is also "gitgud", but I personally don't trust them.

[generalmimon]

Closing as resolved, see #819 (comment).