Set up for testing git-annex on non-GitHub clients

[jwodder]

Closes #57.

To do:

• Sign & verify commits
• Make testannex.py prune local result-* branches that have been deleted remotely
• Handle "duplicate" build-* branches that occur when a build workflow is attempted again after the build-* branch has already been processed & deleted?
  • Not worth the effort
• Configure tinuous to collect build artifacts from the workflow in datalad/git-annex-ci-client-jobs
• Actually set up a client

[jwodder]

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

Also, a GitHub token for writing to the new repository needs to be added to this repository as a secret named CLIENT_JOBS_GITHUB_TOKEN.

[yarikoptic]

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

I setup that repo, but do we need one more repo? I thought we could just use branches in that singular one for badges too

Also, a GitHub token for writing to the new repository needs to be added to this repository as a secret named CLIENT_JOBS_GITHUB_TOKEN.

you are admin in both. Please create one and share (privately)?

[jwodder]

@yarikoptic

I setup that repo, but do we need one more repo? I thought we could just use branches in that singular one for badges too

I'm setting this up so that, when the script runs on a client, it pulls the latest configuration from this repository, which means it has to run in a clone of this repository. Thus, if the script also pulled build-* branches from the same repository, it'd be constantly switching between branches, and things would be needlessly complicated. The alternative would be to use a separate clone of datalad/git-annex for the build-* branches, but at that point it's just cleaner and barely makes a difference to use a separate repo.

[yarikoptic]

note: the same repo could be cloned/checked out into the same/different branches in an infinite number of locations ;-)

[yarikoptic]

note: I was arguing only against (somewhat) creating badges-and-result-* repository of some kind you mentioned in addition to git-annex-ci-client-jobs (already created). May be you just meant "branches", not "repository" in

@yarikoptic Please create a datalad/git-annex-ci-client-jobs repository and give me write permission so I can set up the badges-and-result-* repository.

?

[jwodder]

@yarikoptic I did mean "repository"; git-annex-ci-client-jobs is the build-*, result-*, and badges repository, and nothing else, and there isn't a need for yet another repository.

On further consideration (including the fact that GitHub tokens can't be scoped to a single repository), I may just use datalad/git-annex for all of the branches & badges and not bother with git-annex-ci-client-jobs.

[yarikoptic]

On further consideration (including the fact that GitHub tokens can't be scoped to a single repository), I may just use datalad/git-annex for all of the branches & badges and not bother with git-annex-ci-client-jobs.

my only concern so far against that would be that those build- branches would contain .deb thus making this repo then quite heavy and possibly slowing its own CI clones etc. But I guess it could be non-substantiated.

[jwodder]

@yarikoptic So should we use a separate repository or not?

• If we used separate repositories, then datalad/git-annex's access to the other repo would have to be granted by either (a) using a personal token for an account (I'm not using my account for that; is there a datalad-bot?) or (b) adding a repo-specific SSH public key to datalad/git-annex-ci-client-jobs and using the private key as a workflow secret.
• On the other hand, using a single repository would currently mean that the contents of the build-* branches would get cloned whenever a build was performed despite not being needed. While I can see a way around this, it would complicate the ability to specify an arbitrary commitish when triggering a build manually.

[yarikoptic]

@yarikoptic So should we use a separate repository or not?

if it is easier to go without a separate one with no impact on other operation of datalad/git-annex workflows (e.g. would main cron job get "cloning" of the repo very much slower to start with?) -- using one for everything is ok with me. Your choice.

• a) using a personal token for an account (I'm not using my account for that; is there a datalad-bot?)

there is https://github.com/datalad-tester we could "recover" if needed ;-)

or (b) adding a repo-specific SSH public key to datalad/git-annex-ci-client-jobs and using the private key as a workflow secret.

would be as fine if needed.

[yarikoptic]

@yarikoptic yarikoptic marked this pull request as ready for review 2 days ago

ha -- I don't remember doing that in my clear consciousnesses ;) It is still WiP, right @jwodder ?

[jwodder]

@yarikoptic Yes. I've decided to use separate repositories, and I've set up a deploy key for cloning. There's a test build running right now. After that, I still need to do the items listed in the initial comment.

[jwodder]

@yarikoptic Do you have a preferred GPG key to use for signing the build-* commits, or should I just create a fresh one? Also, exactly how should the clients react if a commit lacks a valid signature?

[jwodder]

@yarikoptic Also, I want to set up tinuous to collect logs & artifacts from datalad/git-annex-ci-client-jobs, and I want to set it up next to the tinuous setup for the datalad/git-annex logs, but I don't remember where that is. It doesn't seem to be under dandi@drogon's account.

[yarikoptic]

@yarikoptic Do you have a preferred GPG key to use for signing the build-* commits, or should I just create a fresh one?

We already have one in secrets of this repo - DATALAD_BUILDER_GPGKEY .

Also, exactly how should the clients react if a commit lacks a valid signature?

I guess Error out would be the safest? I do understand that it might lead to the flood of CRON error notifications since then we would not "react" to present build- commits, but I don't see any other "safe" way, right?

next to the tinuous setup for the datalad/git-annex logs, but I don't remember where that is.

that would be great! drogon contains only dandi related tinuous etc setups. DataLad related setups are all under datalad@smaug:/mnt/datasets/datalad/ci/

[jwodder]

@yarikoptic

I guess Error out would be the safest?

I meant, should it exit immediately or continue with other build-* branches? And should it delete the invalid local and/or remote branch?

[yarikoptic]

I guess we would iron out gory details after trying and seeing. I would rely on your analysis of desired logic, meanwhile

I meant, should it exit immediately or continue with other build-* branches?

I guess "other" in the order sorted by version?

And should it delete the invalid local and/or remote branch?

I guess without deletion we would go in an infinite loop... I feel that we eventually should introduce some way to "protocol" what was tried so we don't come back to it... may be we should push build-*-skipped ref of some kind with logs etc so we could keep skipping them and leave to admin to analyze and either to remove all those build-* branches or just skipped after fixing up what needs to be fixed ?

[jwodder]

@yarikoptic The client setup is working now, though I'm not going to actually create a cronjob until this branch is merged, as the code auto-updates from master. The only problem is that ndoli is very, very slow, and the tests keep hitting the one-hour timeout I set.

[yarikoptic]

The only problem is that ndoli is very, very slow, and the tests keep hitting the one-hour timeout I set.

eh, right -- on the discovery main node under a POSIXy /dartfs/rc/lab/D/DBIC/DBIC/archive/tmp it takes 5576.99s so aiming for two hours :-/ kinda wasteful, but can we boost timeout to two hours?

but in general, if PR is ready -- time to take out of draft mode?

[jwodder]

@yarikoptic Ready for review.

[yarikoptic]

FWIW: seeking possible ideas from @joeyh on speeding tests battery up https://git-annex.branchable.com/todo/speed_up___34__standalone_build__34___and__47__or_tests/?updated

[yarikoptic]

meanwhile, could you please setup client testing on smaug. It should not be as "exciting" as discovery, but at least would give us a sample client to test / report on

[jwodder]

@yarikoptic Should the smaug tests run the tests in any specific directory?

[jwodder]

@yarikoptic Also, what user account should I set this up under?

[yarikoptic]

probably as datalad user and somewhere e.g. like /mnt/datasets/datalad/git-annex-build-client?

[jwodder]

@yarikoptic Could you please merge this PR? Working with a different branch in a single-branch clone is very annoying.

[yarikoptic]

sure -- feel welcome to click Merge whenever you feel you want that state merged (I just see a fresh commit being tested).

[jwodder]

@yarikoptic Client set up on smaug.

[yarikoptic]

FWIW: seeking possible ideas from @joeyh on speeding tests battery up https://git-annex.branchable.com/todo/speed_up___34__standalone_build__34___and__47__or_tests/?updated

@joeyh fixed running standalone tests which might provide drastic boost! Let's see the timing on tomorrow build.