Add Nix-based development shell and related CI #517
Conversation
Turns out the image does have bash and openssh now, hooray. However, the checkout step is failing: > /usr/bin/docker exec a5e8... sh -c "cat /etc/*release | grep ^ID" > OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: no such file or directory: unknown I suppose because there is no `/etc/os-release`?
I have no idea if this is the way to do it
> invalid argument "kernel.unprivileged_userns_clone=1" for "--sysctl" flag: sysctl 'kernel.unprivileged_userns_clone=1' is not whitelisted
If we use a different container for each step then each step has to download and build the whole stack for itself, adding about 8 minutes to the job. :/
|
Ah this finally passed CI. Neat. It can probably be made a bit faster by setting up caching with https://cachix.org/. Usage is free up to 5GB of storage which should be plenty for GridSync (old stuff is automatically GC'd). GridSync should probably get a binary cache set up for it and an API token issued, and then the API token exposed to CI. And then ci.yaml needs a few more tweaks to use and push to the cache. cachix supports orgs so we could create a GridSync org to own those things. Let me know if that sounds good, @crwood, and if you'd like me to do the setup or if you want to (I can't do the part where we give GitHib Actions the API key though since I don't have the right permissions on gridsync/GridSync). This could also be follow-up work. The only other thing to do is turn the rest of the jobs back on in ci.yaml. |
The previous nixpkgs revision was an arbitrary revision of master that just happened to work during development. Instead of enshrining it, switch to something that's compatible with a version of pypi-deps-db we can get (and get that, too).
|
Other jobs now turned on. I think this is probably in good shape for review. |
Oops, nope. CI isn't actually running any of the tox environments. I'm not sure why yet. |
|
Thank you so much for this! Given my current lack of experience with nix (which maybe the landing of this PR will change), I don't have much to add here in the way of a meaningful review and I'm happy to merge if/whenever you feel like this is ready (which, it seems, is soon, pending CI?). I do have some questions about the dependency on python3.9 though: is there a reason this isn't 3.10 right now? What would be required to extend this to use other python versions (not now, but in the future)? Re. cachix: I'm happy to set that up too (and will take a closer look at it shortly) -- though given how painfully slow the other runners are in the matrix, we probably don't stand to gain too much from it right now... |
I picked 3.9 for now because I don't think Tahoe-LAFS or ZKAPAuthorizer has been tested against Python 3.10 yet. I tried those briefly a couple weeks ago and ran into some packaging trouble. While developing this branch, I discovered that I could be doing something much simpler with mach-nix to package both Tahoe-LAFS and ZKAPAuthorizer and perhaps that will address whatever that issue with Python 3.10 was. But I'll try to get those two projects onto Python 3.10 (and magic-folder?) before I work on it for GridSync. Of course, in the above, I am conflating the version of Python GridSync itself uses with the versions used by Tahoe-LAFS (+ ZKAPAuthorizer) and magic-folder. They don't necessarily have to be the same. The work to let them diverge is really just juggling the right version numbers so they land in the right places. Mostly we just need to pass "python310" in certain places where we're currently passing "python39". If we want to be fancy then we might want to arrange it so that we can have more than one version of Python here. Again that's mostly about juggling version strings but also involves one more step, where we invent names for the different configurations. For Tahoe-LAFS (in a branch) I've done something where you can do eg, |
Prior to this change the steps still don't actually run the tox command.
It's not clear the CI problems have anything to do with this, but might as well try.
|
Okay! If this CI run succeeds then I think it's ready for merge. |
And it looks good! Thank you, again, for all of your work on this! With Gridsync dev/testing on NixOS now made easier, I can feel the Nix/NixOS rabbit hole drawing me ever closer... ;) |
This makes
nix developwork with a recent-enough version of Nix (with flakes enabled). The resulting environment contains enough stuff fortox -e py39,integration,lintto succeed.Also included is a new GitHub Actions job to do these things on CI, mostly to ensure they keep working (and so remain useful for development work on GridSync on NixOS or with Nix tools on other OS).