[READY] - add ansible-lint image #49

kylerisse · 2022-03-20T05:19:30Z

Description of PR

Adds a new image for ansible-lint with some typical complementary tools such as yamllint, pytest, pylint, jq, git, etc. Immediate use case is for github actions in the CI workflow for SCaLE network. This method seems like a much better alternative to the "standard" ansible lint github action so others might find it useful. There's also an opportunity to add this to some existing internal projects.

Tests

local build and test

$ nix-build -A imgs.ansible-lint
/nix/store/b3451xr827aasi99y97j52pnxl6gji5k-docker-image-ansible-lint.tar.gz

$ docker load < /nix/store/b3451xr827aasi99y97j52pnxl6gji5k-docker-image-ansible-lint.tar.gz
Loaded image: nebulaworks/ansible-lint:latest

$ docker run -ti nebulaworks/ansible-lint:latest /bin/bash
bash-5.1# git clone https://github.com/socallinuxexpo/scale-network
Cloning into 'scale-network'...
remote: Enumerating objects: 4552, done.
remote: Counting objects: 100% (585/585), done.
remote: Compressing objects: 100% (346/346), done.
remote: Total 4552 (delta 285), reused 444 (delta 204), pack-reused 3967
Receiving objects: 100% (4552/4552), 861.29 KiB | 4.46 MiB/s, done.
Resolving deltas: 100% (2435/2435), done.
bash-5.1# cd scale-network/facts
bash-5.1# pylint *.py
************* Module datasource
datasource.py:124:10: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
datasource.py:124:10: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)

-----------------------------------
Your code has been rated at 9.78/10

bash-5.1# pytest -vv
========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /nix/store/ib89gapvyzncfyjpcwgwikz29jm1iy16-python3-3.10.2-env/bin/python3.10
cachedir: .pytest_cache
rootdir: /scale-network/facts
plugins: flaky-3.7.0
collected 5 items                                                                                                                                                                                                                        

test_datasources.py::test_aplist_csv PASSED                                                                                                                                                                                        [ 20%]
test_datasources.py::test_pilist_csv PASSED                                                                                                                                                                                        [ 40%]
test_datasources.py::test_routerlist_csv PASSED                                                                                                                                                                                    [ 60%]
test_datasources.py::test_serverlist_csv PASSED                                                                                                                                                                                    [ 80%]
test_datasources.py::test_switchtypes_tsv PASSED                                                                                                                                                                                   [100%]

=========================================================================================================== 5 passed in 0.05s ============================================================================================================
bash-5.1# cd ../ansible
bash-5.1# ansible-lint -x 403 -x 503 scale.yml
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: scale.yml
WARNING  Listing 54 violation(s) that are fatal
yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/handlers/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/handlers/main.yml:8

yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/tasks/main.yml:23

yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/tasks/main.yml:24

yaml: truthy value should be one of [false, true] (truthy)
roles/chrony/tasks/main.yml:25

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:8

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:14

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:15

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:16

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:22

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:23

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/handlers/main.yml:24

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:117

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:118

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:119

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:127

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:128

yaml: truthy value should be one of [false, true] (truthy)
roles/dhcpserver/tasks/main.yml:129

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/handlers/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/handlers/main.yml:8

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/handlers/main.yml:9

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/tasks/main.yml:16

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/tasks/main.yml:18

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsclient/tasks/main.yml:19

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/handlers/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/handlers/main.yml:8

yaml: no new line character at the end of file (new-line-at-end-of-file)
roles/dnsserver/handlers/main.yml:9

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/handlers/main.yml:9

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/tasks/main.yml:152

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/tasks/main.yml:153

yaml: truthy value should be one of [false, true] (truthy)
roles/dnsserver/tasks/main.yml:154

yaml: truthy value should be one of [false, true] (truthy)
roles/noapparmor/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/noapparmor/tasks/main.yml:8

yaml: truthy value should be one of [false, true] (truthy)
roles/noapparmor/tasks/main.yml:9

yaml: truthy value should be one of [false, true] (truthy)
roles/noapparmor/tasks/main.yml:16

yaml: truthy value should be one of [false, true] (truthy)
roles/signs/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/signs/tasks/main.yml:26

yaml: truthy value should be one of [false, true] (truthy)
roles/signs/tasks/main.yml:64

yaml: truthy value should be one of [false, true] (truthy)
roles/signs/tasks/main.yml:65

yaml: truthy value should be one of [false, true] (truthy)
roles/techteam/tasks/users.yml:16

yaml: truthy value should be one of [false, true] (truthy)
roles/xymon/handlers/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/xymon/handlers/main.yml:8

yaml: truthy value should be one of [false, true] (truthy)
roles/xymon/handlers/main.yml:14

yaml: truthy value should be one of [false, true] (truthy)
roles/xymon/handlers/main.yml:15

yaml: too many spaces inside brackets (brackets)
roles/xymon/tasks/main.yml:5

yaml: truthy value should be one of [false, true] (truthy)
roles/zabbixagent/tasks/main.yml:7

yaml: truthy value should be one of [false, true] (truthy)
roles/zabbixagent/tasks/main.yml:21

yaml: truthy value should be one of [false, true] (truthy)
scale.yml:4

yaml: truthy value should be one of [false, true] (truthy)
scale.yml:12

yaml: truthy value should be one of [false, true] (truthy)
scale.yml:20

You can skip specific rules or tags by adding them to your configuration file:
# .ansible-lint
warn_list:  # or 'skip_list' to silence them completely
  - yaml  # Violations reported by yamllint

Finished with 54 failure(s), 0 warning(s) on 20 files.
bash-5.1# pylint *.py
************* Module inventory
inventory.py:17:14: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
inventory.py:17:14: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
inventory.py:210:14: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
inventory.py:210:14: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
inventory.py:431:40: C0201: Consider iterating the dictionary directly instead of calling .keys() (consider-iterating-dictionary)

-----------------------------------
Your code has been rated at 9.84/10

bash-5.1# pytest -vv
========================================================================================================== test session starts ===========================================================================================================
platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /nix/store/ib89gapvyzncfyjpcwgwikz29jm1iy16-python3-3.10.2-env/bin/python3.10
cachedir: .pytest_cache
rootdir: /scale-network/ansible
plugins: flaky-3.7.0
collected 18 items                                                                                                                                                                                                                       

test_inventory.py::test_getfilelineshdr PASSED                                                                                                                                                                                     [  5%]
test_inventory.py::test_getfilelinesnobldg PASSED                                                                                                                                                                                  [ 11%]
test_inventory.py::test_getfilelinesbldg PASSED                                                                                                                                                                                    [ 16%]
test_inventory.py::test_dhcp6ranges PASSED                                                                                                                                                                                         [ 22%]
test_inventory.py::test_dhcp4ranges PASSED                                                                                                                                                                                         [ 27%]
test_inventory.py::test_makevlan PASSED                                                                                                                                                                                            [ 33%]
test_inventory.py::test_bitmasktonetmask PASSED                                                                                                                                                                                    [ 38%]
test_inventory.py::test_genvlans PASSED                                                                                                                                                                                            [ 44%]
test_inventory.py::test_ip4toptr PASSED                                                                                                                                                                                            [ 50%]
test_inventory.py::test_ip6toptr PASSED                                                                                                                                                                                            [ 55%]
test_inventory.py::test_isvalidip PASSED                                                                                                                                                                                           [ 61%]
test_inventory.py::test_roomalias PASSED                                                                                                                                                                                           [ 66%]
test_inventory.py::test_populatevlans PASSED                                                                                                                                                                                       [ 72%]
test_inventory.py::test_populateswitches PASSED                                                                                                                                                                                    [ 77%]
test_inventory.py::test_populaterouters PASSED                                                                                                                                                                                     [ 83%]
test_inventory.py::test_populateaps PASSED                                                                                                                                                                                         [ 88%]
test_inventory.py::test_populatepis PASSED                                                                                                                                                                                         [ 94%]
test_inventory.py::test_populateservers PASSED                                                                                                                                                                                     [100%]

=========================================================================================================== 18 passed in 0.15s ===========================================================================================================
bash-5.1# ./inventory.py | jq .
{
  "routers": {
    "hosts": [
      "br-mdf-01",
      "ex-mdf-01",
      "cf-mdf-01"
    ],
    "vars": {}
  },
  "servers": {
    "hosts": [
      "core1",
      "core2",
      "monitoring1",
      "automation1",
      "pkgcache"
    ],
    "vars": {}
  },
  "switches": {
    "hosts": [
      "Rm101-102",
      "NW-IDF",
(........... further output omitted .................)

sarcasticadmin

Thanks for raising this PR @kylerisse!

This method seems like a much better alternative to the "standard" ansible lint github action so others might find it useful. There's also an opportunity to add this to some existing internal projects.

I agree, I think theres a lot of need for this given the situations we find ourselves in with ansible

I wasn't 100% sure how to test this, so I created a nix shell with the same code and ran it with --pure. I then used the shell to validate against some existing repos. I also manually ran through the lint workflow using docker. I would love some further guidance on the best way to validate this before marking READY.

Ah ya the docs really dont call this out 😞 Sounds like the majority of your testing when well though and seems to be valid checks as well. Regarding the overlay if you wanted to test your changes more directly you can also:

cd <repo root>
nix-build -A imgs.ansible-lint

The build should produce a result that can be verified and matched the CI build

The result you build locally can also then be tested into docker with something like:

docker load < result

Itll be named nebulaworks/ansible-lint so then you should be able to just:

docker docker run -t nebulaworks/ansible-lint:latest

That should more or less work for you locally. Let me know how that does when you get a chance to try it out. We should probably update our docs with this basic sanity check. Its also called out in a good ref here: https://nix.dev/tutorials/building-and-running-docker-images

imgs/default.nix

kylerisse · 2022-03-22T02:24:05Z

That should more or less work for you locally. Let me know how that does when you get a chance to try it out. We should probably update our docs with this basic sanity check. Its also called out in a good ref here: https://nix.dev/tutorials/building-and-running-docker-images

Thanks @sarcasticadmin, it all worked as described. I went ahead and updated the README with these steps as well.

sarcasticadmin

@kylerisse considering with how much we've leveraged ansible across our engagements to date, this seems like a fine addition!

sarcasticadmin · 2022-04-14T01:31:17Z

imgs/README.md

@@ -5,3 +5,11 @@ so that the output of the image is `ALWAYS` reproducible. This has been a long s
 of docker images and `Dockerfile`.

 All images are currently hosted in on Dockerhub: https://hub.docker.com/u/nebulaworks
+
+## Local Testing


Thanks for adding this note!

sarcasticadmin · 2022-04-14T01:31:41Z

imgs/ansible-lint/default.nix

+  nwi = import ../../nwi.nix;
+  lib = pkgs.lib;
+  customPython = with pkgs; python310.withPackages
+    (pythonPackages: with pythonPackages; [ ansible-lint pylint pytest yamllint ]);


Nice, looks like we only have the one interpreter now :)

sarcasticadmin · 2022-04-14T01:31:56Z

imgs/ansible-lint/default.nix

+    # make sure /tmp exists
+    mkdir -m 1777 tmp
+    mkdir -p usr/bin
+    ln -s ${pkgs.coreutils}/bin/env usr/bin/env


this is the way

sarcasticadmin · 2022-04-14T01:32:15Z

imgs/ansible-lint/README.md

@@ -0,0 +1,5 @@
+# ansible-lint


Thanks for adding a description

sarcasticadmin · 2022-04-14T01:34:14Z

@kylerisse go ahead and test out the comment build. We still have to fix that docker v1 vs v2 api oddity (at a later date)

kylerisse · 2022-04-14T01:49:09Z

publish

kylerisse · 2022-04-16T01:31:29Z

publish

sarcasticadmin · 2022-04-16T01:41:10Z

publish

sarcasticadmin · 2022-04-16T01:57:18Z

publish

sarcasticadmin · 2022-04-16T02:00:34Z

Ok figured out the initial error since you were on a fork and raise the PR from there apparently v1 doesnt deal with this well 🤦 actions/checkout#318

So i pushed up your branch to the repo directly but it looks like cachix needs to help too: https://github.com/Nebulaworks/nix-garage/runs/6044782853?check_suite_focus=true#step:5:6 will look into this a little later but were close!

kylerisse · 2022-04-25T22:59:30Z

publish

kylerisse · 2022-04-25T23:00:08Z

(also force pushed the rebased branch to this repo's branch of the same name to avoid the actions v1 checkout forks issue)

kylerisse · 2022-04-26T00:21:51Z

Failed https://github.com/Nebulaworks/nix-garage/runs/6166702535?check_suite_focus=true
I'm able to recreate the failure locally in docker using multiple versions of nixos/nix, including the one that maps to latest, switching back to WIP until we figure out what the next fix is.

sarcasticadmin · 2022-04-26T02:23:33Z

Failed https://github.com/Nebulaworks/nix-garage/runs/6166702535?check_suite_focus=true
I'm able to recreate the failure locally in docker using multiple versions of nixos/nix, including the one that maps to latest, switching back to WIP until we figure out what the next fix is.

@kylerisse Thanks for testing this. Im glad the cachix stuff is no longer the issue and this this is reproducible. The issue seems to be with the revision of the nix-channel thats defaulted to in the container. Its honestly hard to tell the revision/version but its just whatever it is when the container was built (in our case broken for gcc). And we dont do anything to specify that at during the build. See the following from inside the container:

sh-4.4# nix-channel --list                                                                                                                                                                                          
nixpkgs https://nixos.org/channels/nixpkgs-unstable
sh-4.4# nix-shell -p nix-info --run "nix-info -m"
...
copying path '/nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux' from 'https://cache.nixos.org'...                                                                                                           
 - system: `"x86_64-linux"`                                                                                                                                                                                         
 - host os: `Linux 5.11.0-27-generic`                                                                                                                                                                               
 - multi-user?: `yes`                                                                                                                                                                                               
 - sandbox: `no`                                                                                          
 - version: `nix-env (Nix) 2.7.0`                                                                                                                                                                                   
 - channels(root): `""`                                                                                   
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

We have a couple of options as I see it:

We can leverage the stable version defined in https://github.com/Nebulaworks/nix-garage/blob/master/release.nix for all publish calls
- Pros: We get a stable consistent version so we can pinpoint where build errors might crop up. Its also known working for any existing imgs/builds weve done historically for the repo.
- Cons: Must be specified locally when running things like nix-build, etc.
We pin https://github.com/Nebulaworks/nix-garage/blob/master/default.nix to a stable pin (probably at or ahead of release.nix):
- Pros: nix-build will use a default version thats consistent.
- Cons: Rework CI https://github.com/Nebulaworks/nix-garage/blob/master/.github/workflows/daily.yml since Im not sure the precedence for environment vars vs args (might not be an issue)
Pin only the CI bits with something similar to release.nix but its instead ci.nix
- Pros: nix-build remains unpinned for local testing and daily workflow. We dont get in a scenario where release.nix is broken for new imgs or pkgs and have to update it prematurely.
- Cons: Extra file and we would have to document how to ensure consistent behavior from publish command vs local nix-build

As Im writing this I feel like option 3: ci.nix seems like the best option so we get a consistent behavior but leave the default.nix unpinning so you can consume it against any version of nixpkgs you happen to be at.

sarcasticadmin · 2022-04-26T02:24:23Z

@kylerisse sorry didnt mean to close

kylerisse · 2022-04-26T20:51:27Z

Failed https://github.com/Nebulaworks/nix-garage/runs/6166702535?check_suite_focus=true
I'm able to recreate the failure locally in docker using multiple versions of nixos/nix, including the one that maps to latest, switching back to WIP until we figure out what the next fix is.

@kylerisse Thanks for testing this. Im glad the cachix stuff is no longer the issue and this this is reproducible. The issue seems to be with the revision of the nix-channel thats defaulted to in the container. Its honestly hard to tell the revision/version but its just whatever it is when the container was built (in our case broken for gcc). And we dont do anything to specify that at during the build. See the following from inside the container:

sh-4.4# nix-channel --list                                                                                                                                                                                          
nixpkgs https://nixos.org/channels/nixpkgs-unstable
sh-4.4# nix-shell -p nix-info --run "nix-info -m"
...
copying path '/nix/store/3k69hbxg04sdxlgi1236ddggs346sxf3-stdenv-linux' from 'https://cache.nixos.org'...                                                                                                           
 - system: `"x86_64-linux"`                                                                                                                                                                                         
 - host os: `Linux 5.11.0-27-generic`                                                                                                                                                                               
 - multi-user?: `yes`                                                                                                                                                                                               
 - sandbox: `no`                                                                                          
 - version: `nix-env (Nix) 2.7.0`                                                                                                                                                                                   
 - channels(root): `""`                                                                                   
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

We have a couple of options as I see it:

1. We can leverage the stable version defined in https://github.com/Nebulaworks/nix-garage/blob/master/release.nix for all `publish` calls
   - Pros: We get a stable consistent version so we can pinpoint where build errors might crop up. Its also known working for any existing imgs/builds weve done historically for the repo.
   - Cons: Must be specified locally when running things like `nix-build`, etc.

2. We pin https://github.com/Nebulaworks/nix-garage/blob/master/default.nix to a stable pin (probably at or ahead of release.nix):
   - Pros: `nix-build` will use a default version thats consistent.
   - Cons: Rework CI https://github.com/Nebulaworks/nix-garage/blob/master/.github/workflows/daily.yml since Im not sure the precedence for environment vars vs args (might not be an issue)

3. Pin only the CI bits with something similar to `release.nix` but its instead `ci.nix`
   - Pros: `nix-build` remains unpinned for local testing and daily workflow. We dont get in a scenario where `release.nix` is broken for new imgs or pkgs and have to update it prematurely.
   - Cons: Extra file and we would have to document how to ensure consistent behavior from `publish` command vs local nix-build

As Im writing this I feel like option 3: ci.nix seems like the best option so we get a consistent behavior but leave the default.nix unpinning so you can consume it against any version of nixpkgs you happen to be at.

We could probably mitigate the cons of either option 1 or 3 by wrapping usage in a Makefile. Thanks for writing all that out. It's very informative. I can't predict which is the best and will lean on your experience for best path forward.

sarcasticadmin · 2022-04-28T03:32:38Z

👀

kylerisse · 2022-04-28T03:33:48Z

publish

nwi-auto · 2022-04-28T03:37:35Z

The following docker images:
Nothing to do
have been published on Docker Hub.

kylerisse · 2022-04-28T03:48:35Z

publish

nwi-auto · 2022-04-28T03:50:49Z

The following docker images:
Nothing to do
have been published on Docker Hub.

kylerisse · 2022-04-28T04:58:00Z

2022-04-28T03:50:24.0778023Z ++ nix-shell --run './publish-imgs -f ci.nix'
2022-04-28T03:50:25.7206886Z error: path '/nix/store/p6v5fpv4dnvzwwfqfb70s8hnjy59jag6-source.drv' is not valid
2022-04-28T03:50:25.7207747Z (use '--show-trace' to show detailed location information)
2022-04-28T03:50:25.7239919Z parse error: Unmatched '}' at line 1, column 17

kylerisse · 2022-04-28T05:17:27Z

This is a weird one. So it seems to bomb out on

nix-garage/publish-imgs

Line 58 in b80ecae

    
           for entry in $(nix-instantiate --eval --strict --json -A $attr $nixtop | jq -r 'keys[]' ); do

$ nix-instantiate --eval --strict --json -A imgs ci.nix --show-trace
error: path '/nix/store/91s8j4lfksvzb4jgwacyjlay1i5f53a9-source.drv' is not valid

       … while realising the context of path '/nix/store/l11fy3pap0lhzdw6iwg0q3d5356lqly3-source'

       at /home/kylerisse/go/src/github.com/kylerisse/nix-garage/pin/default.nix:12:1:

           11| in
           12| import src { }
             | ^

       … while evaluating anonymous lambda

       at /home/kylerisse/go/src/github.com/kylerisse/nix-garage/pin/default.nix:1:1:

            1| { bootstrap ? import <nixpkgs> { }
             | ^
            2| , snapshot

       … from call site

       at /home/kylerisse/go/src/github.com/kylerisse/nix-garage/ci.nix:6:13:

            5| let
            6|   nixpkgs = import ./pin { snapshot = "master_1"; };
             |             ^
            7|   default = import ./default.nix { inherit nixpkgs; };

       … while realising the context of a path

       at /home/kylerisse/go/src/github.com/kylerisse/nix-garage/default.nix:3:10:

            2| let
            3|   pkgs = import nixpkgs.path { overlays = [ (import ./overlay.nix) ]; };
             |          ^
            4| in
{"ansible-lint":}

but only from a clean state. I was able to get the derivation it's complaining about to build by removing --eval like so

nix-instantiate --strict --json -A imgs ci.nix --show-trace
building '/nix/store/91s8j4lfksvzb4jgwacyjlay1i5f53a9-source.drv'...

trying https://github.com/NixOS/nixpkgs/archive/df13cab16c5ead65d69e00ae6187b5f3c8a840cb.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   155  100   155    0     0   1183      0 --:--:-- --:--:-- --:--:--  1192
100 28.9M    0 28.9M    0     0  6133k      0 --:--:--  0:00:04 --:--:-- 7179k
unpacking source archive /build/df13cab16c5ead65d69e00ae6187b5f3c8a840cb.tar.gz
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/53a5miwhq3lhf8443ixn3a7nxamy3ma8-docker-image-ansible-lint.tar.gz.drv
... omitted rest ...

Then the original command works

nix-shell --run './publish-imgs -s -f ci.nix'

and then garbage collect with nix-collect-garbage and it's then broken again

$ nix-shell --run './publish-imgs -s -f ci.nix'error: path '/nix/store/91s8j4lfksvzb4jgwacyjlay1i5f53a9-source.drv' is not valid
(use '--show-trace' to show detailed location information)
parse error: Unmatched '}' at line 1, column 17
Nothing to do

which is probably why we didn't catch it in the other PR, since prior to testing publish-img -f ci.nix we had already built /nix/store/91s8j4lfksvzb4jgwacyjlay1i5f53a9-source.drv. Also I believe the reason this doesn't happen in other steps is all usage of nix-shell directly inline in https://github.com/Nebulaworks/nix-garage/blob/master/.github/workflows/publish_imgs.yml will leverage default.nix instead of ci.nix.

I'm not sure what the best way to "pre-build" the derivation, but I'm pretty sure this is the condition in the clean container. publish-img just fails that steps and keeps going so the bot reports that there's nothing to build. We're closer though!

kylerisse · 2022-04-28T05:36:00Z

here's a simplified way to reproduce the base condition

[nix-shell:~/go/src/github.com/kylerisse/nix-garage]$ nix-instantiate --eval --strict --json -A imgs ci.nix
{"ansible-lint":"/nix/store/cx11fsg47srrbpg41x1kx9zd1ymyqmgr-docker-image-ansible-lint.tar.gz","awsutils":"/nix/store/062y3vxhs7hap0v279gy1wrcnmmnngy4-docker-image-awsutils.tar.gz","flasksample":"/nix/store/9m2k44fp1v7nvijz8mz7nkac10fs8wgv-docker-image-flasksample.tar.gz","helmsman-aws":"/nix/store/fh5ba96qrhkkgilk03srysz5qm7kg0j8-docker-image-helmsman-aws.tar.gz","magic-wormhole-mailbox":"/nix/store/46f5vfsc69l1zmzw2f5z02isjnp7xfsd-docker-image-magic-wormhole-mailbox.tar.gz","nwi-pr-utils":"/nix/store/c8a0x9ddsnb05nv16j8sj8ha4n3giq5h-docker-image-nwi-pr-utils.tar.gz","pki-validator":"/nix/store/fjqp40jq1fvzl1br6mhhr171q27xzm46-docker-image-pki-validator.tar.gz","term-apply":"/nix/store/srm4d2spzfzljc02dvyjbw8db47g0m9v-docker-image-term-apply.tar.gz"}
[nix-shell:~/go/src/github.com/kylerisse/nix-garage]$ nix-collect-garbage &> /dev/null

[nix-shell:~/go/src/github.com/kylerisse/nix-garage]$ nix-instantiate --eval --strict --json -A imgs ci.nix
error: path '/nix/store/91s8j4lfksvzb4jgwacyjlay1i5f53a9-source.drv' is not valid
(use '--show-trace' to show detailed location information)
{"ansible-lint":}
[nix-shell:~/go/src/github.com/kylerisse/nix-garage]$ nix-instantiate --strict --json -A imgs ci.nix &> /dev/null

[nix-shell:~/go/src/github.com/kylerisse/nix-garage]$ nix-instantiate --eval --strict --json -A imgs ci.nix
{"ansible-lint":"/nix/store/cx11fsg47srrbpg41x1kx9zd1ymyqmgr-docker-image-ansible-lint.tar.gz","awsutils":"/nix/store/062y3vxhs7hap0v279gy1wrcnmmnngy4-docker-image-awsutils.tar.gz","flasksample":"/nix/store/9m2k44fp1v7nvijz8mz7nkac10fs8wgv-docker-image-flasksample.tar.gz","helmsman-aws":"/nix/store/fh5ba96qrhkkgilk03srysz5qm7kg0j8-docker-image-helmsman-aws.tar.gz","magic-wormhole-mailbox":"/nix/store/46f5vfsc69l1zmzw2f5z02isjnp7xfsd-docker-image-magic-wormhole-mailbox.tar.gz","nwi-pr-utils":"/nix/store/c8a0x9ddsnb05nv16j8sj8ha4n3giq5h-docker-image-nwi-pr-utils.tar.gz","pki-validator":"/nix/store/fjqp40jq1fvzl1br6mhhr171q27xzm46-docker-image-pki-validator.tar.gz","term-apply":"/nix/store/srm4d2spzfzljc02dvyjbw8db47g0m9v-docker-image-term-apply.tar.gz"}

kylerisse · 2022-04-28T05:41:37Z

publish

nwi-auto · 2022-04-28T05:46:57Z

The following docker images:
/nix/store/cx11fsg47srrbpg41x1kx9zd1ymyqmgr-docker-image-ansible-lint.tar.gz Getting image source signatures Copying blob sha256:15664a6033e943515a29c229a8a1a3062cff21b678fb363bc0f13450de152fa6 Copying config sha256:f1eb0960827ad41e06fd5930cc7a6beff1f4d99d786c2c7059457ed6fda46177 Writing manifest to image destination Storing signatures { "Name": "docker.io/nebulaworks/awsutils", "Digest": "sha256:0589df2c6c5593f088528a7c9dbb0921593684b6c3688c3af1c6f08a97d5fcd1", "RepoTags": [ "062y3vxhs7hap0v279gy1wrcnmmnngy4", "0afjz2x6jzlqbyp6hlsn7xx1j4v595xj", "0q6n72i9f6w35qg70a8fddnh57jr25yb", "0wrw13hq33i9lvliwgrilimblm1by6ga", "1hx4387bjv85cs424znxr9fpbdbm8djk", "43pax9ygdpviwka1lvqkvsxk3qlkg32i", "4iyc3wm1kjz9lwqwsgvjz8g78hdgxc7p", "4yzrc2fjyajqaihgn3jycfjjsrwrdb7z", "5cnip2b81k71qbap8493mmvfp02fk6wh", "5dq3sxqy6ql24adppr4abllkpq51ai25", "7ydhw282rks4v1pv3si8iv0cvaamypw5", "9jfmk9gq677qhks1nagk6kagr21b80zh", "9mz9x2m35npczsjgxznwiwlzj85ij9hq", "a2x2gakpv9acj1zp6q4cfk0x34kakh84", "a5d61h3nvmigz60bins2pgwsq7hx63h1", "a5kccnv7dvfjg6p5z0kvv8f4gi8i6cln", "axjz9vl56z0lm8jwzlri7cb80zpiiv3j", "b2brn67ymrvbi8izvxvayi4bn4adwa27", "b2xpdz7r5fdlnyyalsxlzncwlvy1gh6m", "br0g08h90amarmcsl8ic7iji74wfzwhk", "c3skllz0pvp6a78nf3c4fck7xcg1yxq5", "c4vyb1p0lch8czxncl0nq685wick8fnd", "dz783iaashdby6phqf9pcydxsw1bbjq0", "f9lc1bvrwz8wbjhccp6dyg2yvzy56npg", "fd6mchdar3ja13ph0jp1s8pnw7ixfahk", "g44npkrgrh2qj2y9j1a9n6frcyd2j45z", "gv99dxfp8hy5w9dl16p9wc7km3mg2n1f", "h952m39vshzpihy6cgmp17gjszqq3vks", "hbqmiwillbddpzdjdl0v8y5h1xxy1bwx", "hc1r0zhhmwdvz0ygyyypq0kav5yw6zfn", "hx1a61jwx4r2crri7iph7ygd19gbqab5", "ib0v6zkv2qqaidwda7sp64pa0jfc0vbs", "k2c75rki35d8z8giwynras8nwrsr1l0z", "k3i9bp4w5ndn7ai6cylbimxybqln4nrc", "kbc4fdqiaiwgbiwxf7gbd123gf01ph2z", "kl55657f4yfpi6b8kj4f1vwgk3jdhvza", "l4vf8g6g8wfahhhc7c1m90sm850s2sf2", "mzqlmhc5gl3h3k10bn1l0ghr204n0pqv", "n2m55acawpn7y06h6ifjbyzmn9n44bhy", "nzbazzwdq9vy9rg10jcp2dvc3fs3n1bp", "pbjgfkmi0wqiwd1flzwm7rz8snd50j36", "rss9pb0cmnbhl49in406s3i7hiy466ps", "sg814n929qffnj1xd0637knpgaffzzpf", "sjr0mavplaws1dlxy3vr2880v0jkhh3x", "sy7z5ylf9j3jrbhgd8kcpnnndbch3v4j", "v7dw7988kyhzsgjzxaiybhk0a9wsbm2g", "vbp6wk6y3gn143g5sfsfgpih8shr2sml", "w4cwy90aa2b1rrf9nqd5qjcflrjh0129", "xaacyvsd2q0z4yz111zrj8y0za422gh9", "yfws0rvz76xkiq0yid7cqcg0zi98ab1y", "ynizhwjmd25jf0np5wi4wq34fbd6n9db", "zc6w4ys5ka740i167bd0qp6v8f63dp0i" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "awscli:1.22.88,bash:5.1,coreutils:9.0,curl:7.82.0,git-lfs:3.1.4,git:2.35.3,gnutar:1.34,gzip:1.12,jq:1.6,nss-cacert:3.77", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:31d3904ee778486ecba2e05b987150e8f12d0d942cd46332a1fface881e63951" ], "Env": [ "PATH=/bin/", "SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt" ] } { "Name": "docker.io/nebulaworks/flasksample", "Digest": "sha256:96c7fcca49c569e1bdb9d43270324b3e23747adc2f398f3dbe2099a19e2df0e8", "RepoTags": [ "3hnryiq8q9hy0fm8cfzm92d0yj8anybk", "46436y8vy5dm1pcznyqnbjjsbcqfsa13", "54p4a6ri41prjrp9w9wni8v0bigs8j60", "5fi5d6s9qb0ayfxvd4czrwvrk0jv9q1r", "65j7jsrql5qicfadj3dqbinyaaisi5sg", "68rag50ayc9gsi7n4bwdbjq0ah2pcs0a", "6q1rg8nbx5yzhrs8y16l8czmy5xsqncc", "8qzz162pgn7nhbic9g0vw209fs01n6s6", "8wp5sb35h86bygxchcv9cgcsxzh2dc5p", "9dqy2plqw2n9nynz449adx8rd2633a16", "9m2k44fp1v7nvijz8mz7nkac10fs8wgv", "aggzwxjm6v6jfbjbwn1z0fvfp0xiqid6", "biayqv0sg5v36dvnl7yh6gg941gl0ba6", "c34ssc86npknkb6ppmfi4b9zvkxax4a8", "c44zddyvxg5b9qikb28gva9cznsys502", "d7m3af9b424fnb0niqs1s9428vqn7v76", "fgi5b0x05qfav9cg9ly8mn6pz7a2gibc", "fwwz8b6jm0q8m2w0lgw68mila23r08cp", "iiij8fj3jib4qyf6dzm64a9c79wrhk71", "jb78a05hixi8d08zlv5sg6ml3qsfzisd", "jd35ng454lzbzychimd6pcs0sx8w66ng", "jg3sxqk5msyl62asq4252m5faapqq0k7", "kxhq2jcijnkp8qhnlhy47023jcd0rj52", "lrpkr89z5qgb9sacgy266j3yxgrxi80a", "mk0ldb79sxm75wy99zk195dngl3dk5k0", "nyqmq3dsyzgm84ifcv29f4xaa2g60mnr", "p028d080qwsa8mh0lzfm1vvdh1kw7a8m", "rzk3f7a3w0ww8rlzi3401z3bz4mkmrn4", "vq9wsp3l0yx3vmjazfjf7qmv5952pik9", "wq3kagq6zqjfklx4y1s12gqp8fhd32sk", "xsi7aqn2j1bri2wkgc1zzk159m2dib3f", "yi1b05aik0iqjzpxqyy8nfzhy242s2y7", "z0qrlc5rpybi69hljcvdx3hlzac6f3g3", "zzpm4r45shaicn75lgvlnhxfqk3rgm9x" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "bash:5.1,coreutils:9.0,flasksample:0.1.0", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:35a4a7f33dc06ab38997070fbaaa563e4f9eb19b9663f36c61b97592f452f347" ], "Env": [] } { "Name": "docker.io/nebulaworks/helmsman-aws", "Digest": "sha256:ee3e8ece734b5937a23a3f008909688e12c6134e7704cbf99621d5508e837bbb", "RepoTags": [ "017agn5z2mr7pacaf09m8icbqwa5g01c", "0r4rskk3b5han0zzncj67sjhkm7lip1w", "1bhgpxkhzfc42i0dgxqxaw6bd2lhhap0", "2i051yfnpypb2dljzdbxdz04h3a5qdqc", "2vil2iz4gz0lmcqsg92kxylhh4mds6jv", "2w05w5i610g4jqnhcbhv1ayd81gxysds", "30q2z6wbcbn88851pyyyns0bn5km4nwm", "36na3i0psj09247r3mswgf2czazg959m", "37yxlxhpyp3ai1xs464ldif2gmj3yyjr", "3dz3i6lqdnigfnq7mai6vccnmf4fyjbx", "3ml4yvayn282z32hpr50sl7ag5kdca72", "42ygsh6v9ns61hn7yrrvjrpa7dz0ypak", "4szd9xflnkl3ab9h0mzjkfrybzps3hs4", "4x6ah6c8mgyn20awa9xrq16cw1yz9d4s", "52ifj8li3wjpraps049zyq67zqvw7m04", "54hq16h32rh3g6nbxmf755d40mvq30rg", "686gs4mwpvsisq9c45n0f19wmp1zwada", "72rs42ar65syxrgq6w0qyv98n97zkr6n", "787wzh711cg6l2s92vn5rlnkw24jslyd", "7ar2sigpxxyll06yvgan428qn74k0vqm", "7hl8p275d7f3gcnam52s7hg1d307a2ik", "7mc83swh71lgji4ar8fz8a1n13zq934p", "7vc5g5g4m6wvg75s527h5mfimixh1yl0", "91hn17cblaiv1wb26vvwz05m3787c4zv", "9xnr8m6mj35mh6fwxx1nsxp11arvm1sh", "a5kn44776qgdm8dgjmlym4z79qhnm6mn", "ajgcdsjx2hy33jgrvgqxkjfv28bazfz1", "axk5j5bhv0sv9ys1xj4bmhc34kfw5jkn", "bbjyib4li8ipq2yvyak79ck7kw9k2dkx", "cr9n83qbmcxkll790qhsmqk3x4733zdv", "d5vzm6xvcdv93dl327shyc5k8af6vh5f", "fh5ba96qrhkkgilk03srysz5qm7kg0j8", "g9mxjnw3vmxnzc14hbyl5rvjmhrkzdcb", "h6a3l42pl14r96j5y1kcmwf5b4g8d5rg", "hwjzmzm50q20diz32nlmva91cpjjy0gn", "ibx9pd0yzvspf6gczmdmmhmam9sr25wr", "k345a553cr4aicxcl9l7ns2rjzhiv62v", "knha5b2p70xhrqyy5awl8djdzisn2sal", "l0mz1yxw70ab4kmqnvjbcxklm088q7l8", "lf015xyadsq38mvllc2cvb0n9dfdj6zc", "lnv4scg68mn4ambxibzq7rswrfapcnrk", "lpsiwff3za9ma0hsvwz9kyvsc8han1nc", "m4p9y2j9jwvrlrzcyk0s7q5b1qh3i4xc", "m9kvdhzskjmvd6lbrmx9n195cm08amgq", "mpgikzzyqmx0x97jq9iwmf7r9xvm7x38", "nlpym68q25fyyzdzq7xd2gcbibbwmfyb", "paks7yd7laypwmpg3i3mbvw937hdgdnc", "pbj9q2qyw511wfy18g8lz9jw2mpc5qvw", "pfvnzcg5m26m0bd6cj4v6g7j6qndi2m0", "rz9fgbx5kvzid11v516zk8l3x31bxph0", "s9nhvzqn3acjcfinaqlsd07hnc7bkm0i", "saj7j4vdgy3infrgn865xlbb5n73lsd9", "sb47bkwd854ws9dc4gzsvnln06pidxjf", "sgv2bi85zdbbznixhpck8w5c29np1w87", "v4mlmsr2lpvjh3pykpg3b3755zsp5yip", "v7jsvzj6gr3p02s2lp36r3i7rlrij76w", "vibjzpbdqxyfxppd37sy8sskbn9knd10", "w01q6nzkxlxa2w0kiai4spxc17mq7a04", "xdahy9hh3fnsws09w7ksi65d27kra0i1", "xym39zji91ga5rbw6p8xbwy5dcbizbn3", "zmz5fxsfpmjdf28mmknxaqwp9ig4kyxa" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "awscli:1.22.88,bash:5.1,coreutils:9.0,helm:3.8.2,helmsman:3.8.1,jq:1.6,kubectl:1.23.5,nss-cacert:3.77", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:4e0588d0d53360e0f3a00dbd1cb817a87408324088e8d47660ab3135d1aa9982" ], "Env": [ "PATH=/bin/", "HELM_PLUGINS=/share/helm/plugins/" ] } { "Name": "docker.io/nebulaworks/magic-wormhole-mailbox", "Digest": "sha256:95f8daecebe7363dee0c02e4d8c84a064042742241be8337a8299852c3d70357", "RepoTags": [ "138ca83w6z67cqfj6j3qnrg0cwiq35zg", "1b0l1kbsfnvsp2hc0825s66x6bm3j3w6", "350xjs0shfdl86h6wwn78jckhq2640l5", "3qsccb3a1wv9rrf5vvj7cpv1inmi1ah0", "46f5vfsc69l1zmzw2f5z02isjnp7xfsd", "63insgj15jx0qn14jmcixk8x5svq41cl", "689cg14skn51dqs5rad4579lajq8qa32", "705kflnqwjdm0faxwf9rwsynglb51mi8", "7868xcav57m6jaak54z97b87g0016sv7", "7pdcv4l8yszm7s1za0xd06pbnhqqdjwm", "81j5zcmmz59azhqbyrz3qbp4sybnnsbk", "8zi126fnfyvb5825xpd5qpbiykgkbz72", "8zk1sf2vvxa2w0wzyc05v3qlvyrlsr6w", "9s06afg8b3sgnq8mdx0g5sz6377ai8cb", "bgm92hyjwmaayyagdpyadsz2wy0w7wb9", "dcjzvca7cak7qlqz0ibwgc0k810v7fks", "dmlj2inrm7af7mfc412jkdn6n1s9hg3h", "f5d5asfq9a7d4969bpxn2l9x2p5x5gfb", "i4aiffyr6501ism0si9jvp2ggi89x08k", "ilijgx93k6pc9ih2bq39w6y3gvhlhzhi", "iphf9vs3ppknkagkfv4q02jbyxbwxpgc", "jd975cl6d7gvbhszy74zvh6k94m511gy", "jf9w4rj0gz8qnbnwwqcsrzp1bnmrw009", "k1kzqaw3p8vfz2h6p4ipzmnxa1wfg7gy", "kdwng0vdvf0s9vjz1lwgkr7l69g68fji", "kgj2b4lfv5xsci0r32n7z2kayp65xnhs", "lf9rigldbnkkqw7ahish34qyqc885ffh", "lizc5jamh39zy5xm4cjhnd3a9v3gxzm9", "lizqad1x8vf27gjanjz9sfvz772w7gw8", "mncbrkl3cvkqm6gd3hjiql2ci4djf1gp", "msahb86b470r5636v8imzs8sghgzg38b", "njf33wky3rf1fpfzs4f42nn3pzzqsjb7", "p3krs4mic049x5ljdjyqahfncg7mi4d2", "r3vf7jxiwd7jgddib176nxw3p37n3ipz", "s1gqq531a1r0aqzz11ia3v1sf9wigf9n", "szjnqqlf679f3i3gh66z0kv1fxcd06fv", "v57lk1gziq5d5hlkz99yvwzz91di205z", "xy3wmckpk7imbj3wv3yw9bfzh6bqyzkk", "yg38kbca7ahgm2nms4rddq3kr51lv41m", "yyx0aasi1ypf9jhq0q3qd7gw1v40h6xi" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "bash:5.1,coreutils:9.0,procps:3.3.16,python3:3.7.13-env", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:11e4758951da8190c30684079a1e890acfde5269805936d5041822152ac5d9ee" ], "Env": [ "PATH=/bin/" ] } { "Name": "docker.io/nebulaworks/nwi-pr-utils", "Digest": "sha256:0779d324d703dd93894735c1e9efcc0aed46e79f82e65a76f3f117486fe4c667", "RepoTags": [ "09pymw41x3d1zar4i46pbdjlhf51nc7i", "1qcv3ddhn173p8zzqpps4bgvi96c73gc", "1vvy146w2xc2ia145kn88blwyg9303zi", "4ri9qfysqcry3zzjr1nmfwb0zalfl0li", "5x5r35ws5h8r4pig32s80hw6qfc1wvxw", "6n087vdgnm05ipil8zkrj1ynsw6l1lqd", "7w34va7ghlk4x5pj8h9f6xkb1xq60vlg", "8da78vfisd4431v063f59ybss8n6cgr2", "8glawain89wy3sp6i95ys7l7nyrsw38s", "8s3mm159mxlsgwa57fdfaarcm96m0z1z", "9gga1vwcxr09i95wxzsg0chmswdvsh84", "9n4k95fhxsy144rbnls6biyi74jpiz1m", "9w9qy1d813rxr9kdw4hvbzwjmdswycnp", "af48r4y9qz0rys0z2ph8av42hvg9ffx2", "c8a0x9ddsnb05nv16j8sj8ha4n3giq5h", "ciaif8kc0i1pqwcfpndgb88jpafw65cb", "f0gah5pqksnhf4v79yaf6wzqnf80bmk7", "f4p6mkakbl58807jrhwsbl5p34xr1jmw", "haqgpad651zq3f94zgky4ydz1lw4j93v", "hpaj9r73fl3c65j21lhc2dfpb4x61lrr", "hz31r35m48i4sfr43zmnq1xi7ph8cjc6", "imnysry78shjwnaznjlmjhnixrgkhd07", "iqaqf4a5cq85f9xh22dr2d2jxl1bh27a", "jff36955m87va40f8cfzwgwb04gv0yig", "jnd9wmc78n88k1wz6x4rbprwwrh47gbb", "kcg4l097kn841w814d425ar33y4alzsh", "kxfl84j0xyy3xhr0s6ga03xw9y784n5p", "l8azgxbr5xv6if0fa5jraxn61r9if6yq", "ldmmwra6z32jh7bqxp406c5kag8z492k", "m6b010vncwjifnjrninllnrs5yvdlsyl", "m84gwswh2k38aif8psvja3mjhxsbp41g", "q550kbmhsbm9y7yf5xkg1hghbpl8dgla", "qw1v97v9pwdhgasd755l33kzh1r1ydss", "r2d3pp1nhczk99084nihchn1rbrakiyf", "w8n8sggdlcaad4cx50jfx5yv1r2z6qvj", "zp9846y8pdd3ydfp8bi170dr4rrprclk" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "bash:5.1,coreutils:9.0,tfsec:1.19.0,yamllint:1.26.3", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:cb8109467374f540cac5bff55015d61c50cbfab5e0f5033c08014fb9d91f36f2" ], "Env": [ "PATH=/bin/" ] } { "Name": "docker.io/nebulaworks/pki-validator", "Digest": "sha256:e4593bad42d2ed8411687828e9a71c88019f6da60ff67aecbc06c4b8b3e49065", "RepoTags": [ "0ky9lc8k9cjn8y1yjdrcyjysi4f5mpjn", "12x2y4s04wqd7vzblyknmb4i9wlrbyg7", "171fampm86jhqnb40x2s0n08r4hv55h2", "193ch432hry14dwwgfzw98pr41w36kz6", "2s0sq3h58kpm1gp5sa7xghww5gn9vgaa", "4zdwvwli58ykjiigwqxm6ggwcbivd0ks", "5ydsd6yyfnhilz53bv5f31gwzrpp0yjf", "879vbhnlpb9fihpldnlwd6y2cxh449m7", "8hs2gsf3n1qq8rlg8md4m3571jkddg8g", "a78vqgf0nk9m39f9jz56qcqkz3pg85vi", "bcykjxyzc4wpgfvz7ba4n389vcgls45v", "bsnlb78w0hpzkx4m7ba40lm0qz2212js", "dnvg43r1cpmdjcfphknzgijmg7jx16cn", "fjqp40jq1fvzl1br6mhhr171q27xzm46", "g7hfbpy7vw7xq84zqhqigydyzv60k6s7", "h13qqs6wwmkz9g5hb9ca9hnyzchsp2al", "hg66vz8gd12bc035xaflv9bvyq9xi9yx", "i5h82x58h7qrvcyk2v1gzqwhb6w9kd0s", "ipamimb49hr2xw2l08d1p4r7jg6r5814", "iy254z2954z7zanh9dfbyzw6n00rkych", "jqx100dphlsaqbv9h6abp8ci4a4v7ka5", "kvfqd087z62z3amrvkzpx48l1v0blcdp", "lkr5lv92r9lin6s156x7x005dyikb21n", "pmmsixyc79inwhsifks8vmj53f52i9ph", "pw9il8xx85cgrdszn1vxmv319pyifp72", "qwcky9z36f8q51kd3rjph1fajmgl7d7n", "rhpf73j3qcmzkb5crdjn977iffqd1zqx", "rqs2qm2w6pqkvm8qhg6fpg1ddkjfvyc2", "rwx2sxnrf9b5yyaqyhs28458pzfiw912", "s1l8jmzblv13rnzs9ggsd2xrssalj52a", "v8n2gd8gx7i2hwbmbvpy083gqng3rv4s", "yxdhv9knsflnpfihzksi2m7qix2kircl" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "bash:5.1,coreutils:9.0,gawk:5.1.1,gnugrep:3.7,gnumake:4.3,openssl:1.1.1n", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:91f107800fc4b27ddaae5466f0d9d1253b90f6794217cfb6c4cc6c283f602f1f" ], "Env": [ "PATH=/bin/", "SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt" ] } { "Name": "docker.io/nebulaworks/term-apply", "Digest": "sha256:50f6d4c7569cd3ed289302cee179286926f75c4b0f03d47e761f7a261f7c7900", "RepoTags": [ "589ykblva4s7mycz50bwfmxkhzg1i2j8", "6b6qszrhrxxkbxixw3hbns8rbck0r1iy", "97a542xvxs9n5czccsh4z7chlrc7zhal", "gxk9nj760dj16rmrmflym9kwc06phpi5", "k0lg0i9z1p4zf2yk559scvga62bwcmda", "qkmy576gdjx203zxl4sm0xizrdhxiqmw", "srm4d2spzfzljc02dvyjbw8db47g0m9v" ], "Created": "1970-01-01T00:00:01Z", "DockerVersion": "", "Labels": { "com.nebulaworks.packages": "awscli2:2.5.6,bash:5.1,coreutils:9.0,minio-client:2022-03-17T20-25-06Z,nss-cacert:3.77,term-apply-unstable:2022-03-29", "org.opencontainers.image.authors": "Nebulaworks Inc.", "org.opencontainers.image.source": "https://github.com/Nebulaworks/nix-garage" }, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:6b7c0da3c794fd5c6f8711719fe6071cfabc90a6415d6e4f3b886b6e3f8c9a72" ], "Env": [ "PATH=/bin/", "SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt", "CLICOLOR_FORCE=1", "TA_UPLOAD_DIR=/usr/local/term-apply/resumes", "TA_DATAFILE=/usr/local/term-apply/data/applicants.csv" ] } ansible-lint:cx11fsg47srrbpg41x1kx9zd1ymyqmgr
have been published on Docker Hub.

kylerisse · 2022-04-28T05:51:57Z

@sarcasticadmin so a couple things fall out of this.

this is probably not the "correct" solution, just a quick hack to validate my findings
A failure here doesn't fail the job, it just skips building any containers and is undetectable without checking the log

sarcasticadmin · 2022-04-28T14:53:45Z

@sarcasticadmin so a couple things fall out of this.

this is probably not the "correct" solution, just a quick hack to validate my findings

Im not really sure what alternatives we have either? I wouldnt be opposed to just leaving this and commenting why it exists for the time being. As am looking through the docs it seems like maybe the flag --read-write-mode would work in the single nix-instantiate --eval call: https://nixos.org/manual/nix/stable/command-ref/nix-instantiate.html

When used with --eval, perform evaluation in read/write mode so nix language features that require it will still work (at the cost of needing to do instantiation of every evaluated derivation). If this option is not enabled, there may be uninstantiated store paths in the final output.

You call out in the above comments #49 (comment) this failure scenario with a clean /nix/store makes sense why --eval doesnt work

A failure here doesn't fail the job, it just skips building any containers and is undetectable without checking the log

Ya we need to blow up fantastically instead, totally agree with a follow up to try to make the failure here more obvious going forward. I think a separate issue in the backlog is fine vs fixing it here

kylerisse · 2022-04-28T16:20:32Z

Sounds good @sarcasticadmin

I wouldnt be opposed to just leaving this and commenting why it exists for the time being.

Added more detailed comment around this here

Ya we need to blow up fantastically instead, totally agree with a follow up to try to make the failure here more obvious going forward. I think a separate issue in the backlog is fine vs fixing it here

opened an issue for the other problem here

sarcasticadmin · 2022-04-28T16:25:57Z

publish-imgs

@@ -55,6 +55,14 @@ attr=${1:-'imgs'}
 # For storing all built and published images
 BUILT=()

+# When using a non-blank ${nixtop} (such as in CI), a new derivation will have to be built


Perfect thanks for adding this!

sarcasticadmin

sarcasticadmin · 2022-04-28T16:27:03Z

Great job @kylerisse thanks for working through this with me

kylerisse requested a review from sarcasticadmin March 20, 2022 05:19

sarcasticadmin reviewed Mar 21, 2022

View reviewed changes

imgs/default.nix Outdated Show resolved Hide resolved

kylerisse force-pushed the kr/new_img_ansible-lint branch from 02e50af to 22bb84b Compare March 22, 2022 02:21

kylerisse force-pushed the kr/new_img_ansible-lint branch from 22bb84b to 8cde894 Compare April 14, 2022 00:42

kylerisse changed the title ~~[WIP] - add ansible-lint image~~ [REVIEW] - add ansible-lint image Apr 14, 2022

sarcasticadmin approved these changes Apr 14, 2022

View reviewed changes

kylerisse force-pushed the kr/new_img_ansible-lint branch from 8cde894 to a3fbd1d Compare April 16, 2022 01:29

kylerisse force-pushed the kr/new_img_ansible-lint branch from a3fbd1d to d553f7d Compare April 25, 2022 22:54

kylerisse changed the title ~~[REVIEW] - add ansible-lint image~~ [WIP] - add ansible-lint image Apr 26, 2022

sarcasticadmin closed this Apr 26, 2022

sarcasticadmin reopened this Apr 26, 2022

This was referenced Apr 28, 2022

Should be pinning nixpkgs to something predictable #59

Closed

Make the CI more dumb #60

Open

[READY] - [issue-59] - init ci.nix and GNUmakefile #61

Merged

kylerisse added 2 commits April 27, 2022 20:31

add local testing instructions for imgs

e638e42

adds ansible-lint image

52e7399

kylerisse force-pushed the kr/new_img_ansible-lint branch from d553f7d to 52e7399 Compare April 28, 2022 03:32

add workaround in publish-imgs for ci use case

84f8cd1

kylerisse force-pushed the kr/new_img_ansible-lint branch from d20f3dc to 84f8cd1 Compare April 28, 2022 15:49

kylerisse mentioned this pull request Apr 28, 2022

publish-img doesn't fail properly in CI when nix-instantiate has errors #64

Open

kylerisse changed the title ~~[WIP] - add ansible-lint image~~ [READY] - add ansible-lint image Apr 28, 2022

sarcasticadmin reviewed Apr 28, 2022

View reviewed changes

sarcasticadmin approved these changes Apr 28, 2022

View reviewed changes

sarcasticadmin merged commit 6ef0395 into Nebulaworks:master Apr 28, 2022

[READY] - add ansible-lint image #49

[READY] - add ansible-lint image #49

Conversation

kylerisse commented Mar 20, 2022 • edited Loading

Description of PR

Tests

sarcasticadmin left a comment

Choose a reason for hiding this comment

kylerisse commented Mar 22, 2022

sarcasticadmin left a comment

Choose a reason for hiding this comment

sarcasticadmin Apr 14, 2022

Choose a reason for hiding this comment

sarcasticadmin Apr 14, 2022

Choose a reason for hiding this comment

sarcasticadmin Apr 14, 2022

Choose a reason for hiding this comment

sarcasticadmin Apr 14, 2022

Choose a reason for hiding this comment

sarcasticadmin commented Apr 14, 2022

kylerisse commented Apr 14, 2022

kylerisse commented Apr 16, 2022

sarcasticadmin commented Apr 16, 2022

sarcasticadmin commented Apr 16, 2022

sarcasticadmin commented Apr 16, 2022

kylerisse commented Apr 25, 2022

kylerisse commented Apr 25, 2022

kylerisse commented Apr 26, 2022

sarcasticadmin commented Apr 26, 2022 • edited Loading

sarcasticadmin commented Apr 26, 2022

kylerisse commented Apr 26, 2022

sarcasticadmin commented Apr 28, 2022

kylerisse commented Apr 28, 2022

nwi-auto commented Apr 28, 2022

kylerisse commented Apr 28, 2022

nwi-auto commented Apr 28, 2022

kylerisse commented Apr 28, 2022

kylerisse commented Apr 28, 2022 • edited Loading

kylerisse commented Apr 28, 2022

kylerisse commented Apr 28, 2022

nwi-auto commented Apr 28, 2022

kylerisse commented Apr 28, 2022

sarcasticadmin commented Apr 28, 2022

kylerisse commented Apr 28, 2022

sarcasticadmin Apr 28, 2022

Choose a reason for hiding this comment

sarcasticadmin left a comment

Choose a reason for hiding this comment

sarcasticadmin commented Apr 28, 2022

kylerisse commented Mar 20, 2022 •

edited

Loading

sarcasticadmin commented Apr 26, 2022 •

edited

Loading

kylerisse commented Apr 28, 2022 •

edited

Loading