Implement actions for mac-m1 runners (#4)

* Restart the Nix daemon on MacOS * Use absolute Nix path in post-build-hook * Restore the env vars in post-build-hook * Perform upload as runner * Make upload-paths executable * tee upload-paths * Detach socat twice * Whitespace * Add debug lines * Wait more for nix shell * Fixing m1 mac (#3) We can drop the echo $PATH in the test job, that was just for diagnosis * Update action.yml * Update action.yml * Update action.yml * Update test.yml * Update test.yml * Update test.yml * Update test.yml * Update action.yml * Use ncat instead of socat * Add a sleep * Make a phony upload request * Add a sleep * Add shebang * Perform the actual upload * Let the build hook receive copy logs * Give root user access to aws credentials * Save credentials from environment * Read from env * Redirect to file in sudo * Let post-build-hook perform upload Now that we give root user access * Run nix with full path * Expose multiple actions for various use cases * Fix runner.name condition for install nix * Avoid ternary operator in env var expression * Allow substitution in root path * Set NIX_PATH for mac-m1 * Remove export from NIX_PATH string * Remove unnecessary conditions * Avoid the deprecated ::set-ouput * Add overriden nix.conf opts * Remove comment * Experiment * Try running a nix build * Try adding back post-build-hook * Pass nix_path option to install-nix-action * Experiment with adding sleep * Experiment with increasing sleep * Wait until nix daemon comes back * Update readme * Remove empty testfile * Add disclaimer for external users --------- Co-authored-by: June <38109440+DevopsGoth@users.noreply.github.com>
kadena-io · Aug 28, 2023 · 88ed81f · 88ed81f
1 parent d977b8b
commit 88ed81f
Show file tree

Hide file tree

Showing 7 changed files with 258 additions and 66 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -3,7 +3,7 @@ name: Test the GitHub action of this repository
 on: [push]
 
 jobs:
-  test-build-and-cache:
+  test-build-and-cache-by-root:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -20,17 +20,14 @@ jobs:
         aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
         aws-region: us-east-1
 
-    - name: Run this repo's action
-      # For an external repo wishing to use this action, the uses field needs to be:
-      # uses: kadena-io/setup-nix-with-cache@{The version you want to use}
-      uses: ./
+    - # For an external repo wishing to use this action, the uses field needs to be:
+      # uses: kadena-io/setup-nix-with-cache/by-root@{The version you want to use}
+      uses: ./by-root
       with:
         cache_url: s3://nodemon-nix-cache?region=us-east-1
         signing_private_key: ${{ secrets.NIX_CACHE_PRIVATE_KEY }}
 
-    # This step is probably not needed for external repos that use this action
-    - name: Set up Nix path
-      run: echo "NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/22.11.tar.gz" >> $GITHUB_ENV
+    - uses: ./copy-root-aws-credentials
 
     - name: Attempt to build a new derivation with Nix
       run: |
@@ -40,3 +37,34 @@ jobs:
       run: |
         nix-build -E '(import <nixpkgs> {}).runCommand "fetch-example" {} "echo Example > $out"'
 
+  test-build-and-cache-by-runner:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, mac-m1]
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Set up AWS credentials
+      uses: aws-actions/configure-aws-credentials@v1
+      with:
+        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        aws-region: us-east-1
+
+    - uses: ./by-runner
+      with:
+        cache_url: s3://nodemon-nix-cache?region=us-east-1
+        signing_private_key: ${{ secrets.NIX_CACHE_PRIVATE_KEY }}
+
+    - uses: ./copy-root-aws-credentials
+
+    - name: Attempt to build a new derivation with Nix
+      run: |
+        nix-build -E '(import <nixpkgs> {}).writeText "build-example" (builtins.toString builtins.currentTime)'
+
+    - name: Attempt to fetch an existing cache entry
+      run: |
+        nix-build -E '(import <nixpkgs> {}).runCommand "fetch-example" {} "echo Example > $out"'
diff --git a/README.md b/README.md
@@ -1,7 +1,19 @@
-# Set up Nix along with a custom cache
+*NOTE*: The actions in this repository are meant to be used with Kadena's own github actions infrastructure, therefore they contain Kadena-specific logic. They may still be useful for external users out of the box or after some tweaks, but that's not officially supported by Kadena.
 
-GitHub action for setting up a simple Nix environment that caches to a custom Nix cache
+# Set up Nix along with caching
 
-## Usage
+A set of GitHub actions for setting up a multi-user Nix environment with a binary cache for uploading and fetching derivation outputs.
+
+The repository contains the following actions:
+* `by-root`: Sets up `nix.conf` with a `post-build-hook` so that intermediate and final build outputs are uploaded by the root.
+* `by-runner`: Starts a background upload job as the action runner and sets up `nix.conf` with a `post-build-hook` so that intermediate and final build outputs are passed to the action runner for cache uploads.
+* `copy-root-aws-credentials`: Copies the AWS credentials of the action runner to the root user. This is meant to be used for giving the root user access to binary caches. In the case of a `by-root` workflow, this can be used to give root write permissions and in the case of a private cache, it can be used to give root read permissions.
+
+Depending on the binary cache in question, these actions can be used together in a number of ways, some example use cases:
+* A public AWS S3 bucket: A `by-root` + `copy-root-aws-credentials` setup is recommended. One can also use `by-runner` only, but `by-runner` adds around a minute of setup time to each run.
+* A private AWS S3 bucket: `by-root` + `copy-root-aws-credentials` will work same as before. If `by-runner` is preferred, it will also `copy-root-aws-credentials` since otherwise the Nix builder won't be able to read from the private S3 bucket.
+* Other kind of cache (e.g. over SSH, etc.): As long as the github action runner has access to the cache, `by-runner` can be used to enable cache uploads, but if the cache is private, a custom step will be required to give the root user access to it. If root is given access, then `by-root` can also be used instead of `by-runner` in order to avoid the setup overhead.
+
+## Example
 
 Please check [.github/workflows/test.yml](.github/workflows/test.yml) for an example using an AWS S3 bucket as a Nix cache.
diff --git a/action.yml b/action.yml
diff --git a/by-root/action.yml b/by-root/action.yml
@@ -0,0 +1,79 @@
+name: 'Set up Nix environment with cache uploads by the root'
+description: -|
+  This action installs Nix in multi-user mode and configures it to
+  use a custom cache. It also uploads all the build results to the cache,
+  including intermediate packages built as part of a build activity.
+  The cache uploads are performed by the root user in the post-build-hook,
+  this means that the root user needs to have the proper credentials
+  configured to have write access to the cache.
+inputs:
+  cache_url:
+    description: 'URL for the Nix cache'
+    required: true
+  signing_private_key:
+    description: The private (secret) key used for signing Nix store paths
+    required: true
+runs:
+  using: composite
+  steps:
+    - name: Install Nix
+      uses: cachix/install-nix-action@v22
+      with:
+        nix_path: nixpkgs=https://github.com/NixOS/nixpkgs/archive/23.05.tar.gz
+
+    - name: Set NIX_PATH
+      run: |
+        echo "NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/23.05.tar.gz" >> $GITHUB_ENV
+      shell: bash
+
+    - run: |
+        echo $PATH
+        nix-build --version
+      shell: bash
+
+    - name: Populate the nix.conf with cache fields
+      shell: bash
+      run: |
+        TMP_DIR=$(mktemp -d)
+        echo "${{ inputs.signing_private_key }}" > "$TMP_DIR/key.private"
+
+        tee $TMP_DIR/post-build-hook.sh <<EOF
+        #!/usr/bin/env bash
+
+        set -euo pipefail
+        set -f # disable globbing
+        echo "Built paths:" \$OUT_PATHS
+        $(which nix) copy --to "${{ inputs.cache_url }}" \$OUT_PATHS 2>&1
+        EOF
+        chmod a+x $TMP_DIR/post-build-hook.sh
+
+        CERTFILEOPT=$( [[ "$OSTYPE" =~ darwin ]] && echo "ssl-cert-file = /etc/ssl/cert.pem" || echo "" )
+
+        sudo mkdir -p /etc/nix/
+        sudo tee /etc/nix/nix.conf <<EOF
+        show-trace = true
+        max-jobs = auto
+        $CERTFILEOPT
+        trusted-users = root ${USER:-}
+        substituters = ${{ inputs.cache_url }} https://cache.nixos.org/
+        trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
+        secret-key-files = $TMP_DIR/key.private
+        post-build-hook = $TMP_DIR/post-build-hook.sh
+        experimental-features = nix-command fetch-closure flakes
+        EOF
+
+    - name: Restart the Nix daemon on MacOS
+      shell: bash
+      if: ${{ runner.os == 'macOS' }}
+      run: |
+        sudo launchctl stop org.nixos.nix-daemon
+        sudo launchctl start org.nixos.nix-daemon
+        while ! nix store ping 2>/dev/null; do
+            sleep 1
+        done
+
+    - name: Restart the Nix daemon on Linux
+      shell: bash
+      if: ${{ runner.os == 'Linux' }}
+      run: |
+        sudo systemctl restart nix-daemon.service
diff --git a/by-runner/action.yml b/by-runner/action.yml
@@ -0,0 +1,94 @@
+name: 'Set up Nix environment with cache uploads by the runner'
+description: -|
+  This action installs Nix in multi-user mode and configures it to
+  use a custom cache. It also uploads all the build results to the cache,
+  including intermediate packages built as part of a build activity.
+  The cache uploads are performed by the runner user in the post-build-hook,
+  this means that the runner user needs to have the proper credentials
+  configured to have write access to the cache.
+inputs:
+  cache_url:
+    description: 'URL for the Nix cache'
+    required: true
+  signing_private_key:
+    description: The private (secret) key used for signing Nix store paths
+    required: true
+runs:
+  using: composite
+  steps:
+    - name: Install Nix
+      uses: cachix/install-nix-action@v22
+      with:
+        nix_path: nixpkgs=https://github.com/NixOS/nixpkgs/archive/23.05.tar.gz
+
+    - name: Set NIX_PATH
+      run: |
+        echo "NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/23.05.tar.gz" >> $GITHUB_ENV
+      shell: bash
+
+    - run: |
+        echo $PATH
+        nix-build --version
+      shell: bash
+
+    - name: Populate the nix.conf with cache fields
+      shell: bash
+      run: |
+        TMP_DIR=$(mktemp -d)
+        echo "${{ inputs.signing_private_key }}" > "$TMP_DIR/key.private"
+
+        tee $TMP_DIR/upload-paths.sh <<EOF
+        #!/usr/bin/env bash
+        read -r OUT_PATHS
+        export IFS=' '
+        echo Uploading paths \$OUT_PATHS
+        nix copy --to "${{ inputs.cache_url }}" \$OUT_PATHS 2>&1
+        EOF
+        chmod a+x $TMP_DIR/upload-paths.sh
+
+        NMAP=$(nix-build '<nixpkgs>' -A nmap --no-out-link)
+        $NMAP/bin/ncat -k -l 54321 -e $TMP_DIR/upload-paths.sh &
+
+        sleep 1
+        echo $NMAP | nc localhost 54321
+
+        tee $TMP_DIR/post-build-hook.sh <<EOF
+        #!/usr/bin/env bash
+
+        set -euo pipefail
+        set -f # disable globbing
+        echo "Built paths:" \$OUT_PATHS
+        echo \$OUT_PATHS | nc localhost 54321
+        EOF
+        chmod a+x $TMP_DIR/post-build-hook.sh
+
+        CERTFILEOPT=$( [[ "$OSTYPE" =~ darwin ]] && echo "ssl-cert-file = /etc/ssl/cert.pem" || echo "" )
+
+        sudo mkdir -p /etc/nix/
+        sudo tee /etc/nix/nix.conf <<EOF
+        show-trace = true
+        max-jobs = auto
+        $CERTFILEOPT
+        trusted-users = root ${USER:-}
+        substituters = ${{ inputs.cache_url }} https://cache.nixos.org/
+        trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
+        secret-key-files = $TMP_DIR/key.private
+        post-build-hook = $TMP_DIR/post-build-hook.sh
+        experimental-features = nix-command fetch-closure flakes
+        EOF
+
+    - name: Restart the Nix daemon on MacOS
+      shell: bash
+      if: ${{ runner.os == 'macOS' }}
+      run: |
+        sudo launchctl stop org.nixos.nix-daemon
+        sudo launchctl start org.nixos.nix-daemon
+        while ! nix store ping 2>/dev/null; do
+            sleep 1
+        done
+
+    - name: Restart the Nix daemon on Linux
+      shell: bash
+      if: ${{ runner.os == 'Linux' }}
+      run: |
+        sudo systemctl restart nix-daemon.service
diff --git a/copy-root-aws-credentials/action.yml b/copy-root-aws-credentials/action.yml
@@ -0,0 +1,34 @@
+name: 'Copy the AWS credentials to the root user'
+description: -|
+  This action copies the AWS credentials from the runner user to
+  the root user. This is meant to be used for actions where the
+  Nix daemon needs to read from or write to a binary cache that
+  requires authentication.
+runs:
+  using: composite
+  steps:
+    - name: Determine AWS root path
+      id: aws-root-path
+      shell: bash
+      run: |
+        if [[ "${{ runner.os }}" == "macOS" ]]; then
+          echo "path=/var/root/.aws" >> $GITHUB_OUTPUT
+        else
+          echo "path=/root/.aws" >> $GITHUB_OUTPUT
+        fi
+
+    - name: Copy aws credentials to root and clean up at the end
+      uses: pyTooling/Actions/with-post-step@v0.4.5
+      env:
+        AWS_ROOT_PATH: ${{ steps.aws-root-path.outputs.path }}
+      with:
+        main: |
+          echo Copy aws credentials to $AWS_ROOT_PATH
+          sudo mkdir -p $AWS_ROOT_PATH
+          sudo sh -c "cat - > $AWS_ROOT_PATH/credentials" <<EOF
+          [default]
+          aws_access_key_id = $AWS_ACCESS_KEY_ID
+          aws_secret_access_key = $AWS_SECRET_ACCESS_KEY
+          EOF
+        post: |
+          sudo rm -rf $AWS_ROOT_PATH
diff --git a/testfile b/testfile