feat(ci): update noir to build wasm with a nix flake file (#1208)

* wip * Change repo link * Update workflow * Use jq * Fix yarn * Dependency test * Create noir-script.json * Implement nix/cachix * Update path * Fix path * Change branch for noir-wasm-testing * Update workflow * Remove comment * Fix import * Symlink * Test fix * Output nix result * Revert and build wasm with build-wasm * Fix path * Debug * Test flake * Fix package * implements nix to build-wasm, testing * Run build-wasm * Change path * Update bash file * Output contents * Use wasm-pack again * Change quotes * Revert build-wasm * Update yml to use flake * Fix relative path * Print directory * Update path for wasm-pack * Add chmod * Update buildPhase * Try to fix path * Fix jq path * Fix jq * Fix missing commands * Change buildInputs * Fix missing improts * Install jq * Fix imports * Add dependencies to flake nix file * Debug errors * Debugging wip * Add install wasm step * Debugging * Remove wasm-pack installation on workflow * Ensure logging * Output full logs to a file * Log when failure happens * Typo * Fix logging * Remove workflow step * Debug wip * Debug wip * Stop on failure * Trying to output debug information * Fix path * Fix output file * Fix * Use xtracefd flag * Accidentally removed installPhase * Change workflow * Print from flake * Revert build stage * Remove output from noir_wasm build stage * Revert logging * Further logging tests * Trying experimental features * Other config * Remove experimental config * Remove logs * Output build log file location * Log out Hello World file from nix to workflow * Output hello_world.txt * Add else * Copy file from nix to workflow * Fix flake run * Change hello world to build log * Revert build-wasm * Output build-wasm build * Print all files in build-wasm * Add verbose on wasm-pack * Add which wasm-pack * Replace which with command -v * Manually generate output folders * Add debug flag * Remove further comments * Output wasm-pack version * Output rustc version * Add rustc to build inputs * Set up dev flag * Fix src path * add wasm32 target * wip - fix for section too large error * progress: build with wasm-pack * Downgrade wasm-bindgen - wip * chore: building with correct versions * choir: rely on stock wasm-binden-cli * Cleanup, implement build on flake file * Fix conditional * Test workflow * Edit workflow * Fix workflow * Test fix * Clone nodejs * Fix copy * Add recursive step to copy * Fix output * Remove noir_wasm publish step * Fix branch name * Remove condition on dispatch-noir_wasm * Update cargoArtifacts name * Remove wasm workflow, now noir_wasm runs the flake * Comment all the release workflow except for the dispatch * Readd wasm workflow * Typo * Remove wasm-pack * Update script * Update script * Update targets * wip - changes to build * Fix build * Fix link * Remove unused scripts * Update Cargo.lock * Update wasm-bindgen * Update flake lock * Update wasm-bindgen-cli to 0.2.86 * Rollback flake.lock changes * Rollback Cargo.lock changes * Remove space * Rollback flake.lock * Update packages (Cargo.lock) * Remove nix build flags * Kh noir wasm nix flake (#1657) * chore: ignoring output dir * chore: refactor noir_wasm package build * Uncomment workflows * Revert workflow changes * Revert workflow * Remove unfinished testing workflow * Remove output from build wasm package * Revision * Split nix flake * Add preBuild and postBuild to build.sh * Remove cleanCargoSource * Remove preBuild * Removes package.json generation on installPhase * Remove package version extraction from cargo.toml * Readd preBuild (not in flake) * Remove extra --release flag * Add package.json, output debug info * Add package.json for noir_wasm * Fix package.json path * Bump package version * Fix path * Fix path * Output debug info * Fix order on workflow * Run release-please on wasm workflow * Fix wasm and release workflows * Update crates/wasm/README.md Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> * Update .github/workflows/wasm.yml Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> * Update flake.nix Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> * Update release-please-config.json Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> * Fix README.md * some todos * remove duplicated props * Change browser for web on path --------- Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: kobyhallx <kobyhall@proton.me> Co-authored-by: Koby Hall <102518238+kobyhallx@users.noreply.github.com>
noir-lang · Jun 16, 2023 · 2209369 · 2209369
1 parent 97d6747
commit 2209369
Show file tree

Hide file tree

Showing 16 changed files with 209 additions and 85 deletions.
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -18,15 +18,7 @@ jobs:
         uses: google-github-actions/release-please-action@v3
         with:
           token: ${{ secrets.NOIR_RELEASES_TOKEN }}
-          release-type: simple
-          package-name: noir
-          bump-minor-pre-major: true
-          bump-patch-for-minor-pre-major: true
-          prerelease: true # Marks GitHub Releases for 0.x.x versions as "Pre-release"
-          pull-request-title-pattern: "chore(noir): Release ${version}"
-          extra-files: |
-            Cargo.toml
-            flake.nix
+          command: manifest
 
   update-lockfile:
     name: Update lockfile

diff --git a/.github/workflows/wasm.yml b/.github/workflows/wasm.yml
@@ -2,7 +2,6 @@ name: Wasm
 
 on: [push, pull_request]
 
-# This will cancel previous runs when a branch or PR is updated
 concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.ref || github.run_id }}
   cancel-in-progress: true
@@ -11,19 +10,16 @@ jobs:
   build:
     name: Build Wasm
     runs-on: ubuntu-latest
-
     steps:
       - name: Checkout sources
         uses: actions/checkout@v3
 
-      - name: Setup rust toolchain
-        uses: dtolnay/rust-toolchain@master
+      - name: Setup Nix
+        uses: cachix/install-nix-action@v20
         with:
-          toolchain: 1.66.0
-
-      - name: Install wasm-pack
-        run: cargo install wasm-pack
+          nix_path: nixpkgs=channel:nixos-22.11
+          github_access_token: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Build wasm crate
-        working-directory: ./crates/wasm
-        run: ./build-wasm
+      - name: Build wasm package
+        run: |
+          nix build -L .#wasm
diff --git a/.gitignore b/.gitignore
@@ -6,6 +6,7 @@ node_modules
 pkg/
 
 # Nix stuff
+**/outputs
 result
 .envrc.local
 .direnv/

diff --git a/.release-please-manifest.json b/.release-please-manifest.json
@@ -0,0 +1,4 @@
+{
+  ".": "0.6.0",
+  "crates/wasm": "0.6.0"
+}
diff --git a/crates/wasm/Cargo.toml b/crates/wasm/Cargo.toml
@@ -27,4 +27,4 @@ gloo-utils = { version = "0.1", features = ["serde"] }
 getrandom = { version = "*", features = ["js"] }
 
 [build-dependencies]
-build-data = "0.1.3"
+build-data = "0.1.3"
diff --git a/crates/wasm/README.md b/crates/wasm/README.md
@@ -4,15 +4,10 @@
 
 In order to build the wasm package, the following must be installed:
 
-- [wasm-pack](https://github.com/rustwasm/wasm-pack)
-- [jq](https://github.com/stedolan/jq)
-
 ## Build
 
 The wasm package can be built using the command below:
 
 ```bash
-./build-wasm
+nix build -L #wasm
 ```
-
-Using `wasm-pack` directly isn't recommended as it doesn't generate a complete `package.json` file, resulting in files being omitted from the published package.
diff --git a/crates/wasm/build-wasm b/crates/wasm/build-wasm
diff --git a/crates/wasm/build.sh b/crates/wasm/build.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+function require_command {
+    if ! command -v "$1" >/dev/null 2>&1; then
+        echo "Error: $1 is required but not installed." >&2
+        exit 1
+    fi
+}
+
+require_command toml2json
+require_command jq
+require_command cargo
+require_command wasm-bindgen
+require_command wasm-opt
+
+export pname=$(toml2json < Cargo.toml | jq -r .package.name)
+
+./preBuild.sh
+cargo build --lib --release --package noir_wasm --target wasm32-unknown-unknown
+./postBuild.sh
+./installPhase.sh
+
diff --git a/crates/wasm/installPhase.sh b/crates/wasm/installPhase.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+mkdir -p $out
+cp README.md $out/
+cp ./crates/wasm/package.json $out/
+cp -r ./pkg/* $out/
diff --git a/crates/wasm/package.json b/crates/wasm/package.json
@@ -0,0 +1,23 @@
+{
+  "name": "@noir-lang/noir_wasm",
+  "collaborators": [
+    "The Noir Team <team@noir-lang.org>"
+  ],
+  "version": "0.6.0",
+  "files": [
+    "nodejs",
+    "web",
+    "package.json"
+  ],
+  "main": "./nodejs/noir_wasm.js",
+  "types": "./web/noir_wasm.d.ts",
+  "module": "./web/noir_wasm.js",
+  "sideEffects": false,
+  "peerDependencies": {
+    "@noir-lang/noir-source-resolver": "1.1.2"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/noir-lang/noir_wasm.git"
+  }
+}
diff --git a/crates/wasm/postBuild.sh b/crates/wasm/postBuild.sh
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+# TODO: Handle the wasm target being built in release mode
+WASM_BINARY=./target/wasm32-unknown-unknown/release/${pname}.wasm
+NODE_WASM=./pkg/nodejs/${pname}_bg.wasm
+BROWSER_WASM=./pkg/web/${pname}_bg.wasm
+
+wasm-bindgen $WASM_BINARY --out-dir ./pkg/nodejs --typescript --target nodejs
+wasm-bindgen $WASM_BINARY --out-dir ./pkg/web --typescript --target web
+wasm-opt $NODE_WASM -o $NODE_WASM -O
+wasm-opt $BROWSER_WASM -o $BROWSER_WASM -O
diff --git a/crates/wasm/preBuild.sh b/crates/wasm/preBuild.sh
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+# Clear out the existing build artifacts as these aren't automatically removed by wasm-pack.
+if [ -d ./pkg/ ]; then
+    rm -rf ./pkg/
+fi
diff --git a/crates/wasm/wasm.nix b/crates/wasm/wasm.nix
diff --git a/flake.nix b/flake.nix
@@ -64,6 +64,11 @@
         # We include rust-src to ensure rust-analyzer works.
         # See https://discourse.nixos.org/t/rust-src-not-found-and-other-misadventures-of-developing-rust-on-nixos/11570/4
         extensions = [ "rust-src" ];
+        targets = [ "wasm32-unknown-unknown" ]
+          ++ pkgs.lib.optional (pkgs.hostPlatform.isx86_64 && pkgs.hostPlatform.isLinux) "x86_64-unknown-linux-gnu"
+          ++ pkgs.lib.optional (pkgs.hostPlatform.isAarch64 && pkgs.hostPlatform.isLinux) "aarch64-unknown-linux-gnu"
+          ++ pkgs.lib.optional (pkgs.hostPlatform.isx86_64 && pkgs.hostPlatform.isDarwin) "x86_64-apple-darwin"
+          ++ pkgs.lib.optional (pkgs.hostPlatform.isAarch64 && pkgs.hostPlatform.isDarwin) "aarch64-apple-darwin";
       };
 
       craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;
@@ -164,6 +169,19 @@
         buildInputs = [ ] ++ extraBuildInputs;
       };
 
+      # Combine the environmnet with cargo args needed to build wasm package
+      noirWasmArgs = wasmEnvironment // {
+        pname = "noir_wasm";
+
+        src = ./.;
+
+        cargoExtraArgs = "--package noir_wasm --target wasm32-unknown-unknown";
+
+        buildInputs = [ ] ++ extraBuildInputs;
+
+        doCheck = false;
+      };
+
       # The `port` is parameterized to support parallel test runs without colliding static servers
       testArgs = port: testEnvironment // {
         # We provide `barretenberg-transcript00` from the overlay to the tests as a URL hosted via a static server
@@ -192,6 +210,7 @@
       # Build *just* the cargo dependencies, so we can reuse all of that work between runs
       native-cargo-artifacts = craneLib.buildDepsOnly nativeArgs;
       wasm-cargo-artifacts = craneLib.buildDepsOnly wasmArgs;
+      noir-wasm-cargo-artifacts = craneLib.buildDepsOnly noirWasmArgs;
 
       noir-native = craneLib.buildPackage (nativeArgs // {
         inherit GIT_COMMIT GIT_DIRTY;
@@ -210,6 +229,13 @@
         # We don't want to run checks or tests when just building the project
         doCheck = false;
       });
+
+      wasm-bindgen-cli = pkgs.callPackage ./wasm-bindgen-cli.nix {
+        rustPlatform = pkgs.makeRustPlatform {
+          rustc = rustToolchain;
+          cargo = rustToolchain;
+        };
+      };
     in
     rec {
       checks = {
@@ -252,12 +278,49 @@
           git
           nil
           nixpkgs-fmt
+          toml2json
           llvmPackages.lldb # This ensures the right lldb is in the environment for running rust-lldb
+          wasm-bindgen-cli
         ];
 
         shellHook = ''
           eval "$(starship init bash)"
         '';
       });
+
+      # TODO: This fails with a "section too large" error on MacOS so we should limit to linux targets
+      # or fix the failure
+      packages.wasm = craneLib.buildPackage (noirWasmArgs // {
+
+        inherit GIT_COMMIT;
+        inherit GIT_DIRTY;
+        doCheck = false;
+
+        cargoArtifacts = noir-wasm-cargo-artifacts;
+
+        COMMIT_SHORT = builtins.substring 0 7 GIT_COMMIT;
+        VERSION_APPENDIX = if GIT_DIRTY == "true" then "-dirty" else "";
+        PKG_PATH = "./pkg";
+
+        nativeBuildInputs = with pkgs; [
+          which
+          git
+          jq
+          rustToolchain
+          wasm-bindgen-cli
+          binaryen
+          toml2json
+        ];
+
+        postBuild = ''
+          bash crates/wasm/postBuild.sh
+        '';
+
+        installPhase = ''
+          bash crates/wasm/installPhase.sh
+        '';
+
+      });
     });
 }
+
diff --git a/release-please-config.json b/release-please-config.json
@@ -0,0 +1,26 @@
+{
+  "bootstrap-sha": "63d84a30fcbc117443cd3b404e872cb3c2f26111",
+  "release-type": "simple",
+  "bump-minor-pre-major": true,
+  "bump-patch-for-minor-pre-major": true,
+  "prerelease": true,
+  "group-pull-request-title-pattern": "chore(noir): Release ${version}",
+  "packages": {
+    ".": {
+      "release-type": "simple",
+      "component": "root",
+      "extra-files": ["Cargo.toml", "flake.nix"]
+    },
+    "crates/wasm": {
+      "release-type": "node",
+      "component": "noir_wasm"
+    }
+  },
+  "plugins": [
+    {
+      "type": "linked-versions",
+      "groupName": "noir",
+      "components": ["root", "noir_wasm"]
+    }
+  ]
+}
diff --git a/wasm-bindgen-cli.nix b/wasm-bindgen-cli.nix
@@ -0,0 +1,37 @@
+{ lib
+, rustPlatform
+, fetchCrate
+, nodejs
+, pkg-config
+, openssl
+, stdenv
+, curl
+, darwin
+, runCommand
+}:
+
+rustPlatform.buildRustPackage rec {
+  pname = "wasm-bindgen-cli";
+  version = "0.2.86";
+
+  src = fetchCrate {
+    inherit pname version;
+    sha256 = "sha256-56EOiLbdgAcoTrkyvB3t9TjtLaRvGxFUXx4haLwE2QY=";
+  };
+
+  cargoSha256 = "sha256-4CPBmz92PuPN6KeGDTdYPAf5+vTFk9EN5Cmx4QJy6yI=";
+
+  nativeBuildInputs = [ pkg-config ];
+
+  buildInputs = [ openssl ] ++ lib.optionals stdenv.isDarwin [ curl darwin.apple_sdk.frameworks.Security ];
+
+  doCheck = false;
+
+  meta = with lib; {
+    homepage = "https://rustwasm.github.io/docs/wasm-bindgen/";
+    license = with licenses; [ asl20 /* or */ mit ];
+    description = "Facilitating high-level interactions between wasm modules and JavaScript";
+    maintainers = with maintainers; [ nitsky rizary ];
+    mainProgram = "wasm-bindgen";
+  };
+}