Add e2e tests on ci + commands to trigger it

.github/workflows/dotnet.yml

@@ -1,25 +1,180 @@
-name: Unit testing
+name: CI
 
+# Single pipeline: detect relevant changes → unit tests + e2e tests → build & push the image.
+# The image build `needs` BOTH test jobs, so an image is only published when unit AND e2e pass.
+#
+# "Required only on code changes": the test/build jobs are gated on a paths filter via job-level
+# `if` (NOT `on.paths`). On a docs-only PR the jobs are skipped, and GitHub treats a skipped
+# required status check as passing — so unrelated PRs aren't blocked, while C#/version/dependency
+# PRs must pass unit tests.
+#
+# e2e is opt-in per PR via the `run-e2e-tests` label: it runs only while that label is present.
+# `labeled`/`unlabeled` are in the trigger list so adding the label starts a run and removing it
+# starts a fresh run where e2e is skipped; the `concurrency` block cancels the in-flight run so a
+# still-running e2e is stopped when the label is removed.
 on:
   pull_request:
     branches: [ main ]
-    types: [ opened, synchronize ]
+    types: [ opened, synchronize, reopened, labeled, unlabeled ]
+  push:
+    branches: [ main, master ]
+    tags:
+      - 'v[0-9]+.[0-9]+.[0-9]+*'   # semver release tags, v-prefixed (v1.2.3, v1.2.3-rc1)
+      - '[0-9]+.[0-9]+.[0-9]+*'    # semver release tags, bare (1.2.3)
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: read   # dorny/paths-filter uses the REST API on pull_request events
+  packages: write       # push the image to ghcr
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: ${{ github.repository }}
+
+# One in-flight run per PR (or per ref for pushes). Removing the `run-e2e-tests` label triggers a
+# new run that cancels the previous one, stopping an e2e job that was still executing.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 jobs:
-  build-and-test:
+  # Detects whether C# code, versions, or dependencies changed. Everything else gates on this.
+  changes:
     runs-on: ubuntu-latest
+    outputs:
+      code: ${{ steps.filter.outputs.code }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # full history so the filter can diff push events too
+      - uses: dorny/paths-filter@d1c1ffe0248fe513906c8e24db8ea791d46f8590 # v3.0.3
+        id: filter
+        with:
+          filters: |
+            code:
+              # C# / generated code
+              - '**/*.cs'
+              - '**/*.razor'
+              - '**/*.cshtml'
+              - '**/*.proto'
+              # versions & dependencies
+              - '**/*.csproj'
+              - '*.sln'
+              - 'Directory.Build.props'
+              - 'Directory.Packages.props'
+              - '**/packages.lock.json'
+              - 'global.json'
+              - 'nuget.config'
+              - 'NuGet.config'
+              # build/e2e harness that exercises the above
+              - 'src/Dockerfile'
+              - 'docker-compose.yml'
+              - 'docker/**'
+              - '.github/workflows/dotnet.yml'
 
+  unit-test:
+    needs: changes
+    # Run for code/version/dependency changes, and always on manual dispatch and tag (release) pushes.
+    if: ${{ needs.changes.outputs.code == 'true' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/') }}
+    runs-on: ubuntu-latest
     steps:
-    - name: Checkout code
-      uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+      - uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: '10.0.x'
+      - name: Restore
+        run: dotnet restore
+      - name: Unit tests (excludes e2e)
+        run: dotnet test --filter "Category!=E2E"
+
+  e2e-test:
+    needs: changes
+    # Same code/dispatch/tag gate as unit tests, but on PRs it additionally requires the
+    # `run-e2e-tests` label. Adding the label starts an e2e run; removing it re-runs the workflow
+    # with e2e skipped (a skipped required check counts as passing, so the PR isn't blocked).
+    if: >-
+      ${{ ( needs.changes.outputs.code == 'true'
+            && ( github.event_name != 'pull_request'
+                 || contains(github.event.pull_request.labels.*.name, 'run-e2e-tests') ) )
+          || github.event_name == 'workflow_dispatch'
+          || startsWith(github.ref, 'refs/tags/') }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Run e2e tests
+        # Clean slate first (DbInitializer only funds the dev wallets on an empty DB), then `run`
+        # brings up the chain (bitcoind → alice/bob/carol → setup-e2e + extract-env → nodeguard →
+        # e2e-runner). The runner's exit code becomes the job result.
+        run: |
+          COMPOSE_PROFILES=polar,e2e docker compose -f docker-compose.yml down -v --remove-orphans || true
+          COMPOSE_PROFILES=polar,e2e docker compose -f docker-compose.yml run --rm --build e2e-runner
+      - name: Tear down
+        if: always()
+        run: COMPOSE_PROFILES=polar,e2e docker compose -f docker-compose.yml down -v --remove-orphans
+
+  # Build + push the image ONLY after unit + e2e pass, on: pushes to main/master (with relevant
+  # changes) and semver tag pushes (releases). Never on PRs or manual dispatch — those only test.
+  docker-build:
+    needs: [ changes, unit-test, e2e-test ]
+    # Test success is asserted explicitly so a future change to the test jobs' triggers can't
+    # silently let an image build (or silently skip it).
+    if: >-
+      ${{ !cancelled()
+          && needs.unit-test.result == 'success'
+          && needs.e2e-test.result == 'success'
+          && github.event_name == 'push'
+          && ( startsWith(github.ref, 'refs/tags/')
+               || ((github.ref == 'refs/heads/main' || github.ref == 'refs/heads/master')
+                   && needs.changes.outputs.code == 'true') ) }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log into the Container registry
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
 
-    - name: Setup .NET
-      uses: actions/setup-dotnet@v3
-      with:
-        dotnet-version: '8.0.x'
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          # Branch pushes → branch name (+ `latest` on the default branch). Semver tag pushes →
+          # full version + major.minor + major (the `v` prefix is stripped by type=semver).
+          tags: |
+            type=ref,event=branch
+            type=raw,value=latest,enable={{is_default_branch}}
+            type=semver,pattern={{version}}
+            type=semver,pattern={{major}}.{{minor}}
+            type=semver,pattern={{major}}
+          labels: |
+            commit=${{ github.sha }}
+            actions_run=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
 
-    - name: Restore dependencies
-      run: dotnet restore
+      - name: Build and push the Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: src/Dockerfile
+          push: true
+          platforms: linux/amd64
+          provenance: false
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
 
-    - name: Build and test
-      run: dotnet test
+      - name: Scan SBOM
+        uses: anchore/scan-action@3343887d815d7b07465f6fdcd395bd66508d486a # v3.6.4
+        with:
+          image: ${{ steps.meta.outputs.tags }}
+          add-cpes-if-none: true
+          output-format: table
+          severity-cutoff: critical
+          fail-build: false

.justfile

@@ -106,7 +106,16 @@ docker-down:
 
 # Stops the development docker containers and removes the volumes, add DOCKER_COMPOSE_FILE to override the default file
 docker-rm:
-    docker compose --profile polar --profile loop --profile 40swap -f {{DOCKER_COMPOSE_FILE}} down -v
+    docker compose --profile polar --profile loop --profile 40swap --profile e2e --profile mempool -f {{DOCKER_COMPOSE_FILE}} down -v
+
+# Runs the option-B end-to-end rebalance test in containers: brings up the regtest stack + a live
+# NodeGuard, then the runner opens a channel via gRPC and rebalances. Exit code = test result.
+# Starts from a CLEAN slate (down -v first) — DbInitializer only funds the dev wallets when the DB
+# has none, so a stale postgres volume would leave the wallet unfunded ("no UTXOs" on OpenChannel).
+test-e2e:
+    -docker compose --profile polar --profile e2e -f {{DOCKER_COMPOSE_FILE}} down -v --remove-orphans
+    docker compose --profile polar --profile e2e -f {{DOCKER_COMPOSE_FILE}} run --rm --build e2e-runner
+    docker compose --profile polar --profile e2e -f {{DOCKER_COMPOSE_FILE}} down -v --remove-orphans
 
 ##########
 # Dapr #

Tiltfile

@@ -2,7 +2,7 @@ ci_settings(readiness_timeout = '10m')
 
 docker_compose([
   "./docker-compose.yml",
-], profiles = ["polar", "loop", "mempool", "40swap"])
+], profiles = ["polar", "loop", "mempool", "40swap", "e2e"])
 
 # Labels are used to group containers on the UI.
 labels = {
@@ -45,16 +45,27 @@ labels = {
   'grafana': [
     'grafana',
   ],
+
+  # Option-B end-to-end stack (live NodeGuard + test runner). Disabled by default — manual
+  # trigger only, consistent with NodeGuard itself not being Tilt-managed. Run via `just test-e2e`.
+  'e2e': [
+    'setup-e2e',
+    'extract-env',
+    'nodeguard',
+    'e2e-runner',
+  ],
 }
 
 for (label, services) in labels.items():
   for s in services:
-    # Mempool and 40swap-frontend services are included but stopped by default (auto_init=False)
+    # Mempool, 40swap, grafana and e2e services are included but stopped by default (auto_init=False)
     if label == 'mempool':
       dc_resource(s, auto_init=False, labels = [label])
     elif label == '40swap':
       dc_resource(s, auto_init=False, labels = [label])
     elif label == 'grafana':
       dc_resource(s, auto_init=False, labels = [label])
+    elif label == 'e2e':
+      dc_resource(s, auto_init=False, labels = [label])
     else:
       dc_resource(s, labels = [label])

docker-compose.yml

@@ -7,3 +7,4 @@ include:
   - ./docker/mempool/docker-compose.yml
   - ./docker/40swap/docker-compose.yml
   - ./docker/grafana/docker-compose.yml
+  - ./docker/e2e/docker-compose.yml

docker/e2e/Dockerfile.runner

@@ -0,0 +1,12 @@
+# E2E test runner image. Runs the [Category=E2E] tests, which drive a live NodeGuard purely
+# over gRPC and mine via NBitcoin's RPCClient — so the runner needs nothing but the .NET SDK
+# and the repo (no grpcurl/curl). Build context = repo root.
+FROM mcr.microsoft.com/dotnet/sdk:10.0
+WORKDIR /src
+COPY . .
+RUN dotnet restore test/NodeGuard.Tests/NodeGuard.Tests.csproj
+
+# Forces [E2EFact] on regardless of the reachability probe (the runner targets nodeguard:50051).
+ENV RUN_E2E_TESTS=1
+ENTRYPOINT ["dotnet", "test", "test/NodeGuard.Tests/NodeGuard.Tests.csproj", \
+            "--filter", "Category=E2E", "--logger", "console;verbosity=detailed"]

docker/e2e/README.md

@@ -0,0 +1,53 @@
+# Containerized rebalance e2e (option B)
+
+A true end-to-end test of Lightning rebalancing: a .NET test runner drives a **live NodeGuard**
+entirely over gRPC — it **opens the source channel via NodeGuard's `OpenChannel` API** (so the e2e
+also covers channel opening), mines via NBitcoin's `RPCClient`, then performs a circular rebalance
+Alice→Bob→Carol→Alice and asserts success.
+
+## Run it
+
+```bash
+just test-e2e          # or: COMPOSE_PROFILES=polar,e2e docker compose run --rm --build e2e-runner
+```
+
+Everything is profile-gated (`e2e`), so a normal `tilt up` / `docker compose up` ignores it. In the
+Tilt UI the e2e services appear under the `e2e` label, disabled (manual trigger only).
+
+## Pieces
+
+| File | Role |
+|------|------|
+| `setup-e2e.sh` | Loads the bitcoind wallet, funds the LND nodes, opens **only** Bob→Carol + Carol→Alice (NodeGuard opens Alice→Bob). docker.sock pattern, like the polar `setup` service. |
+| `extract-env.sh` | Writes `nodeguard-macaroons.env` (LND host/macaroon/pubkey) from the mounted LND data volumes + a network `lncli getinfo`. The LND certs include the service-name SANs (`alice`/`bob`/`carol`), so TLS verifies. |
+| `nodeguard-entrypoint.sh` | Sources that env file, then launches NodeGuard. |
+| `Dockerfile.runner` | .NET SDK image that runs `dotnet test --filter Category=E2E`. The test does the gRPC + mining itself — no grpcurl/curl. |
+| `docker-compose.yml` | Wires `setup-e2e` + `extract-env` → `nodeguard` → `e2e-runner` with the right `depends_on` ordering. |
+
+The test itself is `test/NodeGuard.Tests/E2E/RebalanceE2ETests.cs`, gated by `[E2EFact]` (runs when a
+NodeGuard gRPC is reachable on `NODEGUARD_GRPC_ENDPOINT`, or `RUN_E2E_TESTS=1`).
+
+## Status / shakeout notes
+
+Every **runtime step** was validated manually against a live NodeGuard (open channel via gRPC →
+`ONCHAIN_CONFIRMED` → rebalance `Succeeded`, 500 sat fee). The **compose wiring** has not yet been run
+as a full stack in CI; watch these on the first CI run:
+
+- **App config env (fixed):** the published image (`dotnet NodeGuard.dll`) does NOT read
+  `launchSettings.json`, so `Constants`' required vars (`MINIMUM_CHANNEL_CAPACITY_SATS`,
+  `TRANSACTION_CONFIRMATION_MINIMUM_BLOCKS`, `ANCHOR_CLOSINGS_MINIMUM_SATS`, `DEFAULT_DERIVATION_PATH`,
+  `NBXPLORER_URI`, …) are set explicitly on the `nodeguard` service. Keep them in sync with the dev
+  launch profile.
+- **Clean slate required (fixed):** `DbInitializer` only funds the dev wallets when the DB has none
+  (`!Wallets.Any()`). A stale postgres volume from an interrupted run leaves the wallet rows present
+  but unfunded against the fresh chain → `OpenChannel` fails with "Error generating template PSBT"
+  (no UTXOs). `just test-e2e` and the CI job now `down -v` before running.
+- **Critical ordering:** `nodeguard` must start only after `setup-e2e` completes — NodeGuard funds its
+  hot wallet via bitcoind RPC, which fails ("No wallet is loaded") if the bitcoind wallet isn't loaded
+  yet. The `depends_on: service_completed_successfully` enforces this.
+- **Shared volumes** (`alice_lnd_data`/`bob_lnd_data`/`carol_lnd_data`) are declared in the polar
+  compose; they're referenced here by name within the same merged project.
+- **`nonroot` entrypoint:** the NodeGuard image runs as uid 65532; the entrypoint wrapper + `/shared`
+  mount must be readable by it.
+- **docker.sock** must be available to `setup-e2e` (it is on GitHub-hosted runners).
+- **Hot wallet id** is assumed to be `3` (the dev single-sig wallet); override via `E2E_HOT_WALLET_ID`.