feat(ci): add post-publish verification for npm package #323
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,239 @@ | ||
| name: Verify Published Package | ||
|
|
||
| # This workflow verifies that the published npm package works correctly | ||
| # across multiple Node.js versions using both global install and npx. | ||
| # | ||
| # Triggered: | ||
| # - Automatically after publish workflow completes | ||
| # - Manually via workflow_dispatch | ||
| # - On PR (for testing the workflow itself) | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - ".github/workflows/verify-publish.yml" | ||
| - "scripts/post-publish-verify/**" | ||
| workflow_dispatch: | ||
| inputs: | ||
| version: | ||
| description: "Package version to verify (default: latest)" | ||
| required: false | ||
| type: string | ||
| default: "latest" | ||
| workflow_call: | ||
| inputs: | ||
| version: | ||
| description: "Package version to verify" | ||
| required: false | ||
| type: string | ||
| default: "latest" | ||
|
|
||
| jobs: | ||
| verify: | ||
| name: Verify Node ${{ matrix.node }} | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| node: ["18", "20", "22"] | ||
|
|
||
| steps: | ||
| - name: Checkout (for scripts) | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js ${{ matrix.node }} | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: ${{ matrix.node }} | ||
|
|
||
| - name: Get package spec | ||
| id: pkg | ||
| run: | | ||
| VERSION="${{ inputs.version || 'latest' }}" | ||
| if [ "$VERSION" = "latest" ]; then | ||
| SPEC="agent-relay" | ||
| else | ||
| SPEC="agent-relay@${VERSION}" | ||
| fi | ||
| echo "spec=$SPEC" >> $GITHUB_OUTPUT | ||
| echo "Testing: $SPEC" | ||
|
|
||
| # Wait for npm to propagate the package | ||
| - name: Wait for npm propagation | ||
| if: inputs.version != 'latest' | ||
| run: | | ||
| echo "Waiting for npm to propagate version ${{ inputs.version }}..." | ||
| for i in {1..30}; do | ||
| if npm view agent-relay@${{ inputs.version }} version 2>/dev/null; then | ||
| echo "Package found on npm registry" | ||
| break | ||
| fi | ||
| echo "Attempt $i: Package not yet available, waiting 10s..." | ||
| sleep 10 | ||
| done | ||
|
Comment on lines
+66
to
+73
Contributor
There was a problem hiding this comment. 🔴 npm propagation wait loop silently continues if package is never found The wait loop attempts 30 times to find the package, but if all attempts fail, it silently continues to the next step instead of failing the workflow. Click to expandHow it happensLines 66-73 loop but never fail: for i in {1..30}; do
if npm view agent-relay@${{ inputs.version }} version 2>/dev/null; then
echo "Package found on npm registry"
break
fi
echo "Attempt $i: Package not yet available, waiting 10s..."
sleep 10
doneAfter 30 failed attempts (5 minutes), execution continues to the install step which will then fail with a confusing error. ImpactThe verification will proceed against a non-existent package version, causing cryptic npm install failures instead of a clear "package not found after timeout" error. Recommendation: Add a flag to track if the package was found, and exit 1 after the loop if not found: 
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| # Test 1: Global npm install | ||
| - name: "Test: Global npm install" | ||
| run: | | ||
| echo "Installing ${{ steps.pkg.outputs.spec }} globally..." | ||
| npm install -g ${{ steps.pkg.outputs.spec }} | ||
| # Add npm global bin to PATH | ||
| echo "$(npm config get prefix)/bin" >> $GITHUB_PATH | ||
|
|
||
| - name: "Test: Global --version" | ||
| run: | | ||
| # Ensure npm global bin is in PATH | ||
| export PATH="$(npm config get prefix)/bin:$PATH" | ||
| VERSION=$(agent-relay --version) | ||
| echo "Version output: $VERSION" | ||
| if echo "$VERSION" | grep -qE '[0-9]+\.[0-9]+\.[0-9]+'; then | ||
| echo "Version check passed" | ||
| else | ||
| echo "Version check failed - no version number found" | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: "Test: Global -V flag" | ||
| run: | | ||
| export PATH="$(npm config get prefix)/bin:$PATH" | ||
| agent-relay -V | ||
|
|
||
| - name: "Test: Global version command" | ||
| run: | | ||
| export PATH="$(npm config get prefix)/bin:$PATH" | ||
| OUTPUT=$(agent-relay version) | ||
| echo "$OUTPUT" | ||
| if echo "$OUTPUT" | grep -qE '[0-9]+\.[0-9]+\.[0-9]+'; then | ||
| echo "Version command check passed" | ||
| else | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: "Test: Global --help" | ||
| run: | | ||
| export PATH="$(npm config get prefix)/bin:$PATH" | ||
| agent-relay --help | head -20 | ||
|
|
||
| - name: Cleanup global install | ||
| run: | | ||
| export PATH="$(npm config get prefix)/bin:$PATH" | ||
| npm uninstall -g agent-relay | ||
|
|
||
| # Test 2: npx execution | ||
| - name: "Test: npx --version" | ||
| run: | | ||
| # npx with @ syntax - downloads and runs in one command | ||
| VERSION=$(npx ${{ steps.pkg.outputs.spec }} -- --version 2>&1) | ||
| echo "npx version output: $VERSION" | ||
| if echo "$VERSION" | grep -qE '[0-9]+\.[0-9]+\.[0-9]+'; then | ||
| echo "npx version check passed" | ||
| else | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: "Test: npx --help" | ||
| run: npx ${{ steps.pkg.outputs.spec }} -- --help | head -20 | ||
|
|
||
| - name: "Test: npx version command" | ||
| run: | | ||
| OUTPUT=$(npx ${{ steps.pkg.outputs.spec }} -- version 2>&1) | ||
| echo "$OUTPUT" | ||
| if echo "$OUTPUT" | grep -qE '[0-9]+\.[0-9]+\.[0-9]+'; then | ||
| echo "npx version command check passed" | ||
| else | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Test 3: Local project install | ||
| - name: "Test: Local project install" | ||
| run: | | ||
| mkdir -p /tmp/test-project | ||
| cd /tmp/test-project | ||
| npm init -y | ||
| npm install ${{ steps.pkg.outputs.spec }} | ||
|
|
||
| - name: "Test: Local npx execution" | ||
| run: | | ||
| cd /tmp/test-project | ||
| VERSION=$(npx agent-relay --version) | ||
| echo "Local npx version: $VERSION" | ||
| if echo "$VERSION" | grep -qE '[0-9]+\.[0-9]+\.[0-9]+'; then | ||
| echo "Local npx check passed" | ||
| else | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: "Test: Local bin executable" | ||
| run: | | ||
| cd /tmp/test-project | ||
| if [ -x "./node_modules/.bin/agent-relay" ]; then | ||
| VERSION=$(./node_modules/.bin/agent-relay --version) | ||
| echo "Local bin version: $VERSION" | ||
| else | ||
| echo "Local bin not found or not executable" | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Cleanup | ||
| run: rm -rf /tmp/test-project | ||
|
|
||
| # Docker-based verification (more isolated) | ||
| verify-docker: | ||
| name: Verify Docker (Node ${{ matrix.node }}) | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| node: ["18", "20", "22"] | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Build verification image | ||
| run: | | ||
| VERSION="${{ inputs.version || 'latest' }}" | ||
| docker build \ | ||
| --build-arg NODE_VERSION=${{ matrix.node }} \ | ||
| --build-arg PACKAGE_VERSION=$VERSION \ | ||
| -t agent-relay-verify:node${{ matrix.node }} \ | ||
| -f scripts/post-publish-verify/Dockerfile \ | ||
| scripts/post-publish-verify/ | ||
|
|
||
| - name: Run verification | ||
| run: | | ||
| docker run --rm agent-relay-verify:node${{ matrix.node }} | ||
|
|
||
| summary: | ||
| name: Verification Summary | ||
| needs: [verify, verify-docker] | ||
| runs-on: ubuntu-latest | ||
| if: always() | ||
|
|
||
| steps: | ||
| - name: Summary | ||
| run: | | ||
| echo "## Post-Publish Verification Summary" >> $GITHUB_STEP_SUMMARY | ||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| echo "**Package**: \`agent-relay@${{ inputs.version || 'latest' }}\`" >> $GITHUB_STEP_SUMMARY | ||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| echo "### Native Tests" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node Version | Status |" >> $GITHUB_STEP_SUMMARY | ||
| echo "|--------------|--------|" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 18 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 20 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 22 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
|
Comment on lines
+223
to
+225
Contributor
There was a problem hiding this comment. 🟡 Workflow summary displays same status for all Node versions instead of individual results The verification summary in Click to expandIssueThe summary section (lines 212-214 and 219-221) shows three separate rows for Node 18, 20, and 22, but all rows display the same value from echo "| Node 18 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Node 20 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Node 22 | ${{ needs.verify.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARYIn GitHub Actions, Actual vs Expected
ImpactThis is misleading to users reviewing the workflow summary - they may think all versions failed when only one did, or vice versa. The overall pass/fail logic is correct, but the granular display is inaccurate. Recommendation: Either (1) remove the per-version table and just show aggregate pass/fail, (2) add a note clarifying this is aggregate status, or (3) use job outputs to capture individual matrix results for accurate per-version reporting.
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| echo "### Docker Tests" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node Version | Status |" >> $GITHUB_STEP_SUMMARY | ||
| echo "|--------------|--------|" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 18 | ${{ needs.verify-docker.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 20 | ${{ needs.verify-docker.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
| echo "| Node 22 | ${{ needs.verify-docker.result == 'success' && '✅' || '❌' }} |" >> $GITHUB_STEP_SUMMARY | ||
| echo "" >> $GITHUB_STEP_SUMMARY | ||
| echo "### Overall" >> $GITHUB_STEP_SUMMARY | ||
| if [ "${{ needs.verify.result }}" = "success" ] && [ "${{ needs.verify-docker.result }}" = "success" ]; then | ||
| echo "✅ All verification tests passed!" >> $GITHUB_STEP_SUMMARY | ||
| else | ||
| echo "❌ Some verification tests failed" >> $GITHUB_STEP_SUMMARY | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Wait step runs with empty version on pull_request trigger causing invalid npm command
The condition
if: inputs.version != 'latest'evaluates to true wheninputs.versionis empty (which happens onpull_requesttrigger), causing the npm view command to run with an empty version.Click to expand
How it happens
On
pull_requesttrigger (lines 12-15), no inputs are provided, soinputs.versionis empty/null. The condition at line 63:evaluates to
truebecause empty string != 'latest'.This causes line 67 to execute:
Which becomes
npm view agent-relay@ version- an invalid command.Impact
The workflow will fail with a confusing npm error when triggered on PRs that modify the workflow files.
Recommendation: Change the condition to also check for empty:
if: inputs.version && inputs.version != 'latest'or use the resolved VERSION variable from the previous step.Was this helpful? React with 👍 or 👎 to provide feedback.