Skip to content

ci: add nightly smoke tests for cross-OS download validation#2

Merged
naorpeled merged 3 commits into
mainfrom
feat/nightly-smoke-tests
Mar 18, 2026
Merged

ci: add nightly smoke tests for cross-OS download validation#2
naorpeled merged 3 commits into
mainfrom
feat/nightly-smoke-tests

Conversation

@naorpeled
Copy link
Copy Markdown
Owner

Summary

  • Adds a nightly CI workflow that downloads the latest GitHub Release binary on Linux, macOS, and Windows, runs it, and asserts the welcome screen renders correctly
  • Also tests the npm install -g @aitutor/cli path on all three platforms
  • Uses script for PTY capture on Linux/macOS; verifies no-crash on Windows
  • Triggers daily at 6am UTC and via workflow_dispatch

Test plan

  • Trigger workflow manually via workflow_dispatch and verify all 3 OS jobs pass
  • Confirm TUI output capture works (check logs for "All checks passed")
  • Confirm npm install path works on all platforms

Runs daily at 6am UTC to verify GitHub Release binaries and npm
package install correctly and render the welcome screen on
Linux, macOS, and Windows.
Copilot AI review requested due to automatic review settings March 18, 2026 22:03
@qodo-code-review
Copy link
Copy Markdown

Review Summary by Qodo

Add nightly smoke tests for cross-OS binary validation

🧪 Tests

Grey Divider

Walkthroughs

Description
• Adds nightly CI workflow for cross-OS smoke testing
• Tests GitHub Release binary downloads on Linux, macOS, Windows
• Validates welcome screen renders with expected content
• Tests npm package installation path on all platforms
Diagram
flowchart LR
  A["Nightly Schedule<br/>or Manual Trigger"] --> B["Get Latest<br/>Release Tag"]
  B --> C["Map OS/Arch<br/>to Archive Name"]
  C --> D["Download<br/>Release Archive"]
  D --> E["Extract Binary"]
  E --> F["Run Smoke Tests"]
  F --> G["Test npm<br/>Installation"]
  G --> H["Verify Output<br/>on All Platforms"]
Loading

Grey Divider

File Changes

1. .github/workflows/smoke.yml 🧪 Tests +157/-0

New nightly smoke test workflow for releases

• Creates new nightly CI workflow triggered daily at 6am UTC and via manual dispatch
• Implements matrix strategy to test on Ubuntu, macOS, and Windows runners
• Downloads latest GitHub Release binary and validates it runs without crashing
• Captures and validates TUI output contains expected content (AITutor, difficulty levels,
 description)
• Tests npm global installation path and verifies binary functionality
• Uses platform-specific extraction and testing methods (tar/zip, script/pwsh)

.github/workflows/smoke.yml


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Copy Markdown

qodo-code-review Bot commented Mar 18, 2026

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 📐 Spec deviations (0)

Grey Divider


Action required

1. macOS timeout missing🐞 Bug ⛯ Reliability
Description
The macOS smoke test steps invoke timeout but the workflow never installs/provides it, so the
macOS jobs can fail with “command not found” (or lose the intended kill-switch and hang). This makes
the nightly smoke workflow unreliable on macOS.
Code

.github/workflows/smoke.yml[85]

+          timeout 5 script -q output.txt ./aitutor || true
Evidence
Both macOS smoke steps wrap script with timeout, but there is no step that installs coreutils or
otherwise ensures a timeout implementation is available in the environment.

.github/workflows/smoke.yml[81-86]
.github/workflows/smoke.yml[129-136]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
macOS steps use `timeout` but the workflow does not ensure it exists on macOS runners, causing flaky/failed macOS smoke runs.
### Issue Context
Linux can rely on `timeout`, but macOS often needs either coreutils (`gtimeout`) or a small portable timeout wrapper.
### Fix Focus Areas
- .github/workflows/smoke.yml[81-96]
- .github/workflows/smoke.yml[129-140]
### Suggested fix
Choose one:
1) Install coreutils on macOS and use `gtimeout` in macOS steps.
2) Replace `timeout ...` in macOS steps with a portable timeout wrapper (e.g., `python3`/`perl`-based) so the workflow doesn’t depend on a preinstalled `timeout` binary.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Windows unzip inconsistency🐞 Bug ⚙ Maintainability
Description
The Windows extraction step uses unzip, which is inconsistent with the repo’s own Windows
installer logic that extracts zip assets via tar -xf. Aligning on one extraction approach reduces
portability surprises and keeps CI behavior consistent with the supported installation path.
Code

.github/workflows/smoke.yml[R61-63]

+      - name: Extract binary (Windows)
+        if: runner.os == 'Windows'
+        run: unzip ${{ steps.archive.outputs.name }}
Evidence
The workflow extracts Windows archives with unzip, but the in-repo installer extracts the Windows
zip using tar -xf, indicating tar is the project’s established approach for Windows zip
extraction.

.github/workflows/smoke.yml[61-63]
install.js[76-81]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Windows extraction uses `unzip`, which diverges from the repo’s own installer (`install.js`) that uses `tar -xf` for zip extraction.
### Issue Context
Keeping the smoke workflow’s extraction method consistent with the supported installer reduces environment/tooling assumptions and makes failures easier to reason about.
### Fix Focus Areas
- .github/workflows/smoke.yml[61-64]
### Suggested fix
Update the Windows extraction step to use the same approach as `install.js` (e.g., `tar -xf &amp;lt;archive&amp;gt;` in PowerShell) or use PowerShell-native `Expand-Archive`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Missing OS/arch defaults🐞 Bug ⛯ Reliability
Description
The archive-name mapping step has no default/error branches, so unexpected RUNNER_OS/RUNNER_ARCH
values can leave OS/ARCH unset and produce a broken asset name, leading to confusing download
failures. Failing fast with explicit *) cases would make the workflow more robust.
Code

.github/workflows/smoke.yml[R34-42]

+          case "$RUNNER_OS" in
+            Linux)  OS=linux  ;;
+            macOS)  OS=darwin ;;
+            Windows) OS=windows ;;
+          esac
+          case "$RUNNER_ARCH" in
+            X64)   ARCH=amd64 ;;
+            ARM64) ARCH=arm64 ;;
+          esac
Evidence
The workflow’s case statements don’t handle unknown OS/arch values, while the repo’s installer
explicitly errors on unsupported platform/arch inputs—suggesting the intended behavior is to
validate and fail clearly.

.github/workflows/smoke.yml[34-42]
install.js[24-33]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`RUNNER_OS` and `RUNNER_ARCH` are mapped via `case` statements without `*)` defaults; if values change, the workflow will compute an invalid archive name and fail later during download.
### Issue Context
The installer (`install.js`) already validates platform/arch and throws a clear error for unsupported combinations.
### Fix Focus Areas
- .github/workflows/smoke.yml[30-50]
### Suggested fix
Add `set -euo pipefail` and `*)` branches for both OS and ARCH cases that print the unexpected value and `exit 1`, so misconfigurations fail immediately with a clear message.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

ⓘ The new review experience is currently in Beta. Learn more

Grey Divider

Qodo Logo

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new scheduled GitHub Actions workflow to run nightly cross-OS smoke tests validating both the latest GitHub Release binary and the npm install -g @aitutor/cli installation path.

Changes:

  • Introduces a nightly (cron) + manual (workflow_dispatch) “Smoke Tests” workflow.
  • Downloads and runs the latest GitHub Release artifact on Linux/macOS/Windows with basic output/no-crash validation.
  • Installs the CLI via npm on all three platforms and performs a similar smoke check.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml
Comment thread .github/workflows/smoke.yml Outdated
Comment thread .github/workflows/smoke.yml Outdated
- Replace macOS `timeout` with `perl -e 'alarm 5; exec @argv'` (timeout
  is not available on macOS by default)
- Use PowerShell `Expand-Archive` for Windows zip extraction instead of
  `unzip`
- Add default `*)` cases with error messages for OS/arch mapping
- Add `set -euo pipefail` to the mapping step
BSD sed (macOS) does not support \x1b hex escapes, so ANSI code
stripping would silently fail. Switch all sed calls to perl -pe
which handles \e portably on both Linux and macOS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants