Skip to content

Add weekly scheduled workflow to test all features with retry and issue creation#167

Merged
sebst merged 2 commits intomainfrom
copilot/add-github-action-workflow-test
Mar 22, 2026
Merged

Add weekly scheduled workflow to test all features with retry and issue creation#167
sebst merged 2 commits intomainfrom
copilot/add-github-action-workflow-test

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 22, 2026

Features can silently break due to upstream changes. A periodic test workflow is needed to catch these regressions automatically, retry transient failures, and surface persistent ones as tracked issues.

New workflow: .github/workflows/scheduled-test.yml

  • Trigger: Weekly cron (0 0 * * 0) + workflow_dispatch
  • Feature discovery: get-all-features job dynamically lists all dirs under src/ — no hardcoding
  • Test jobs (mirror structure of test.yaml):
    • test-autogenerated — matrix of feature × baseImage (debian, ubuntu, mcr devcontainers base); runs devcontainer features test --skip-scenarios
    • test-scenarios — matrix of feature; runs --skip-autogenerated --skip-duplicated
  • Retry logic: Each matrix job internally retries up to 3× with 30s back-off before failing. Because each feature is an isolated matrix job with fail-fast: false, only failing features are retried — passing ones are never re-run
  • Failure artifacts: On exhausted retries, the full attempt log is uploaded as a failure-* artifact
  • Issue creation: create-issue job runs if: always(), downloads all failure-* artifacts, and opens one GitHub issue with a collapsible per-feature log section:
### `failure-autogen-ripgrep-ubuntu-latest`
<details><summary>Expand log</summary>

=== Attempt 1 of 3 ===
...
=== Attempt 3 of 3 ===
All 3 attempts failed

</details>
  • issues: write permission scoped to create-issue only; all other jobs run with contents: read
  • max-parallel: 10 to avoid saturating runners across 150+ potential concurrent matrix jobs
  • Uses ${{ runner.temp }} for log/artifact paths and ${{ github.run_started_at }} for issue titles
Original prompt

This section details on the original issue you should resolve

<issue_title>Create a github action workflow to test features periodically</issue_title>
<issue_description>We need to test if feature installation break due to changes in the upstream.

Create a github actions workflow that runs once a week to test all features (installation and runtime tests).

If the workflow fails, it should automatically retry the failed steps up to 3 times. This is because we might run into rate limits or network issues during installation. Make sure ONLY the failed features are retried, not the entire workflow.

If the workflow fails even after retry, an issue should be created with log output (per feature)</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

@sebst sebst marked this pull request as ready for review March 22, 2026 11:56
Copilot AI changed the title [WIP] Add GitHub action workflow to test features periodically Add weekly scheduled workflow to test all features with retry and issue creation Mar 22, 2026
Copilot AI requested a review from sebst March 22, 2026 11:56
@sebst sebst merged commit 21ea5f9 into main Mar 22, 2026
@sebst sebst deleted the copilot/add-github-action-workflow-test branch March 22, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a github action workflow to test features periodically

2 participants