Skip to content

CI: Nightly matrix failure reporter (artifacts + workflow_run aggregator) #74

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 12, 2025
Merged

CI: Nightly matrix failure reporter (artifacts + workflow_run aggregator) #74

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3aac03e
ci: add TRX logging and failures.json to matrix jobs; set run-name fo…
jbrinkman Sep 9, 2025
1d52050
CI: add reporter workflow for nightly and manual full-matrix failures…
jbrinkman Sep 9, 2025
19f8cf8
CI: fix container failures.json generation by replacing heredoc with …
jbrinkman Sep 9, 2025
e871238
CI: add failure reporter for matrix-errors (#72)
jbrinkman Sep 9, 2025
f76d1c2
ci: reduce matrix for testing
jbrinkman Sep 9, 2025
7900aef
reporter: list only failing legs; parse TRX via xmlstarlet; restore j…
jbrinkman Sep 9, 2025
6ac11b2
Revert "ci: reduce matrix for testing"
jbrinkman Sep 9, 2025
bd1c02b
CI: add inputs.json artifact to tests and gate reporter on full-matri…
jbrinkman Sep 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
208 changes: 208 additions & 0 deletions .github/workflows/report-failures.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
name: C# failure reporter

on:
workflow_run:
workflows: ["C# tests"]
types: [completed]
branches: [jbrinkman/matrix-errors]

permissions:
actions: read
contents: read
issues: write

jobs:
report:
# Only run this job when the triggering run failed and was schedule or workflow_dispatch
if: ${{ github.event.workflow_run.conclusion == 'failure' && (github.event.workflow_run.event == 'schedule' || github.event.workflow_run.event == 'workflow_dispatch') }}
runs-on: ubuntu-latest
steps:
- name: Download inputs.json artifact (if present)
id: inputs
uses: dawidd6/action-download-artifact@v6
continue-on-error: true
with:
run_id: ${{ github.event.workflow_run.id }}
name: inputs.json
path: inputs_artifact

- name: Gate on full-matrix runs
id: gate
uses: actions/github-script@v7
with:
github-token: ${{ github.token }}
script: |
const {owner, repo} = context.repo;
const runId = context.payload.workflow_run.id;
const event = context.payload.workflow_run.event;
const { data: run } = await github.rest.actions.getWorkflowRun({ owner, repo, run_id: runId });
const title = run.display_title || run.name || '';
// Prefer inputs.json when workflow_dispatch
let should = false;
if (event === 'schedule') {
should = true;
} else if (event === 'workflow_dispatch') {
try {
const fs = require('fs');
const path = 'inputs_artifact/inputs.json';
if (fs.existsSync(path)) {
const data = JSON.parse(fs.readFileSync(path, 'utf8'));
const fm = !!(data.inputs && data.inputs['full-matrix']);
should = fm;
core.info(`Gate via inputs.json: full-matrix=${fm}`);
} else {
core.info('inputs.json not found; falling back to title gate');
should = title.startsWith('C# Matrix Tests');
}
} catch (e) {
core.warning(`Failed to read inputs.json: ${e}`);
should = title.startsWith('C# Matrix Tests');
}
}
core.info(`Trigger event: ${event}; run title: ${title}; should process: ${should}`);
return should;

- name: Download artifacts from triggering run
uses: dawidd6/action-download-artifact@v6
if: ${{ steps.gate.outputs.result == 'true' }}
with:
run_id: ${{ github.event.workflow_run.id }}
path: downloaded

- name: Aggregate failures
id: aggregate
shell: bash
if: ${{ steps.gate.outputs.result == 'true' }}
run: |
set -euo pipefail
sudo apt-get update -y >/dev/null 2>&1 || true
sudo apt-get install -y jq xmlstarlet >/dev/null 2>&1 || true
shopt -s nullglob

total_failed=0
total_legs=0
failing_legs=0
md='<!-- run:${{ github.event.workflow_run.id }} -->
'
md+="# Nightly CI failures for ${{ github.repository }}
"$'\n'
md+="Triggered: ${{ github.event.workflow_run.run_started_at }} • [Run link](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }})
"$'\n\n'
md_details=""

# Iterate each artifact folder downloaded
for dir in downloaded/*; do
[ -d "$dir" ] || continue
base="$(basename "$dir")"
case "$base" in
test-reports-*) ;;
*) continue ;;
esac
total_legs=$((total_legs+1))
job_display="$base"

# Prefer failures.json
fjson="$dir/failures.json"
failed=0; total=0
if [ -f "$fjson" ]; then
failed=$(jq -r '.summary.failed // 0' "$fjson" 2>/dev/null || echo 0)
total=$(jq -r '.summary.total // 0' "$fjson" 2>/dev/null || echo 0)
fi

# Fallback to TRX parse if counts are zero
if [ "$failed" = "0" ] && [ "$total" = "0" ]; then
trx=$(ls "$dir"/*.trx 2>/dev/null | head -n1 || true)
if [ -n "$trx" ]; then
total=$(xmlstarlet sel -T -t -v 'string(//*[local-name()="Counters"]/@total)' "$trx" 2>/dev/null || echo 0)
failed=$(xmlstarlet sel -T -t -v 'string(//*[local-name()="Counters"]/@failed)' "$trx" 2>/dev/null || echo 0)
fi
fi

# Emit only failing legs with per-test bullets
if [ "$failed" != "0" ]; then
failing_legs=$((failing_legs+1))
total_failed=$((total_failed+failed))
md+="- ${job_display}: ${failed}/${total} failed\n"
# Try to get failed tests either from failures.json or directly from TRX
bullets=""
if [ -f "$fjson" ]; then
bullets=$(jq -r '.failed[]? | " - " + (.displayName // .fullyQualifiedName // .name // "(unknown)") + (if .errorMessage then " — " + .errorMessage else (if .message then " — " + .message else "" end) end)' "$fjson" 2>/dev/null || true)
fi
if [ -z "$bullets" ]; then
if [ -n "${trx:-}" ] && [ -f "$trx" ]; then
while IFS= read -r line; do
name="${line%%||*}"
msg="${line#*||}"
# Trim to first line and limit length
msg="${msg%%$'\n'*}"
if [ ${#msg} -gt 200 ]; then msg="${msg:0:197}..."; fi
echo " - ${name} — ${msg}"
done < <(xmlstarlet sel -T -t -m '//*[local-name()="UnitTestResult" and @outcome="Failed"]' -v 'concat(@testName,"||", normalize-space(.//*[local-name()="Message"]))' -n "$trx" 2>/dev/null || true) > "$dir/.bullets.txt"
bullets=$(cat "$dir/.bullets.txt" 2>/dev/null || true)
fi
fi
if [ -n "$bullets" ]; then
md_details+=$'\n'
md_details+="### ${job_display}\n\n"
md_details+="$bullets\n"
fi
fi
done

if [ "$failing_legs" = "0" ]; then
md+="The pipeline failed, but no individual test failures were found. This may indicate a job or infrastructure failure.\n"
fi
md+=$'\n'
md+="Total failed tests across ${failing_legs} of ${total_legs} matrix jobs: ${total_failed}\n\n"
md+="$md_details"

printf "%b" "$md" > summary.md
echo "summary_path=$(pwd)/summary.md" >> "$GITHUB_OUTPUT"
echo "total_failed=${total_failed}" >> "$GITHUB_OUTPUT"
echo "total_legs=${total_legs}" >> "$GITHUB_OUTPUT"

- name: Find or create rolling failure issue
id: ensure_issue
uses: actions/github-script@v7
if: ${{ steps.gate.outputs.result == 'true' }}
env:
SUMMARY_PATH: ${{ steps.aggregate.outputs.summary_path }}
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const fs = require('fs');
const body = fs.readFileSync(process.env.SUMMARY_PATH, 'utf8');
const marker = '<!-- run:${{ github.event.workflow_run.id }} -->';
const {owner, repo} = context.repo;
const label = 'ci-failure';
const title = 'Nightly CI failures: C# tests';

// Ensure label exists? Assume it's created manually as per repo policy.
const {data: issues} = await github.rest.issues.listForRepo({
owner, repo, state: 'open', labels: label, per_page: 100
});
let issue = issues.find(i => i.title === title);

if (!issue) {
const created = await github.rest.issues.create({
owner, repo, title, body, labels: [label]
});
core.setOutput('issue_number', String(created.data.number));
core.info(`Created issue #${created.data.number}`);
return;
}

// Avoid duplicate comments for the same run using marker
const {data: comments} = await github.rest.issues.listComments({
owner, repo, issue_number: issue.number, per_page: 100
});
const exists = comments.some(c => c.body && c.body.includes(marker));
if (!exists) {
await github.rest.issues.createComment({
owner, repo, issue_number: issue.number, body
});
core.info(`Commented on issue #${issue.number}`);
} else {
core.info('A comment for this run already exists; skipping.');
}
core.setOutput('issue_number', String(issue.number));
97 changes: 92 additions & 5 deletions .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ concurrency:
cancel-in-progress: true

run-name:
# Set custom name if job is started manually and name is given
${{ github.event_name == 'workflow_dispatch' && (inputs.name == '' && format('{0} @ {1} {2}', github.ref_name, github.sha, toJson(inputs)) || inputs.name) || '' }}
# Use a distinct name for full-matrix or scheduled runs; otherwise keep manual-dispatch naming behavior
${{ (github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && github.event.inputs['full-matrix'] == 'true')) && 'C# Matrix Tests' || (github.event_name == 'workflow_dispatch' && (inputs.name == '' && format('{0} @ {1} {2}', github.ref_name, github.sha, toJson(inputs)) || inputs.name)) || '' }}

env:
CARGO_TERM_COLOR: always
Expand All @@ -45,6 +45,33 @@ jobs:
host-matrix-output: ${{ steps.get-matrices.outputs.os-matrix-output }}
version-matrix-output: ${{ steps.get-matrices.outputs.version-matrix-output }}
steps:
- name: Write inputs.json
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
fs.mkdirSync('.run-meta', {recursive: true});
const event = context.eventName;
const out = {event};
if (event === 'workflow_dispatch') {
const inputs = context.payload.inputs || {};
const raw = inputs['full-matrix'];
const fm = (raw === true) || (String(raw).toLowerCase() === 'true');
out.inputs = {
'full-matrix': fm,
name: inputs.name || ''
};
}
fs.writeFileSync('.run-meta/inputs.json', JSON.stringify(out));
core.info(`inputs.json => ${JSON.stringify(out)}`);

- name: Upload inputs.json
uses: actions/upload-artifact@v4
with:
name: inputs.json
path: .run-meta/inputs.json
if-no-files-found: error

- uses: actions/checkout@v5
- id: get-matrices
uses: ./.github/workflows/create-test-matrices
Expand Down Expand Up @@ -98,7 +125,35 @@ jobs:
key: rust-${{ matrix.host.TARGET }}

- name: Test dotnet ${{ matrix.dotnet }}
run: dotnet test --configuration Debug --framework net${{ matrix.dotnet }} --logger "html;LogFileName=TestReport.html" --logger "console;verbosity=detailed" --results-directory .
run: dotnet test --configuration Debug --framework net${{ matrix.dotnet }} --logger "html;LogFileName=TestReport.html" --logger "trx;LogFileName=TestResults.trx" --logger "console;verbosity=detailed" --results-directory .

- name: Derive failures.json (from TRX)
if: always()
shell: bash
run: |
set -euo pipefail
SUMMARY_FILE=failures.json
if command -v xmllint >/dev/null 2>&1 && command -v jq >/dev/null 2>&1; then
FAILED_COUNT=$(xmllint --xpath "string(//Counters/@failed)" TestResults.trx 2>/dev/null || echo 0)
PASSED_COUNT=$(xmllint --xpath "string(//Counters/@passed)" TestResults.trx 2>/dev/null || echo 0)
TOTAL_COUNT=$(xmllint --xpath "string(//Counters/@total)" TestResults.trx 2>/dev/null || echo 0)
SKIPPED_COUNT=$(xmllint --xpath "string(//Counters/@skipped)" TestResults.trx 2>/dev/null || echo 0)
echo "{" > "$SUMMARY_FILE"
echo " \"workflow\": \"C# tests\"," >> "$SUMMARY_FILE"
echo " \"runId\": ${GITHUB_RUN_ID}," >> "$SUMMARY_FILE"
echo " \"jobName\": \"net${{ matrix.dotnet }}, server ${{ matrix.server.version }}, ${{ matrix.host.TARGET }}\"," >> "$SUMMARY_FILE"
echo " \"matrix\": {" >> "$SUMMARY_FILE"
echo " \"dotnet\": \"${{ matrix.dotnet }}\"," >> "$SUMMARY_FILE"
echo " \"server\": { \"type\": \"${{ matrix.server.type }}\", \"version\": \"${{ matrix.server.version }}\" }," >> "$SUMMARY_FILE"
echo " \"host\": { \"OS\": \"${{ matrix.host.OS }}\", \"ARCH\": \"${{ matrix.host.ARCH }}\", \"RUNNER\": \"${{ matrix.host.RUNNER }}\" }" >> "$SUMMARY_FILE"
echo " }," >> "$SUMMARY_FILE"
echo " \"summary\": { \"total\": $TOTAL_COUNT, \"passed\": $PASSED_COUNT, \"failed\": $FAILED_COUNT, \"skipped\": $SKIPPED_COUNT }," >> "$SUMMARY_FILE"
echo " \"failed\": []," >> "$SUMMARY_FILE"
echo " \"links\": { \"runUrl\": \"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\" }" >> "$SUMMARY_FILE"
echo "}" >> "$SUMMARY_FILE"
else
echo '{"workflow":"C# tests","summary":{"note":"Install xmllint+jq for rich failures"}}' > "$SUMMARY_FILE"
fi

- name: Run benchmark
if: ${{ contains(matrix.host.RUNNER, 'ubuntu') }}
Expand All @@ -120,6 +175,8 @@ jobs:
name: test-reports-dotnet-${{ matrix.dotnet }}-${{ matrix.server.type }}-${{ matrix.server.version }}-${{ matrix.host.OS }}-${{ matrix.host.ARCH }}
path: |
TestReport.html
TestResults.trx
failures.json
valkey-glide/benchmarks/results/*
valkey-glide/utils/clusters/**

Expand Down Expand Up @@ -187,16 +244,46 @@ jobs:
key: rust-${{ matrix.host.IMAGE }}

- name: Test dotnet ${{ matrix.dotnet }}
run: dotnet test --framework net${{ matrix.dotnet }} --logger "html;LogFileName=TestReport.html" --logger "console;verbosity=detailed" --results-directory .
run: dotnet test --framework net${{ matrix.dotnet }} --logger "html;LogFileName=TestReport.html" --logger "trx;LogFileName=TestResults.trx" --logger "console;verbosity=detailed" --results-directory .

- name: Derive failures.json (from TRX)
if: always()
shell: bash
run: |
set -euo pipefail
SUMMARY_FILE=failures.json
if command -v xmllint >/dev/null 2>&1 && command -v jq >/dev/null 2>&1; then
FAILED_COUNT=$(xmllint --xpath "string(//Counters/@failed)" TestResults.trx 2>/dev/null || echo 0)
PASSED_COUNT=$(xmllint --xpath "string(//Counters/@passed)" TestResults.trx 2>/dev/null || echo 0)
TOTAL_COUNT=$(xmllint --xpath "string(//Counters/@total)" TestResults.trx 2>/dev/null || echo 0)
SKIPPED_COUNT=$(xmllint --xpath "string(//Counters/@skipped)" TestResults.trx 2>/dev/null || echo 0)
echo "{" > "$SUMMARY_FILE"
echo " \"workflow\": \"C# tests\"," >> "$SUMMARY_FILE"
echo " \"runId\": ${GITHUB_RUN_ID}," >> "$SUMMARY_FILE"
echo " \"jobName\": \"net${{ matrix.dotnet }}, server ${{ matrix.server.version }}, ${{ matrix.host.TARGET }}\"," >> "$SUMMARY_FILE"
echo " \"matrix\": {" >> "$SUMMARY_FILE"
echo " \"dotnet\": \"${{ matrix.dotnet }}\"," >> "$SUMMARY_FILE"
echo " \"server\": { \"type\": \"${{ matrix.server.type }}\", \"version\": \"${{ matrix.server.version }}\" }," >> "$SUMMARY_FILE"
echo " \"host\": { \"OS\": \"${{ matrix.host.OS }}\", \"ARCH\": \"${{ matrix.host.ARCH }}\", \"RUNNER\": \"${{ matrix.host.RUNNER }}\" }" >> "$SUMMARY_FILE"
echo " }," >> "$SUMMARY_FILE"
echo " \"summary\": { \"total\": $TOTAL_COUNT, \"passed\": $PASSED_COUNT, \"failed\": $FAILED_COUNT, \"skipped\": $SKIPPED_COUNT }," >> "$SUMMARY_FILE"
echo " \"failed\": []," >> "$SUMMARY_FILE"
echo " \"links\": { \"runUrl\": \"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\" }" >> "$SUMMARY_FILE"
echo "}" >> "$SUMMARY_FILE"
else
echo '{"workflow":"C# tests","summary":{"note":"Install xmllint+jq for rich failures"}}' > "$SUMMARY_FILE"
fi

- name: Upload test reports
if: always()
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: test-reports-dotnet-${{ matrix.dotnet }}-${{ matrix.server.type }}-${{ matrix.server.version }}-${{ env.IMAGE }}-${{ matrix.host.ARCH }}
name: test-reports-dotnet-${{ matrix.dotnet }}-${{ matrix.server.type }}-${{ matrix.server.version }}-${{ matrix.host.IMAGE }}-${{ matrix.host.ARCH }}
path: |
TestReport.html
TestResults.trx
failures.json
valkey-glide/utils/clusters/**

lint:
Expand Down
Loading