diff --git a/.claude/skills/actions-ci/SKILL.md b/.claude/skills/actions-ci/SKILL.md new file mode 100644 index 0000000..2913613 --- /dev/null +++ b/.claude/skills/actions-ci/SKILL.md @@ -0,0 +1,87 @@ +--- +name: actions-ci +description: Author and debug GitHub Actions workflows, automated tests, and CI pipelines the CEDA way. Covers the reusable-workflow orchestration pattern, the uv + matrix test setup, security scanning (bandit/semgrep/codeql/sonarcloud), and release/publish. Use when adding or fixing a workflow, test job, or pipeline in a GitHub repo. +--- + +# actions-ci + +CEDA GitHub repos use uv-based Python CI. There are **two generations** — match +the one the repo already uses rather than mixing them. + +## When this applies + +This is a **knowledge skill** — it loads (explicitly via `/actions-ci`, or +automatically) when you add or fix a GitHub Actions workflow, a test job, a +coverage/security-scan step, or a release workflow in a CEDA GitHub repo. It is +reference/convention, not a step-by-step procedure. For the GitLab config repos, +use /gitlab-ci instead. + +## Which generation is this repo on? +- **Modern / orchestrated** (e.g. `1cijferho`): a thin `ci.yml` that calls + reusable workflows via `uses: ./.github/workflows/.yml` + `workflow_call`. + Uses `ruff` (blocking), `uv sync --group dev`, and a security suite. Prefer this + for new/serious repos. +- **Legacy quartet** (e.g. `eencijfer`, `minio-file`, `sdp-tools`): + `dev.yml` / `preview.yml` / `release.yml` / `test.yml`, `uv pip install -e ".[dev,test]"`, + flake8/black/isort/mypy run **non-blocking** (`|| true`), Codecov upload. + Extend in place; don't rewrite to the modern style unless asked. + +## Conventions that hold across both +- **uv** for everything: `astral-sh/setup-uv`, `actions/setup-python@v5`, + Python **3.13** on modern repos (matrix 3.9–3.12 + multi-OS on legacy). +- Triggers: `push`/`pull_request` on `main` (modern also scans `feat/**`,`fix/**`). +- `actions/checkout@v4`; `fetch-depth: 0` when a tool needs full history (Sonar). +- **Windows in the matrix** matters here — past pain with `ErrorActionPreference` + and Scoop writing to **stderr** (treat stderr as non-fatal unless exit code≠0). +- Env `PYTHONUTF8: "1"` on Windows jobs. + +## Reusable workflow pattern (modern) +- A leaf workflow declares `on: workflow_call:` and does one thing (unit tests, + a MinIO integration run, a pipeline config). +- `ci.yml` wires them as jobs with `uses: ./.github/workflows/.yml`. +- Add a new check = new leaf workflow + one `uses:` line in `ci.yml`. +- A workflow under `tests/eencijferho/` is auto-collected by the unit job — no CI + edit needed to add a test file there. + +## Tests & coverage +- `uv run pytest tests/... --cov= --cov-report=xml` (cov target is + the **package import name**, not a filesystem path). +- Integration tests needing services: start MinIO via `docker run`, Postgres via + a `services:` container; install extras with `uv sync --group dev --extra minio --extra postgres`. +- Merge unit+integration coverage with `--cov-append` before uploading to Sonar/Codecov. +- See /gate for reading a failing SonarCloud/CodeQL gate. + +## Security scanning (modern repos) +- **bandit** (`bandit[sarif]`, `--severity-level medium`), **semgrep**, **codeql**, + **sonarcloud** — each uploads SARIF via `github/codeql-action/upload-sarif@v3` + with a distinct `category`. Dependabot for deps. +- Scan steps are typically non-blocking on findings but surface results in the + Security tab. + +## Release / publish (legacy quartet) +- `release.yml` triggers on `v*` tags → PyPI **trusted publishing** + (`permissions: id-token: write`, `environment: pypi`). No stored token. + +## Debugging CI +- `unset GITHUB_TOKEN && gh run list/view/watch` to inspect runs (an invalid + `$GITHUB_TOKEN` can shadow the `gh` login, so prefix `gh` with `unset GITHUB_TOKEN &&`). +- Reproduce failures locally where possible before pushing fix-attempts — avoid + guess-and-check CI loops. Verify a fix actually resolves the failure before + calling it done. +- Heavy Polars tests can segfault under local x86/Rosetta emulation — that's the + dev machine, not CI (native runners are fine). + +## GitLab note +The `*-config` GitOps repos use **GitLab CI** (`.gitlab-ci.yml`, reusable +`.gitlab/*.yaml` with `spec.inputs`), not Actions — see /surf-sdp-helm-flux and /gitlab-ci. +Don't apply Actions patterns there. + +## Important +- **Match the repo's existing generation** (modern orchestrated vs legacy + quartet); don't rewrite one style into the other unless asked. +- This skill covers **GitHub Actions only** — it does NOT apply to the GitLab + config repos (use /gitlab-ci there). +- Reproduce CI failures locally where possible; avoid guess-and-check loops. +- **gh auth**: prefix `gh` with `unset GITHUB_TOKEN &&` (an invalid token may + shadow the login). +- Applies to cedanl repos. diff --git a/.claude/skills/branch-pr/SKILL.md b/.claude/skills/branch-pr/SKILL.md new file mode 100644 index 0000000..cfc64c2 --- /dev/null +++ b/.claude/skills/branch-pr/SKILL.md @@ -0,0 +1,34 @@ +--- +name: branch-pr +description: Open a new PR or finalize an existing one — assess the diff vs main, draft a real title and body, fill in the repo's PR template instead of leaving it blank, and handle the gh auth quirk. Use when creating a PR or turning a WIP/draft into a review-ready PR. +--- + +# branch-pr + +Create or finalize a PR properly — no blank template bodies, no WIP left in the title. + +## Workflow + +When the user invokes `/branch-pr [optional: PR number]`: + +1. Understand the change set: + - `git log main..HEAD --oneline` and `git diff main...HEAD --stat` + - Check for an existing PR: `unset GITHUB_TOKEN && gh pr list --head {branch} --state all` +2. If a repo PR template exists (`.github/pull_request_template.md` or the body + GitHub pre-fills), FILL IT IN — don't submit the raw template. Cover: type of + change, description (bullet the real changes), testing instructions, deps. +3. Draft a concise title (<70 chars), no "WIP". Details go in the body. +4. Write the body to a temp file to avoid shell-quoting issues with apostrophes: + `gh pr create --base main --head {branch} --title "..." --body-file /tmp/pr_body.md` + or `gh pr edit {n} --title "..." --body-file /tmp/pr_body.md` to finalize. +5. Return the PR URL. + +## Important +- Dutch is fine for the body if the team uses it. +- Be honest in testing instructions about what's automated vs manual. +- Flag deployment caveats (e.g. "init.sql only runs on a fresh volume"). +- **Never submit the raw/blank PR template** and never leave "WIP" in the title. +- **gh auth**: prefix every `gh` command with `unset GITHUB_TOKEN &&` (an invalid + `$GITHUB_TOKEN` may shadow the login). If `gh auth status` fails, stop and ask + the user to re-auth rather than working around it. +- Applies to cedanl repos. diff --git a/.claude/skills/gate/SKILL.md b/.claude/skills/gate/SKILL.md new file mode 100644 index 0000000..71b6d46 --- /dev/null +++ b/.claude/skills/gate/SKILL.md @@ -0,0 +1,43 @@ +--- +name: gate +description: Decode and fix a failing SonarCloud or CodeQL quality gate on a PR. Pulls the actual issues via API, separates real defects from metric artifacts, fixes the cheap and legitimate ones, and reports remaining items as an explicit judgment call. Use when a Sonar/CodeQL gate is red. +--- + +# gate + +A failing quality gate usually mixes real issues with metric artifacts. Decode it +before touching code, and never game a metric silently. + +## Workflow + +When the user invokes `/gate [optional: PR number]`: + +1. Get the gate status and per-condition breakdown: + `curl -s "https://sonarcloud.io/api/qualitygates/project_status?projectKey={KEY}&pullRequest={N}"` +2. Pull the ACTUAL issues, don't reason from ratings alone: + - Bugs/vulns: `curl -s "https://sonarcloud.io/api/issues/search?componentKeys={KEY}&pullRequest={N}&types=BUG,VULNERABILITY&resolved=false"` + - Hotspots: `curl -s "https://sonarcloud.io/api/hotspots/search?projectKey={KEY}&pullRequest={N}&status=TO_REVIEW"` + - Smells: same issues endpoint with `types=CODE_SMELL` +3. Triage each into: real defect / metric artifact / false positive. Common artifacts: + - **Coverage 0%**: no report uploaded, or the code is genuinely untestable + (UI/frontend/JS). Fix by generating real coverage for testable code AND + excluding untestable code via `sonar.coverage.exclusions`. + - **Duplication**: often shared boilerplate — extract it. + - **JS "expected assignment/expression"**: a template placeholder Sonar parses + as raw JS. Make placeholders comments. + - **"prefer top-level await"**: false positive when code runs in an injected + async wrapper (e.g. streamlit-js). +4. Fix the cheap, legitimate ones. For coverage gaps in DB/integration code, add + integration tests with real service containers rather than excluding. +5. Push, wait for the rescan (`gh run watch`), re-query the gate to CONFIRM green. +6. Report any remaining red condition as an explicit judgment call to the user — + don't over-exclude to force a pass. + +## Important +- Distinguish "real security/correctness problem" from "metric hygiene"; report + the distinction honestly and fix what's genuinely worth fixing. +- **Never game a metric** (e.g. over-broad coverage exclusions) just to force a + green gate — report a remaining red condition as an explicit judgment call. +- **gh auth**: prefix `gh` with `unset GITHUB_TOKEN &&` (an invalid token may + shadow the login). SonarCloud's read API needs no token for public projects. +- Applies to cedanl repos. diff --git a/.claude/skills/pr-reply/SKILL.md b/.claude/skills/pr-reply/SKILL.md new file mode 100644 index 0000000..cb32c06 --- /dev/null +++ b/.claude/skills/pr-reply/SKILL.md @@ -0,0 +1,40 @@ +--- +name: pr-reply +description: Fetch a PR's review comments, assess each on its merits, and post threaded replies (in Dutch) plus a summary comment citing the fixing commit, then re-request review. Use when a reviewer has left comments on a PR and you want to respond to all of them. +--- + +# pr-reply + +Respond to review feedback on a GitHub PR, thoroughly and honestly. + +## Workflow + +When the user invokes `/pr-reply [optional: PR number]`: + +1. Identify the PR (ask if ambiguous). Get the latest review + inline comments: + - `unset GITHUB_TOKEN && gh api repos/{owner}/{repo}/pulls/{n}/comments --jq 'sort_by(.created_at) | .[] | {id, path, line, in_reply_to: .in_reply_to_id, body}'` + - Also check `gh pr view {n} --json reviews` and issue comments. + - Filter to comments newer than your last reply so you don't re-answer old ones. +2. For EACH comment, read the actual code at that path/line before responding. + Assess it on merits — agree when right, push back with reasoning when not. + Do not rubber-stamp. If a comment is "vanuit claude" (relayed AI analysis), + still verify it against the real code. +3. Make the fixes (or hand off to `/ship`), commit, and note the commit SHA. +4. Post a THREADED reply under each comment: + `unset GITHUB_TOKEN && gh api repos/{owner}/{repo}/pulls/{n}/comments/{comment_id}/replies -f body="..."` + - Write replies in the team's working language (Dutch for CEDA). + - Say what changed, why, and cite the commit SHA. +5. Post ONE summary comment tagging the reviewer, recapping each point + SHA: + `unset GITHUB_TOKEN && gh pr comment {n} --body "@reviewer ..."` +6. Be honest about test coverage gaps (e.g. "regressiewachter, geen live-browsertest"). + +## Important + +- Use the team's working language for all PR-facing text (Dutch for CEDA). +- Cite the specific commit that addresses each comment. +- **Never claim something is fixed without having verified it** — this skill does + NOT rubber-stamp comments or mark them resolved on the author's behalf. +- **gh auth**: an invalid `$GITHUB_TOKEN` may shadow the real `gh` login — prefix + every `gh` command with `unset GITHUB_TOKEN &&`. If `gh auth status` still fails, + ask the user to re-auth; do NOT fall back to curl for authenticated endpoints. +- Applies to cedanl repos. diff --git a/.claude/skills/pypi-project/SKILL.md b/.claude/skills/pypi-project/SKILL.md new file mode 100644 index 0000000..ebfb018 --- /dev/null +++ b/.claude/skills/pypi-project/SKILL.md @@ -0,0 +1,109 @@ +--- +name: pypi-project +description: Set up or publish a CEDA Python package to PyPI the house way — src-layout + setuptools + uv, PEP 621 pyproject metadata, and GitHub Actions Trusted Publishing (OIDC) on v* tags with a TestPyPI preview. Use when making a repo pip-installable, adding a release workflow, or cutting a PyPI release. +--- + +# pypi-project + +Package and publish a Python project matching the modern CEDA convention +(`1cijferho`/`eencijferho`). Read the target repo's existing `pyproject.toml` +first and match it; don't invent a different toolchain. + +## When this applies + +This is a **knowledge skill** — it loads (explicitly via `/pypi-project`, or +automatically) when you make a repo pip-installable, add/edit a release or +TestPyPI workflow, or cut a PyPI release. It is reference/convention plus a short +release ritual, not a wizard. + +## Toolchain (modern house style) +- **Build backend:** setuptools. **Env/deps:** uv. **Layout:** `src//`. +- **Publishing:** GitHub Actions + **Trusted Publishing (OIDC)** — no API tokens + stored anywhere. + +## pyproject.toml shape +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "" # may differ from import name (e.g. eencijferho) +version = "0.1.0" +description = "..." +readme = "README.md" +requires-python = ">=3.12,<3.14" +license = {text = "MIT"} +authors = [{name = "CEDA", email = "info@cedanl.nl"}] +keywords = [...] +classifiers = [...] # Dev Status, Intended Audience, License, Py versions, Topic +dependencies = [...] + +[project.urls] # Homepage, Documentation, Repository, Issues (github.com/cedanl/...) + +[project.optional-dependencies] # feature extras, e.g. frontend / minio / postgres +[project.scripts] # console entry point, e.g. = ".cli:main" + +[dependency-groups] +dev = ["pytest>=9.0.2", "pytest-cov>=6.0"] + +[tool.setuptools.packages.find] +where = ["src"] +include = ["*"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["src"] +``` +Notes: dist name and import name can differ. Use `requires-python` with an upper +bound. Extras for optional features; a `[project.scripts]` entry if there's a CLI. + +## Release workflow (modern: pypi-publish.yml) +Trigger on version tags; three jobs, each `needs:` the previous: +1. **test** — `uv sync --group dev` + `uv run pytest`. +2. **build** — `pip install build` + `python -m build`, upload `dist/` as an + artifact (`persist-credentials: false` on checkout). +3. **publish-to-pypi** — `if: startsWith(github.ref, 'refs/tags/')`, + `environment: {name: pypi, url: https://pypi.org/project//}`, + `permissions: id-token: write`, download the artifact, then + `pypa/gh-action-pypi-publish@release/v1`. +```yaml +on: + push: + tags: ['v*.*.*'] +``` +- The `pypi` GitHub **environment** gates publish behind **manual approval**. +- Trusted Publishing must be configured once in PyPI (publisher = this + repo/workflow/environment `pypi`). Flag this to the user — it's a one-time + PyPI-side setup you can't do from code. + +## TestPyPI preview (preview.yml) +Publish dev builds to **TestPyPI** on push to main / `workflow_dispatch`: +`environment: test-pypi`, `id-token: write`, and +`pypa/gh-action-pypi-publish@release/v1` with +`repository-url: https://test.pypi.org/legacy/`. Bump to a `-dev.N` version so +each build is unique. + +## Cutting a release +1. Bump `version` in `pyproject.toml` (repo may have a "bump versie naar vX" commit convention). +2. Verify locally: `uv build` then `uv run twine check dist/*` (twine check is the + legacy-repo habit; keep it). +3. Commit, then tag `vX.Y.Z` and push the tag — the workflow runs; approve the + `pypi` environment when prompted. +4. Confirm the release on PyPI; don't call it "published" until the run + approval + complete — verify, don't assume. + +## Related +CI matrix / test conventions: /actions-ci. + +## Important +- **Match the repo's existing toolchain.** Modern repos use setuptools + uv; + legacy quartet repos (`eencijfer`, `minio-file`, `sdp-tools`) use poetry + tox + in `preview.yml`/`release.yml`. Don't convert one to the other unasked. +- **Trusted Publishing needs a one-time PyPI-side setup** (linking repo/workflow/ + environment) that can't be done from code — flag it to the user. +- Don't call a release "published" until the workflow run + environment approval + complete. +- **gh auth**: prefix `gh` with `unset GITHUB_TOKEN &&` (an invalid token may + shadow the login). +- Applies to cedanl repos. diff --git a/.claude/skills/ship/SKILL.md b/.claude/skills/ship/SKILL.md new file mode 100644 index 0000000..c8d2c17 --- /dev/null +++ b/.claude/skills/ship/SKILL.md @@ -0,0 +1,48 @@ +--- +name: ship +description: Land a code change safely — review, fix, commit, push — with guardrails that verify the change actually works first. Use when finishing a piece of work that needs to land on a branch. +--- + +# ship + +Land a change safely: verify the work actually runs before calling it done, keep +git state clean, and commit with a conventional message. + +## Workflow + +When the user invokes `/ship`: + +### 1. Make the change + +### 2. Test and lint +Run the relevant tests + linter (`uv run pytest ...`, `uvx ruff check ...`). +Note: heavy Polars tests may segfault locally under x86/Rosetta emulation — +that's environmental, not a code failure; CI runners are native and fine. + +### 3. Verify behavior +For anything user-facing, verify it actually works; be explicit about coverage +gaps. For UI/browser changes that can't be tested headless, SAY SO and ask for +confirmation — never claim a UI change works without evidence. Distinguish +"static checks pass" from "feature verified". + +### 4. Review and stage +Review the diff. Stage only intended files by name (never `git add -A`). + +### 5. Commit and push +Commit with a conventional message (`fix:`/`feat:`/`test:`/`ci:` ...); body in +Dutch is fine. If the team convention is to attribute AI-assisted commits, add a +trailing `Co-Authored-By: Claude ` line. Then push. If a +PR is involved, consider /pr-reply or /branch-pr next. + +## Important + +- **Verify before "done"** — distinguish "static checks pass" from "feature + verified"; be honest about what wasn't tested. +- **Check git state after checkouts/rebases** — run `git status` and confirm no + unintended staged changes (a checkout can silently stage reverting changes); + reconcile local vs remote before committing. +- **Never commit stray files** — stage by name to avoid sweeping in local build + artifacts, reports, or editor junk (`coverage.xml`, generated HTML, etc.). +- **gh auth**: an invalid `$GITHUB_TOKEN` can shadow the `gh` login — prefix `gh` + commands with `unset GITHUB_TOKEN &&`. +- Applies to cedanl repos.