Skip to content

Formalize extra CI tests: new tests/extra/ pytest suite + URL-backend regression - #285

Open
yarikoptic wants to merge 14 commits into
masterfrom
enh-testing
Open

Formalize extra CI tests: new tests/extra/ pytest suite + URL-backend regression#285
yarikoptic wants to merge 14 commits into
masterfrom
enh-testing

Conversation

@yarikoptic

@yarikoptic yarikoptic commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

Formalises the "extra tests" that CI runs on top of git annex test and DataLad's own battery into a real pytest suite under tests/extra/, and adds a first new regression test for URL-backend-key retrieval (the URL--yt&chttps&c%%… bug fixed upstream in 8fd9b67 / release 10.20260420).

New in this PR:

  • tests/extra/ — a small, upstreamable pytest suite. Each test declares its own skip / xfail conditions; the default is "run on every platform, skip only where a dep is unavailable".
  • dynlibs test — the previous inline Seek of dynlibs step of test-annex-more, rewritten as pytest with proper timeout + strace exit-code check so a seccomp-restricted runner can't silently pass it.
  • url_backend test — clones datasets.datalad.org/repronim/ReproTube/DataLad/.git/ with --no-single-branch (need the git-annex branch for URL-key metadata), asserts git annex whereis decodes the URL-encoded key back to youtube.com/watch?v=, and git annex get retrieves the ~18 MB video file (served over HTTPS by the origin remote, no yt-dlp needed).
  • New test-extra CI job — replaces test-annex-more (which was Ubuntu/macOS only, single inline step), runs on all four platforms (Ubuntu, macOS Intel, macOS ARM64, Windows), one pip install pytest on each.
  • pytest_report_header prints the versions of git, git-annex (all critical fields from git annex version), yt-dlp, youtube-dl, strace, plus a "git-annex releases in this repo newer than installed (VER): N" line derived from git tag --list "10.*".
Design decisions made along the way
  • pytest, not bats. The tests/extra/ directory was prototyped in both frameworks side-by-side; five independent senior-engineer code reviews unanimously recommended pytest. Reasons preserved in tests/extra/README.md: (a) cross-platform install cost — bats needs three distinct install recipes and its Windows story via Git Bash is fragile; (b) no real xfail primitive in bats (only skip, which hides regressions); (c) conftest.py + fixtures give session-cached version parsing and module-scoped clones out of the box. The bats prototype was dropped in 0168c89909.
  • URL-backend xfail marker. test_get_url_backend_key is xfail(condition=git_annex_version < 10.20260420, strict=False) so local dev on an old branch doesn't red the run, but on CI ($CI set) the xfail is forcibly disabled — every failure on the specific build under test is loud.
  • tests/_helpers.py + pythonpath = tests in setup.cfg. So from _helpers import … works under any pytest --import-mode. Prior review flagged from conftest import … as fragile.
  • Windows-safe teardown. The module-scoped ReproTube clone uses an _make_tree_writable walk (mirroring chmod -R u+w) plus an onexc rmtree handler, because git-annex marks both object files and their containing key directories as 0500.

Test plan

  • pytest tests passes locally on git-annex 10.20260421 (4 passed).
  • CI=1 pytest tests also passes (xfail path forcibly disabled).
  • pytest --import-mode=importlib tests/extra/pytest/test_url_backend.py passes (import path is not conftest-magic-dependent).
  • Workflow YAML lints clean for all four regenerated build-*.yaml.
  • reuse lint still reports full compliance.
  • Confirm the four build-{ubuntu,macos,macos-arm64,windows}.yaml runs land green on the first cron / dispatch after merge.

@yarikoptic yarikoptic added github_actions Pull requests that update GitHub Actions code testing Relating to testing infrastructure/setup we have labels Aug 3, 2026
yarikoptic and others added 9 commits August 12, 2026 16:07
@yarikoptic @claude
Add tests/extra/ framework for CI-only regression tests
e5176e3 
Formalise the "extra" tests that CI runs on top of `git annex test`
and DataLad's own battery, so new real-world-regression checks can be
added by dropping in a file rather than editing a workflow step. Two
parallel prototype implementations are shipped side-by-side (pytest
and bats) to compare styles before picking one.

New under tests/extra/:

- README.md documents the layout, per-test skip semantics, and local
  invocation for both frameworks.
- pytest/ (test_dynlibs.py, test_url_backend.py) and bats/ (helpers.bash,
  setup_suite.bash, dynlibs.bats, url_backend.bats).
- Shared pytest helpers + version-reporting hook live in
  tests/conftest.py (rather than tests/extra/pytest/conftest.py) so
  `pytest_report_header` fires regardless of the caller's argument
  path — pytest loads ancestor conftests eagerly but descendants
  lazily during collection, too late for the header.

Tests:

- dynlibs: strace-based guard against libpcre ENOENT-lookup regressions
  during `git-annex version` / `git-annex init`. Linux-only (auto-
  skipped where strace is absent). Replaces the inline "Seek of
  dynlibs" step of the previous test-annex-more job, verbatim, in both
  frameworks.
- url_backend: clones the real repronim/ReproTube DataLad dataset with
  --no-single-branch (need the git-annex branch for URL-key metadata),
  asserts `git annex whereis` decodes the URL-encoded key back to
  `youtube.com/watch?v=`, and `git annex get` retrieves the file
  (~18 MB, served over HTTPS by the origin remote, so no yt-dlp
  required). The `get` variant is xfailed (strict=False) below the
  fix version so old builds do not red the run but a fixed-build
  regression fails loudly. Bats has no xfail primitive; it uses `skip`
  as the nearest equivalent (documented in-file).
- Fix version 10.20260420 identified from upstream commit 8fd9b67
  "factor out extendUrlWithPath and use for git http remote key urls"
  (Joey Hess, 2026-02-16), first shipped in that release. The bug
  originally motivating this test is a URL-encoded key path (from
  `URL--yt&chttps&c%%…` produced by keyFile) yielding an invalid
  URI when concatenated with a git http remote base — see also the
  con/git-annex-side patch 20260212-43a3f3aaf2 that shipped the fix
  ahead of upstream (removed once absorbed upstream).

Version-reporting hook (mirrors the dandi-schema pytest_plugin
pattern):

- pytest prints an "extra-tests tools" line with `git`, `bats`,
  `yt-dlp`, `youtube-dl`, `strace` first-lines (or `(missing)`);
  a `platform:` line; the critical fields from `git annex version`
  (git-annex has no `--json` for version, so the human output is
  parsed); and a "git-annex releases in this repo newer than
  installed (VER): N" line derived from `git tag --list "10.*"`.
- bats prints the same header via setup_suite.bash -> print_versions
  in helpers.bash, using bats fd 3 so it's visible without opting
  into --show-output-of-passing-tests.

Workflow template:

- `test-annex-more` (Ubuntu/macOS-only, single inline "Seek of dynlibs"
  step) is replaced by `test-extra`, which runs on all four platforms
  (Ubuntu, macOS Intel, macOS ARM64, Windows). On each platform it
  installs the platform-appropriate bats (apt / brew / git-clone), sets
  up Python 3.12 for pytest, then runs both `python -m pytest -v
  tests/extra/pytest/` and `bats tests/extra/bats/`. Regenerated the
  four concrete build-*.yaml files via the existing mkworkflows.py.
- REUSE.toml gets `tests/**` added to the MIT/DataLad-Team default
  block so `reuse lint` stays clean.

The bats variant will be eliminated in a follow-up commit once the
comparison has been reviewed; kept here so the review is against
tree, not against a diff.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Drop bats prototype; pytest is the extra-tests suite
2ed43b7 
Five independent senior-engineer reviews of the prior
side-by-side bats + pytest prototype unanimously recommended pytest.
The short version of their reasoning, preserved in
tests/extra/README.md so this question doesn't get re-litigated:

- Cross-platform install cost: bats needs three distinct install
  recipes (Ubuntu apt / macOS brew / Windows git-clone bootstrap)
  and runs under Git Bash on Windows where `timeout`, `chmod -R u+w`,
  and `sort -V` behave subtly differently. pytest is one `pip install`
  on all four runners, and Python is already required by test-datalad.
- No real xfail primitive in bats: `skip` cannot distinguish
  "known-broken on this version" from "unexpectedly passed" — a
  materially weaker regression signal for the URL-backend `get` test.
- Fixtures and shared helpers: `conftest.py` gives cached
  `git annex version` parsing, session-scoped module clones, and
  parametrization out of the box; the bats port re-implemented each
  by hand.

Removed:

- tests/extra/bats/{dynlibs.bats,url_backend.bats,helpers.bash,
  setup_suite.bash}.
- The "Install bats" (apt / brew / git-clone) template branches and
  the "Run bats suite" step in
  .github/workflows/template/build-{{ostype}}.yaml.j2. The Ubuntu
  step is now "Install strace" (previously bundled strace + bats).
- Regenerated the four concrete workflow YAMLs via
  .github/workflows/template/mkworkflows.py; YAML-linted clean.

tests/extra/README.md rewritten to describe the pytest-only setup and
to explain why bats was evaluated and rejected. `reuse lint` still
clean (45/45). Local pytest run: 4 passed.

Concrete bugs the reviewers also flagged in the pytest side (strace
exit-code check, `from conftest import ...` fragility, xfail
evaluated-at-import without a git-annex-present guard, Windows
read-only rmtree cleanup, dead git_annex_repo fixture) are left for
a follow-up commit to keep the "drop bats" change reviewable in
isolation.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Delete unused git_annex_repo fixture from tests/conftest.py
91e47ad 
The fixture was seeded at the start of the extra-tests scaffolding
work but no test ever consumed it — test_dynlibs and test_url_backend
each build their own repo inline.  Reviewer #1 flagged it as dead
code; removing it (along with the now-unused Path import) keeps
conftest.py to just the version-helper API and the report hook.

No behaviour change; 4/4 pytest tests still pass.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Extract version helpers to tests/_helpers.py
c9e8b84 
Reviewer #1/#2/#3/#5 flagged `from conftest import ...` in
test_url_backend.py as fragile: it relies on pytest's default
--import-mode=prepend adding tests/ to sys.path via the conftest at
that level.  Under --import-mode=importlib the import fails, and
running the test file directly from an unusual cwd is at the mercy of
pytest's rootdir heuristics.

Fix: move URL_BACKEND_FIX_VERSION and the git_annex_version /
git_annex_version_below / git_annex_releases_since helpers into a
plain module tests/_helpers.py.  conftest.py now only holds the
`pytest_report_header` hook (plus its two report-specific helpers
`_first_line` and `_git_annex_summary`), and imports the shared bits
from `_helpers`.

To make `from _helpers import ...` work under any --import-mode,
setup.cfg gets a `[tool:pytest]` section with `pythonpath = tests`.
This is the pytest-native way to guarantee a directory ends up on
sys.path regardless of import-mode / rootdir resolution.

Verified both import modes:

  $ pytest tests                                    # prepend (default)
  4 passed
  $ pytest --import-mode=importlib tests/extra/...  # importlib
  2 passed

Both invocations still print the "extra-tests tools" / git-annex
version / releases-newer-than-installed header.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Harden dynlibs _count_enoent against silent strace failures
8193bdc 
All five reviewers flagged the same issue: `subprocess.run(["strace",
...])` had no `check=`, no timeout, and no sanity check on stderr.
On a runner where strace can't attach (`kernel.yama.ptrace_scope=1`
or `=2`, seccomp filters, some container profiles) stderr is empty,
the ENOENT count is 0, and the assertion `0 < 7` / `0 < 260` passes
vacuously — meaning a regression that reintroduces hundreds of
lookups would be missed.

Fix: on top of the existing invocation, `_count_enoent` now

- passes `timeout=120` so a hung strace fails the test instead of
  the whole CI job,
- raises `RuntimeError` on non-zero exit, echoing the last 2KB of
  stderr for diagnostics,
- raises when stderr contains no syscall / exit lines at all
  (`+++ exited`, ` ENOENT `, ` = `), which is the observable symptom
  of a blocked/refused strace.

Local run still: 2 passed.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
test_url_backend: Windows-safe teardown of the ReproTube clone
2e621cf 
Reviewers #3/#4 flagged: the module-scoped `cloned_repo` fixture used
`tmp_path_factory.mktemp` and returned the repo, leaving cleanup to
pytest's own later `shutil.rmtree`.  On Windows that fails because
git-annex sets object files (and their containing key directories)
to read-only; the same trap bites POSIX teardown when the
containing directory is 0500 — locally reproduced with a
`PermissionError` on unlinking
`.git/annex/objects/vJ/G8/URL--yt...`.

Fix:

- Convert the fixture to `yield` + explicit teardown.
- Walk the tree bottom-up and chmod every dir and file `u+rwx` before
  rmtree (mirrors `chmod -R u+w` from the dropped bats teardown).
- Keep an rmtree `onexc` / `onerror` handler as a belt-and-braces
  fallback if a new read-only entry appears between the walk and the
  unlink.  Uses `onexc` on Python 3.12+ and `onerror` on older
  Pythons (workflow pins 3.12, but local dev on 3.11 stays green).

Local run: 4 passed; the parent `/tmp/pytest-of-USER/pytest-N/`
directory is now empty on exit.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Colocate URL_BACKEND_FIX_VERSION with its only consumer
20d16a6 
Moved from tests/_helpers.py to tests/extra/pytest/test_url_backend.py
so the fix-version constant, the DEP-3-style comment explaining WHY
that specific version, and the xfail marker that uses it all live in
one place.  _helpers.py drops back to purely generic version-parsing
utilities that make no assumption about which bug is being probed.

No behaviour change; 4/4 pytest tests still pass.

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
Forbid xfails on CI so failures are always loud
d3f2e74 
On CI we build a specific git-annex and run against it: every failure
is signal.  The xfail(strict=False) marker on test_get_url_backend_key
was designed for the case "local dev on an older branch where the bug
is known" — silently accepting the failure keeps CI reviewers focused
on real regressions.  That trade-off doesn't apply on CI itself:
there, an XFAIL is a hidden failure.

Fix: gate the xfail condition on `not os.environ.get("CI")`.  In
concrete terms:

- Locally, xfail engages when installed git-annex < fix version, as
  before.  Silent XFAIL / XPASS on older versions; loud FAIL / PASS
  on current versions.
- On CI (CI env var set — GitHub Actions, Travis, GitLab, and every
  other standard runner set this), the xfail marker is inert.  A
  regression that happens to hit an old version on CI (e.g.  a
  workflow_dispatch of a pre-fix commitish) will red the run — which
  is what we want, because CI's job is to say pass or fail on the
  exact build under test.

Verified: `pytest tests` and `CI=1 pytest tests` both green locally
on 10.20260421 (post-fix, xfail inert either way).

Co-Authored-By: Claude Code 2.1.220 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic
[DATALAD RUNCMD] Regenerate workflows after rebase etc
e0d012a 
=== Do not change lines below ===
{
 "chain": [
  "024459120c9868becff366655e3e5bc1533dddae"
 ],
 "cmd": "make -C .github/workflows/template",
 "exit": 0,
 "extra_inputs": [],
 "inputs": [],
 "outputs": [],
 "pwd": "."
}
^^^ Do not change lines above ^^^
yarikoptic and others added 3 commits August 12, 2026 22:27
@yarikoptic @claude
CI: short TMPDIR + core.longpaths for Windows test-extra
a2e936c 
Windows test-extra failed at fixture setup cloning the ReproTube
DataLad dataset:

    error: unable to create symlink authors.tsv: Filename too long
    subprocess.CalledProcessError: git clone … returned non-zero exit
    status 128

DataLad datasets store every annexed file as a symlink whose target
is `.git/annex/objects/XX/YY/SHA256E-s<size>--<64hex>.ext/SHA256E-
s<size>--<64hex>.ext` (~200 chars).  Combined with the runner's
default temp base (`C:\\Users\\runneradmin\\AppData\\Local\\Temp`
plus pytest's per-run subdir, ~90 chars) the resolved paths cross
Windows' 260-char MAX_PATH, and git checkout can't create the
symlink.

Fix at the workflow level rather than inside the test, so a local
Windows run keeps using the developer's usual TMPDIR and no
test-specific globals leak into their filesystem:

- Job-level `env: TMP: C:\\t` / `TEMP: C:\\t` for test-extra on
  Windows, so `tempfile.gettempdir()` (and hence pytest's
  `tmp_path_factory`) roots at a very short path.  Any test that
  shells out to `git clone` on repos with long paths benefits, not
  just test_url_backend.
- `mkdir -p /c/t` step to create the dir before pytest tries to
  use it.
- `git config --system core.longpaths true` step, matching the
  build-package job.  Each `runs-on: windows-2025` job runs on its
  own fresh runner instance, so the system-level setting doesn't
  carry from build-package into test-extra.

Non-Windows jobs are untouched — the whole block is inside an
`{% if ostype == "windows" %}` gate.

Ref failing CI: build-windows job 94254219522 (test-extra step,
2026-08-12 20:35 UTC).

Co-Authored-By: Claude Code 2.1.228 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
test_url_backend: keep exists() check, add diagnostics on failure
983c015 
On 2026-08-13, Windows test-extra (job 94328458021) reproduced a
puzzling failure: `git annex get` reported success —

    7.76 KiB → 6.95 MiB (26 MiB/s)
    ok
    (recording state in git...)

— but Python's `target.exists()` returned False on the working-tree
path.  Two possibilities we don't yet have enough evidence to
choose between:

  (a) a Windows-specific git-annex bug / race where content lands in
      `.git/annex/objects/…` but the working-tree symlink/pointer
      isn't updated; or
  (b) an incorrect assertion in this test that doesn't match the
      crippled-fs / adjusted-branch working-tree layout used on
      Windows.

Rather than guess (as an earlier proposal to switch the check to
`git annex find --in=here` would have), keep the strict
`Path.exists()` + `st_size > 0` invariant — that's what a user
expects after `annex get` — and instrument the failure path to
dump enough state for an upstream issue on con/git-annex:

- Working-tree entry: os.path.lexists, exists, is_symlink, lstat.
- If it's a symlink: readlink target, resolved path, resolved.exists.
- git-annex's own view: find --in=here, whereis, info --bytes,
  lookupkey, version, annex.crippledfilesystem, core.symlinks,
  core.longpaths, git status --porcelain, git log for the path.
- Annex object path (via `examinekey --format=${objectpath}`) and
  whether the object actually landed there with expected size.
- Parent directory listing (did the intermediate dirs get created?)

Also add an explicit `find --in=here` pre-check so we can distinguish
"get silently didn't download" from "get downloaded but working-tree
entry isn't visible" — different upstream bugs.

All diagnostic subprocess calls use a `_run()` wrapper that never
raises (uses timeout=30, returns a CompletedProcess with rc=-1 on
error), so a hang in one diagnostic doesn't mask the root failure.

Non-failure path is unchanged (no diagnostics collected on the
green path, keeping the test fast).

Co-Authored-By: Claude Code 2.1.228 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic
[DATALAD RUNCMD] Remove -q from annex init so we see what annex decid…
b19d133 
…ed on that system

=== Do not change lines below ===
{
 "chain": [],
 "cmd": "git-sedi '\"init\", \"-q\"' '\"init\"'",
 "exit": 0,
 "extra_inputs": [],
 "inputs": [],
 "outputs": [],
 "pwd": "."
}
^^^ Do not change lines above ^^^
yarikoptic and others added 2 commits August 13, 2026 11:34
@yarikoptic @claude
TEMP: trim CI to just build-package + test-extra on Ubuntu/Windows
ee5c72d 
Purpose: quickly confirm whether Windows test-extra is
intermittently failing (as `983c015184` suggested with the hashdir
mismatch that then vanished on the very next run) or reliably
passing.  Skipping the expensive matrix jobs lets us re-trigger many
times without waiting ~40 min per iteration for test-datalad.

Skipped via `if: false` (jobs still appear in the check list as
"skipped", but their steps don't run):

  build-ubuntu.yaml:  test-annex, test-datalad
  build-windows.yaml: test-annex, test-datalad
  build-macos.yaml:       build-package  (cascades → all)
  build-macos-arm64.yaml: build-package  (cascades → all)

Kept: build-package + test-extra on Ubuntu and Windows, plus the
top-level reuse and typing checks.

MUST be reverted before merging PR #285.  Simplest way:
`git revert <this-commit-sha>` which restores all four workflow
files to their template-generated state.

Co-Authored-By: Claude Code 2.1.228 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@yarikoptic @claude
test_url_backend: capture annex-init output + fix diag config queries
8d7f19d 
Two problems in the diagnostic output shown by the last failed run
(job 94504607393, Windows test-extra on ee5c72d):

1. `git annex init` output was invisible.  The module-scoped fixture
   runs setup once, and pytest attaches its captured stdout/stderr
   to the *first* test that used the fixture — which passed
   (test_whereis).  When test_get later fails, pytest shows only
   *its* captured output, so annex init's messages (any adjusted-
   branch conversion, crippled-fs detection, etc.) get eaten.
   Fix: run annex init with `capture_output=True` and stash the
   result in a module-level `_FIXTURE_LOG` that `_collect_diagnostics`
   dumps alongside the rest.

2. The `git annex config --get annex.crippledfilesystem` query was
   wrong — `git annex config` reads the *git-annex-branch-stored*
   config (which errored with "not a configuration setting that can
   be stored in the git-annex branch") instead of the local
   .git/config where crippledfilesystem is actually stored.  Fix:
   use `git config annex.crippledfilesystem` (which reads .git/config).

Also add:
 - `git branch --show-current` + `git symbolic-ref HEAD` → confirm
   whether HEAD is on a normal branch or an `adjusted/*` one.
 - `git config annex.direct` / `annex.version` / `annex.uuid` → full
   picture of what annex init decided.

Next Windows failure should now show what git-annex actually
detected during init (crippled? adjusted? normal-with-symlinks?),
which is the missing puzzle piece before filing upstream.

Co-Authored-By: Claude Code 2.1.228 / Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

github_actions Pull requests that update GitHub Actions code testing Relating to testing infrastructure/setup we have

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@yarikoptic