Skip to content

test: add HTML-fixture regression scaffolding for brittle compat parsing#70

Merged
erseco merged 2 commits into
mainfrom
issue-68-html-fixtures
Jul 7, 2026
Merged

test: add HTML-fixture regression scaffolding for brittle compat parsing#70
erseco merged 2 commits into
mainfrom
issue-68-html-fixtures

Conversation

@erseco

@erseco erseco commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

Adds fast, network-free characterization tests under tests/unit/ that exercise the version-sensitive HTML parsers in py_moodle.compat against hand-authored HTML fixture files. Previously these fragile parsers (login token, sesskey, folder file listing, dashboard version) had no coverage in the fast make test-unit gate — the only compat tests live in tests/test_compat.py, which is auto-marked integration and skipped by default. Now an accidental selector/regex regression, or a future Moodle-markup change, is caught by the fast unit layer. Test-only: no source files changed.

Linked issue

Closes #68

Specification (SDD)

Add a fixtures directory tests/unit/fixtures/html/ with small, representative, fake-secret HTML documents (login page with a logintoken hidden input, a page embedding the M.cfg sesskey, a folder view listing pluginfile.php filenames, and modern/legacy/no-version dashboards), each with a provenance comment. Add tests/unit/test_html_fixtures.py loading those fixtures via pathlib relative to __file__ and asserting the expected extraction for extract_login_token, extract_sesskey (present and absent), extract_folder_filenames, and extract_version_from_dashboard (modern, legacy, and absent-marker), plus at least one legacy-vs-modern contrast. Uses the existing public compat surface read-only: DEFAULT_COMPATIBILITY / LEGACY_COMPATIBILITY strategy methods and the module-level extract_version_from_dashboard / get_strategy_for_version. No behavior change; this is regression/characterization coverage.

TDD evidence

Red-green with the source held read-only, so the natural "red" is the fixtures not existing yet.

  1. Wrote tests/unit/test_html_fixtures.py first and ran it before creating any fixture:
$ .venv/bin/pytest tests/unit/test_html_fixtures.py -q
FileNotFoundError: [Errno 2] No such file or directory:
  '.../tests/unit/fixtures/html/dashboard_modern.html'
... 11 failed
  1. Authored the six fixtures, re-ran, and one test surfaced a real defect: my provenance comment literally contained the sesskey regex "sesskey":"([^\"]+)", which the parser matched first (returning ([^ instead of the real value). Fixed the comment to not embed a matching literal:
$ .venv/bin/pytest tests/unit/test_html_fixtures.py -q
E  AssertionError: assert '([^' == 'fakeSesskey1a2b3c'
  1. After the fix, focused tests are green and the full unit suite has no regressions:
$ .venv/bin/pytest tests/unit/test_html_fixtures.py
11 passed, 1 warning in 0.04s
$ .venv/bin/pytest tests/unit -q
192 passed

Non-vacuous proof (one-off, NOT committed): renaming the login-token attribute in the fixture (name="logintoken" -> name="logintoken_RENAMED", simulating a Moodle markup change) makes the parser return '' and the test fail, then reverting restores green:

$ .venv/bin/pytest tests/unit/test_html_fixtures.py::test_extract_login_token_from_modern_login_page -q
E  AssertionError: assert '' == 'fake-logintoken-9f8a7b6c5d4e3f2a'
... 1 failed
# (fixture reverted -> 11 passed again)

Test plan

  • Added/authored failing test(s) first.
  • Confirmed red before implementing (fixtures absent -> FileNotFoundError).
  • Implemented minimal change (fixtures only; no source touched).
  • Focused tests green (11 passed).
  • Full pytest tests/unit green (192 passed).

Commands run:

.venv/bin/pytest tests/unit/test_html_fixtures.py -q          # red: 11 failed (FileNotFoundError)
.venv/bin/pytest tests/unit/test_html_fixtures.py -q          # after fixtures: 1 failed (comment matched regex)
.venv/bin/pytest tests/unit/test_html_fixtures.py             # 11 passed
.venv/bin/pytest tests/unit -q                                # 192 passed
.venv/bin/black --check src tests                             # 84 files unchanged
.venv/bin/isort --check-only src tests                        # clean
.venv/bin/flake8                                              # clean
# non-vacuous demo (not committed):
.venv/bin/pytest tests/unit/test_html_fixtures.py::test_extract_login_token_from_modern_login_page -q  # 1 failed after corrupting fixture

Backwards compatibility

Fully backwards compatible. Test-only change: adds new files under tests/unit/, touches no source and no existing tests, uses only the existing public/read-only compat surface.

Documentation

None. Per the issue, the contributor-facing "how brittle-fixture tests work" note is owned by the sibling test-layer-docs issue; docs/development.md is intentionally untouched. docs/cli.md is unaffected (no CLI options changed).

Notes for reviewers

  • Stayed strictly within file-ownership boundaries. Files added (only these):
    • tests/unit/test_html_fixtures.py
    • tests/unit/fixtures/html/login_page_modern.html
    • tests/unit/fixtures/html/sesskey_config_modern.html
    • tests/unit/fixtures/html/folder_view_modern.html
    • tests/unit/fixtures/html/dashboard_modern.html
    • tests/unit/fixtures/html/dashboard_legacy.html
    • tests/unit/fixtures/html/dashboard_no_version.html
  • src/py_moodle/compat.py, tests/test_compat.py, tests/conftest.py, and docs/* are all untouched.
  • extract_folder_filenames takes a BeautifulSoup (not a raw string), so the folder test parses the fixture with the same lxml parser the production code uses.
  • The legacy-vs-modern contrast is covered two ways: extract_folder_filenames is asserted identical across DEFAULT_COMPATIBILITY and LEGACY_COMPATIBILITY, and get_strategy_for_version is asserted to pick LegacyCompatibilityStrategy for the 3.x dashboard vs ModernCompatibilityStrategy for the 5.x dashboard.
  • All embedded tokens/sesskeys/versions are obviously-fake placeholders; each fixture carries a provenance comment.

erseco added 2 commits July 7, 2026 23:39
Add fast, network-free characterization tests under tests/unit/ that
exercise the brittle version-sensitive parsers in py_moodle.compat
against hand-authored HTML fixture files, so a future Moodle-markup
change or an accidental selector regression is caught by make test-unit.

Covers extract_login_token, extract_sesskey (present and absent),
extract_folder_filenames (across both legacy and modern strategies),
and extract_version_from_dashboard (modern, legacy, and absent-marker
cases), plus a legacy-vs-modern strategy-selection contrast via
get_strategy_for_version. All embedded tokens, sesskeys and versions
are obviously-fake placeholders with provenance comments.
@erseco erseco merged commit 57bc3f8 into main Jul 7, 2026
22 of 23 checks passed
@erseco erseco deleted the issue-68-html-fixtures branch July 7, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add HTML-fixture regression scaffolding for brittle parsing

1 participant