docs: add the platform repo to the testing comparison (F10, F11) - #164
Merged
Conversation
platform was missing and is the most useful comparison here: same kind of service, same problem, different choices. Figures from main@75dc795 (2026-09-04); PR #107 adds practices main lacks, recorded separately in 1.6. New sections: - 1.5 how platform tests: 205 backend tests, the same DI singleton-swap pattern executed slightly better (its DB fixtures pair the restore with data cleanup and a comment naming the bug it prevents), real Mongo, Keycloak and Temporal rather than mocks, zero frontend tests, and no coverage instrumentation of any kind. - 1.6 what #107 adds: a 4x6 failure matrix over MockTransport, zero-upstream- request assertions, leak assertions, and OpenAPI contract tests in pytest. main uses no MockTransport at all, which is why the live calls in F10 get through. Two new findings, both only visible by comparing: F10: the two repos fail at gating in OPPOSITE directions and both are wrong. Here everything real is skipped because the credential is absent. In platform the CI invocation is a bare `pytest`, so three tests hit third-party services on every PR in a required check -- one of them carrying no marker at all, so it survives even the documented `-m "not integration"`. Shared cause: in neither repo does the marker decide where the test runs. That is the argument for F.c. Note platform's own smoke.yaml header states the right policy, which its backend suite then does not follow. F11: pytest takes the first pytest.ini and never merges the pyproject block, so one config is dead in platform, sms-api and biosim-client. platform's dead block contains python_files = "main.py", which would collect ZERO tests if the file shadowing it were ever removed as redundant -- a green suite running nothing. This repo has only the pyproject block and is unaffected. Existing findings refined rather than duplicated: F2 gains platform as the other end of the same failure (no codecov config at all, no pytest-cov, and two docs advertising a --cov command that cannot run); F3 gains platform as proof that branch protection is achievable in this org, requiring two build contexts with strict:true and zero approvals -- while smoke, its only cross-service test, is not required. Practice worth copying gains platform's real-identity RBAC tests (a mocked layer and a real-Keycloak layer over the same endpoints, each labelled), its smoke job, fixture teardown that names the bug it prevents, and negative-authz assertions inside the happy path. Practice worth avoiding gains its release path publishing images with no test dependency -- which our build-containers workflow shares -- and documenting a command that cannot run. Old F10 renumbered F12. All twelve findings are referenced by an action. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018TgVbCXXWAWhNjtKTzrxvN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
platformwas missing fromdocs/plan-testing.mdand is the most useful comparison in it: the sibling service, same problem, different choices. Figures frommain@75dc795(2026-09-04); PR biosimulations/platform#107 adds practicesmainlacks, recorded separately.Two findings that only appear by comparing
F10 — the two repos fail at gating in opposite directions, and both are wrong.
SLURM_SUBMIT_KEY_PATHabsent in CIOne of those three carries no marker and no skipif, so it survives even the documented
-m "not integration".lefthook.ymlruns no tests at all, so the drift is never discovered locally. Shared cause: in neither repo does the marker decide where the test runs — which is the argument for F.c. Worth notingplatform's ownsmoke.yamlheader states the right policy ("minutes and flaky external state"), which its backend suite then does not follow.F11 — config shadowing silently disables settings in three repos.
pytesttakes the firstpytest.iniand never merges[tool.pytest.ini_options].platform,sms-apiandbiosim-clienthave both.platform's dead block containspython_files = "main.py"— if thatpytest.iniwere ever removed as redundant, collection would drop to zero tests with a green suite. This repo has only the pyproject block and is unaffected.Existing findings refined, not duplicated
platformas the other end of the same failure: nocodecov.yamlat all, nopytest-covin the lockfile, nothing measured — and two documents advertising a--covcommand that cannot run. Four repos carry an unenforced 90% target; the fifth carries none. Neither end produces a number anyone acts on.platformas proof branch protection is achievable here: two required contexts,strict: true, zero required approvals. Notablysmoke, its only cross-service test, is not required — the check that would catch a broken integration cannot block.New in Part 1
platformtests: 205 backend tests, the same DI singleton-swap pattern executed slightly better (its DB fixtures pair the restore with data cleanup and a comment naming the bug it prevents), real Mongo/Keycloak/Temporal rather than mocks, zero frontend tests.mainlacks: a 4×6 failure matrix overMockTransport, zero-upstream-request assertions, leak assertions, OpenAPI contract tests in pytest.mainuses noMockTransportanywhere, which is why F10's live calls get through.Practice sections
Copying:
platform's real-identity RBAC tests (a mocked layer and a real-Keycloak layer over the same endpoints, each labelled in its docstring), itssmoke.yaml, fixture teardown that names the bug it prevents, negative-authz assertions inside the happy path.Avoiding: its release path publishing three images with no test dependency — which our own
build-containers.ymlshares — and documenting a command that cannot run.Housekeeping
Old F10 renumbered F12; all twelve findings are referenced by at least one action.
make checkclean;mkdocs build -spasses.🤖 Generated with Claude Code
https://claude.ai/code/session_018TgVbCXXWAWhNjtKTzrxvN