verify: incorrect custom-endpoint test in Ops library docs - #8
Closed
dwilding wants to merge 4 commits into
Closed
Conversation
Adversarial test of the 'Test custom endpoint names' section of the Ops docs.
Restructure to two libraries (correct + broken) with a parametrized test: the correct lib passes, the broken lib (docs' verbatim 'Write a library' example) fails and is xfailed so CI stays green.
…n libs jump out The custom-endpoint-name claim has nothing to do with secrets, so drop the credential_secret payload, snapshot/restore override, and typing.cast. Each library is now 40 lines and the two differ only in charm.on[endpoint] vs charm.on['database']. The test is self-contained (charm defined inline in the test).
…oken lib Add test_docs_start_pattern, which reproduces the docs' verbatim test (run start, check saw_start) against both libraries with no xfail. It passes for both -- proving the docs' test pattern cannot tell a wrapper that honours the custom endpoint name from one that ignores it. The existing genuine verification (test_custom_endpoint_name_emits_ready) does distinguish them: passes for correct, xfails for broken.
Owner
Author
|
Verification complete. canonical/operator#2657 tracks the doc issues uncovered. |
tonyandrewmeyer
added a commit
to tonyandrewmeyer/operator
that referenced
this pull request
Jul 27, 2026
Matches the reference test in dwilding/basic-charms#8. Without the extra endpoint declared, a requirer that hard-coded 'database' fails at charm construction with an AttributeError on database_relation_changed, rather than failing because no DatabaseReadyEvent was emitted -- so the example demonstrated the wrong failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The claim
How to manage libraries — Test custom endpoint names states, of its example test:
The section presents this as the way to test that a relation-endpoint wrapper library supports custom endpoint names (i.e. that the wrapper honours the endpoint name the charm passes in, rather than a hardcoded one).
What this PR does
It adds a targeted refutation to the
microncharm, structured as two libraries and two parametrized tests.The two libraries
lib/charms/micron/v0/my_lib.py— aDatabaseRequirerthat correctly supports custom endpoint names: it observescharm.on[endpoint](the endpoint name passed in). This is the docs' "Write a library"DatabaseRequirerwith the constructor parameter renamed toendpointand the hardcoded'database'replaced byendpoint.lib/charms/micron/v0/my_lib_broken.py— aDatabaseRequirerthat does not support custom endpoint names: it observescharm.on['database']regardless of the endpoint name passed in. This is the docs' "Write a library"DatabaseRequirerverbatim (again with the parameter renamed toendpointto match the docs' test call signature).The two libraries are identical except for one line:
charm.on[endpoint]vscharm.on['database'].The two tests
Both tests are parametrized over the two libraries, using a charm that wraps a custom-named endpoint (
foo).test_docs_start_patternreproduces the docs' verbatim test pattern: run thestartevent and checksaw_start. It has noxfail.test_custom_endpoint_name_emits_readyis a genuine verification: firerelation_changedon the custom endpoint and assert the wrapper'sreadycustom event fires.readyevent is never emitted, because the library observescharm.on['database'], not the customfooendpoint), and is markedxfail(strict=True)so that CI stays green while still demonstrating the failure.What this proves
The docs' claim does not hold as written:
The library the docs tell you to write (the "Write a library"
DatabaseRequirer) does not support custom endpoint names — it hardcodes'database'. The "Test custom endpoint names" section's premise (that you have a wrapper supporting custom names) is therefore false against the docs' own example.The docs' "Test custom endpoint names" test pattern cannot detect this:
test_docs_start_patternpasses for both the correct and the broken library, because it only runsstartand checkssaw_start— it never exercises the relation endpoint. A genuine verification (test_custom_endpoint_name_emits_ready) does distinguish them: it passes for the correct library and fails for the docs' verbatim one.The docs should either show a
DatabaseRequirerthat actually uses the endpoint name it is given (charm.on[endpoint]), or qualify the "Test custom endpoint names" test as only checking that the charm constructs/runs with a custom endpoint name (not that the wrapper honours it).tox -e lintpasses;tox -e unitreports4 passed, 1 xfailed; the pre-existingtest_starttest still passes.