Skip to content

verify: incorrect custom-endpoint test in Ops library docs - #8

Closed
dwilding wants to merge 4 commits into
mainfrom
adversarial-custom-endpoint-names
Closed

verify: incorrect custom-endpoint test in Ops library docs#8
dwilding wants to merge 4 commits into
mainfrom
adversarial-custom-endpoint-names

Conversation

@dwilding

@dwilding dwilding commented Jul 19, 2026

Copy link
Copy Markdown
Owner

The claim

How to manage libraries — Test custom endpoint names states, of its example test:

Verify that the charm executes regardless of how we name the requirer endpoint.

The section presents this as the way to test that a relation-endpoint wrapper library supports custom endpoint names (i.e. that the wrapper honours the endpoint name the charm passes in, rather than a hardcoded one).

What this PR does

It adds a targeted refutation to the micron charm, structured as two libraries and two parametrized tests.

The two libraries

  1. lib/charms/micron/v0/my_lib.py — a DatabaseRequirer that correctly supports custom endpoint names: it observes charm.on[endpoint] (the endpoint name passed in). This is the docs' "Write a library" DatabaseRequirer with the constructor parameter renamed to endpoint and the hardcoded 'database' replaced by endpoint.

  2. lib/charms/micron/v0/my_lib_broken.py — a DatabaseRequirer that does not support custom endpoint names: it observes charm.on['database'] regardless of the endpoint name passed in. This is the docs' "Write a library" DatabaseRequirer verbatim (again with the parameter renamed to endpoint to match the docs' test call signature).

The two libraries are identical except for one line: charm.on[endpoint] vs charm.on['database'].

The two tests

Both tests are parametrized over the two libraries, using a charm that wraps a custom-named endpoint (foo).

  1. test_docs_start_pattern reproduces the docs' verbatim test pattern: run the start event and check saw_start. It has no xfail.

    • Against the correct library: passes.
    • Against the broken library: passes — proving the docs' test pattern cannot distinguish a wrapper that honours the custom endpoint name from one that ignores it.
  2. test_custom_endpoint_name_emits_ready is a genuine verification: fire relation_changed on the custom endpoint and assert the wrapper's ready custom event fires.

    • Against the correct library: passes.
    • Against the broken library: fails (the ready event is never emitted, because the library observes charm.on['database'], not the custom foo endpoint), and is marked xfail(strict=True) so that CI stays green while still demonstrating the failure.

What this proves

The docs' claim does not hold as written:

  1. The library the docs tell you to write (the "Write a library" DatabaseRequirer) does not support custom endpoint names — it hardcodes 'database'. The "Test custom endpoint names" section's premise (that you have a wrapper supporting custom names) is therefore false against the docs' own example.

  2. The docs' "Test custom endpoint names" test pattern cannot detect this: test_docs_start_pattern passes for both the correct and the broken library, because it only runs start and checks saw_start — it never exercises the relation endpoint. A genuine verification (test_custom_endpoint_name_emits_ready) does distinguish them: it passes for the correct library and fails for the docs' verbatim one.

The docs should either show a DatabaseRequirer that actually uses the endpoint name it is given (charm.on[endpoint]), or qualify the "Test custom endpoint names" test as only checking that the charm constructs/runs with a custom endpoint name (not that the wrapper honours it).

tox -e lint passes; tox -e unit reports 4 passed, 1 xfailed; the pre-existing test_start test still passes.

dwilding added 4 commits July 20, 2026 07:09
Adversarial test of the 'Test custom endpoint names' section of the Ops docs.
Restructure to two libraries (correct + broken) with a parametrized test: the correct lib passes, the broken lib (docs' verbatim 'Write a library' example) fails and is xfailed so CI stays green.
…n libs jump out

The custom-endpoint-name claim has nothing to do with secrets, so drop the credential_secret payload, snapshot/restore override, and typing.cast. Each library is now 40 lines and the two differ only in charm.on[endpoint] vs charm.on['database']. The test is self-contained (charm defined inline in the test).
…oken lib

Add test_docs_start_pattern, which reproduces the docs' verbatim test (run start, check saw_start) against both libraries with no xfail. It passes for both -- proving the docs' test pattern cannot tell a wrapper that honours the custom endpoint name from one that ignores it. The existing genuine verification (test_custom_endpoint_name_emits_ready) does distinguish them: passes for correct, xfails for broken.
@dwilding dwilding changed the title break: custom endpoint name test in micron validate: incorrect custom-endpoint test in Ops library docs Jul 21, 2026
@dwilding dwilding changed the title validate: incorrect custom-endpoint test in Ops library docs verify: incorrect custom-endpoint test in Ops library docs Jul 21, 2026
@dwilding

Copy link
Copy Markdown
Owner Author

Verification complete. canonical/operator#2657 tracks the doc issues uncovered.

@dwilding dwilding closed this Jul 21, 2026
tonyandrewmeyer added a commit to tonyandrewmeyer/operator that referenced this pull request Jul 27, 2026
Matches the reference test in dwilding/basic-charms#8. Without the extra
endpoint declared, a requirer that hard-coded 'database' fails at charm
construction with an AttributeError on database_relation_changed, rather
than failing because no DatabaseReadyEvent was emitted -- so the example
demonstrated the wrong failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant