Which tests protect evaluated-property propagation through anyOf?

[Rul1an]

Hi Jason,

I read #70 and the json-schema-coverage README. This is a different question from schema-location coverage: which existing tests would catch a specific regression in the validator?

I maintain Corpus Adequacy and am considering a small pass over draft 2020-12 anyOf and unevaluatedProperties. Two candidate changes are letting evaluated properties from a failing anyOf branch reach the parent, and stopping anyOf after its first successful branch so later evaluated properties are lost. These are deliberate changes to test, not bugs I have found in Hyperjump.

I would pin Hyperjump, the official Test Suite and dependencies, leave the tests unchanged, and record which tests distinguish each change in valid/invalid results. A known-detectable change and a no-op would check the harness. No overall coverage percentage, annotation-output claim, or optional-test obligations. If both changes are already caught, that is useful too. If one survives, I would first check that it really changes required behavior, then try to reduce it to a regression test.

Are these useful cases to investigate, or is there an existing check or design constraint I should account for first? No integration work requested. I have read the relevant code but have not run the experiment.

Prepared with AI assistance; any reported findings will be checked against reproducible runs.

[jdesrosiers]

Sorry, I'm having difficult understanding exactly what you're asking.

You'll find the unevaluatedProperties tests at https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft2020-12/unevaluatedProperties.json. You could grep them for anyOf to identify the tests that use both. If you think there are cases that aren't adequately covered, you're probably right. That repo would be the place to propose adding additional tests.

Two candidate changes are letting evaluated properties from a failing anyOf branch reach the parent, and stopping anyOf after its first successful branch so later evaluated properties are lost.

I don't know what changes you're talking about, but I can tell you that anyOf can't stop after it's first successful branch. It has to check every branch because any successful branch can contribute evaluated properties. An implementation that does that short-circuiting would be incorrect and if we don't have a test exposing that bug, we should add it.

I read #70 and the json-schema-coverage README.

I'm not seeing how coverage is related to anything else in this discussion.

[Rul1an]

Thanks, Jason. I made the question more complicated than it needed to be. By “changes” I meant deliberately introducing a bug in a local copy of the validator, then running the unchanged tests to see whether they catch it. I wasn't proposing a change to Hyperjump's intended behaviour.

The anyOf cases you linked give me a concrete starting point. I'll check those first; I haven't run the altered validator yet. The coverage reference was a distraction.

If that turns up a reproducible missing test, I'll bring a minimal example to the Test Suite repo, as you suggest. Thanks for pointing me there.

[jdesrosiers]

Oh, I think I understand now. You want to use this library to do mutation testing to identify gaps in the test suite. That's a great idea. Good luck.