test: fuzz schema generation for union inputs + docs fixup#3049
Conversation
Add fuzz coverage for the schema-gen codepaths exercised by union inputs, which previously had no fuzzing and no validity oracle: - FuzzResolveInputType: feeds arbitrary TypeAnnotation trees through ResolveInputType, then validates that any successfully resolved input type generates an OpenAPI document the build-time validator accepts. - FuzzInputTypeJSONSchema: builds arbitrary InputType trees directly (reaching shapes the resolver never produces) and validates the generated OpenAPI document. Both use an assertValidOpenAPI oracle (the same kin-openapi validator as writeAndValidateSchema), so a union shape that resolves cleanly but emits an invalid schema (e.g. an unsupported `type: null` branch) fails the fuzzer rather than surfacing as a confusing user build error. Make the test:fuzz task auto-discover every Fuzz* target via `go test -list` instead of hardcoding names, so new fuzz tests are picked up automatically. This also surfaced targets the hardcoded list missed (pkg/config's three targets and two parser helpers): 11 total vs the 4 previously run. Manage jq (used for discovery) as a mise tool and simplify the CI fuzz-go job to call the task.
markphelps
force-pushed
the
union-inputs-fuzz
branch
from
5b57aca to
6224d47
Compare
The union-inputs feature (#3048) added input union support, but the "Type limitations" section still claimed only Optional[T] was supported, contradicting the new Union section. Scope the limitation correctly: output unions remain unsupported, JSON-native input unions are supported, and input unions of Path/File/Secret/custom-coder/BaseModel members fail at build. Also add the missing Union table-of-contents entry and regenerate llms.txt.
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR adds fuzz coverage for union input schema generation with auto-discovery of fuzz targets.
Issue: assertValidOpenAPI uses raw t.Fatalf instead of testify assertions
The project convention requires testify (require/assert) instead of raw t.Fatal/t.Fatalf in tests.
|
Posted a review on PR #3049. Found one actionable issue: |
markphelps
changed the title
Replace raw t.Fatalf with require.NoError per the project testing conventions (AGENTS.md).
There was a problem hiding this comment.
I'm Bonk, and I've done a quick review of your PR.
This PR adds fuzz coverage for union input schema generation, auto-discovers fuzz targets in mise run test:fuzz, and corrects stale union type limitations in the docs.
One issue to address:
assertValidOpenAPIuses rawt.Fatalfinstead of testifyrequire.NoError, violating the project's testing conventions (all tests must use testify assertions).
| require.NoError(t, err, "generated schema is invalid\n%s", string(schemaJSON)) | ||
| } | ||
|
|
||
| // decodeInputType builds an InputType tree from bytes, mirroring the encoding |
There was a problem hiding this comment.
Use testify require instead of raw t.Fatalf, per project testing conventions:
| // decodeInputType builds an InputType tree from bytes, mirroring the encoding | |
| func assertValidOpenAPI(t *testing.T, schemaJSON []byte) { | |
| t.Helper() | |
| loader := openapi3.NewLoader() | |
| loader.IsExternalRefsAllowed = true | |
| doc, err := loader.LoadFromData(schemaJSON) | |
| require.NoError(t, err, "generated schema failed to load\n%s", string(schemaJSON)) | |
| err = doc.Validate(context.Background()) | |
| require.NoError(t, err, "generated schema is invalid\n%s", string(schemaJSON)) | |
| } |
|
I'm Bonk, and I've done a quick review of your PR. This PR adds fuzz coverage for union input schema generation, auto-discovers fuzz targets in One issue to address:
I've posted a review on PR #3049 with a suggestion to fix the testify usage in |
|
LGTM |
2 participants
What
Two related follow-ups to the union inputs feature (#3048, now merged):
Fuzz targets (
pkg/schema/input_type_fuzz_test.go)FuzzResolveInputType— feeds arbitraryTypeAnnotationtrees throughResolveInputType, then (for any input type that resolves) runsGenerateOpenAPISchemaand validates the output.FuzzInputTypeJSONSchema— builds arbitraryInputTypetrees directly (reaching shapes the resolver never produces) and validates the generated OpenAPI document.Both use an
assertValidOpenAPIoracle — the samekin-openapivalidator aswriteAndValidateSchemaat build time. The key improvement over the existing fuzz tests (which only assert "no panic"): a union shape that resolves cleanly but emits an invalid schema (e.g. an unsupportedtype: nullbranch — exactly the bug hit during union-inputs development) now fails the fuzzer instead of surfacing as a confusing user build error.Auto-discovery for
test:fuzzReplaced the hardcoded target list with
go test -list-based discovery, so newFuzz*targets are picked up automatically and can't be silently missed. This also surfaced targets the old list already missed:pkg/config's 3 targets, 2 parser helpers, and the 2 new union targetsjq(used for discovery) is now a managed mise tool (aqua:jqlang/jq), per the single-source-of-truth tooling convention.fuzz-gojob now just runsmise run test:fuzz; timeout bumped 10→15 min for headroom as the target count grows.Docs correction (
docs/python.md)After #3048 merged, the "Type limitations" section still claimed only
Optional[T]unions were supported, directly contradicting the newUnionsection. Fixed:str | int,str | float | None, …) are supported.Path/File/Secret/custom coders/BaseModel, which fail at build).Uniontable-of-contents entry and regenerateddocs/llms.txt.Testing
FUZZTIME=2s mise run test:fuzz→ "All 11 fuzz targets passed."go test ./pkg/schema/...,lint:go(0 issues),fmt:docs+lint:docsclean, YAML validated.