PyDABs: acceptance-test authoring skill + per-resource coverage guard - #6527
Open
Sankalp-Mittal wants to merge 6 commits into
Open
Conversation
An AI Agent Skill that guides an agent to author the acceptance test for a newly-onboarded PyDABs resource: the acceptance/bundle/python/<plural>-support/ fixture (databricks.yml + resources.py + mutators.py + script + test.toml + generated output.txt). Includes fill-in templates (.tmpl so they stay out of linters). Complements the schema-synthesized unit-test generation; realistic field values are adapted from the resource's invariant config. Co-authored-by: Isaac <no-reply@databricks.com>
test_python_support_coverage fails until each resource in the _ResourceType
registry has an acceptance/bundle/python/<plural>-support/ fixture, so coverage
cannot silently regress as resources are onboarded. Mirrors the invariant-config
coverage guard; shrink-only _LACKING allowlist ({jobs}, whose coverage predates
the convention). Lives in the python test suite (runs in CI via pydabs-test) since
it checks the filesystem rather than exercising the CLI.
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
marked this pull request as ready for review
September 4, 2026 12:01
Collaborator
Integration test reportCommit: 43de1d9
Top 6 slowest tests (at least 2 minutes):
|
Per review feedback (skills aren't reliably loaded, and the examples plus a verbose failure are enough): drop the pydabs-acceptance-test skill in favor of the repo's dresources pattern — a path-scoped .agents/rules/ file (auto-loaded when working under acceptance/bundle/python/**) pointing to a concise acceptance/bundle/python/README.md that leans on the existing fixtures. Retarget the coverage guard's message at the README. Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Sep 4, 2026
| @@ -0,0 +1,8 @@ | |||
| --- | |||
| description: Rules for authoring PyDABs resource acceptance tests | |||
| globs: acceptance/bundle/python/** | |||
Member
There was a problem hiding this comment.
.agents/rules/*.md also need to be symlinked to .cursor/rules/<rule>.mdc. would be great if you could write a lint rule that auto-adds these symlinks 🙏
Collaborator
Author
There was a problem hiding this comment.
added, the rule. Please TAL once
Comment on lines
+19
to
+20
| # jobs predates the <plural>-support convention; covered across the suite instead. | ||
| "jobs", |
Member
There was a problem hiding this comment.
are you going to migrate the test for this resource?
Collaborator
Author
There was a problem hiding this comment.
I plan on doing it eventually, but I am focussing on completing the codegen for now, the migration can be done later on as well
Add tools/validate_cursor_rules.py (wired into `task checks`) so a rule under .agents/rules/ without its .cursor/rules/<name>.mdc symlink fails CI; `--fix` auto-creates missing symlinks and drops stale ones. Also add the symlink for the new pydabs-acceptance-tests rule. Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Sep 4, 2026
| cmds: | ||
| - "! git grep -lF databricks.com -- '*uv.lock' '*.py.lock'" | ||
|
|
||
| check-cursor-rules: |
Member
There was a problem hiding this comment.
this should ideally be in a different PR
Collaborator
Author
There was a problem hiding this comment.
added PR#6529
Remove tools/validate_cursor_rules.py and its check-cursor-rules task; the symlink-mirror check is being shipped on its own. Keep the .cursor/rules/pydabs-acceptance-tests.mdc symlink for the rule added here. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two complementary pieces that make every PyDABs resource get a hand-written acceptance test:
pydabs-acceptance-testskill (.agents/skills/pydabs-acceptance-test/) — an AI Agent Skill that guides an agent to author the acceptance test for a newly-onboarded PyDABs resource: theacceptance/bundle/python/<plural>-support/fixture (databricks.yml+resources.py+mutators.py+script+test.toml+ generatedoutput.txt). Ships fill-in.tmpltemplates. The 7-step flow: verify the resource is wired → find required fields (VariableOr[...]in the generated_models) → adapt realistic values from the resource's invariant config (dropping$VARinterpolation and cloud-only blocks) → author the fixture → generate the golden with-update→ re-run without-updateto prove determinism → confirm coverage + lint.test_python_support_coverageguard (python/databricks_tests/core/test_python_support.py) — asserts every resource in the_ResourceTyperegistry has a<plural>-supportfixture, so coverage can't silently regress as resources are onboarded. Mirrors the existing invariant-config coverage guard (acceptance/invariant_test.go). Shrink-only_LACKINGallowlist, currently{jobs}(jobs' acceptance coverage predates the<plural>-supportconvention and is spread across thebundle/pythonsuite).Why
PyDABs acceptance tests are hand-written, one per resource, because the OpenAPI spec has no examples — an agent finding realistic values and writing a deterministic test is the scalable path. The skill is the authoring aid; the guard is the enforcement that turns "every new resource needs an acceptance test" into a CI gate. Together they are a prerequisite for full PyDABs resource coverage: onboard a resource → guard goes red → run the skill → green.
Both the skill and the guard derive the resource set from generated/runtime state (the
_ResourceTyperegistry), not theRESOURCE_NAMESPACEallowlist, so they follow the generated set whether it is allowlist-driven today or schema-derived later.Testing
test_python_support_coveragepasses (6/6); verified it fails with a clear message when a fixture is absent.ruff check/ruff formatclean; template files use the.tmplsuffix so the placeholder sources stay out of the linters.Stacked on #6421 (PyDABs deterministic unit-test generation).
This pull request and its description were written by Isaac.