Migrate the rest of python unit tests to conformance tests#1266
Migrate the rest of python unit tests to conformance tests#1266nan-yu merged 18 commits intogoogle:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the initial implementation of the A2UI C++ SDK, covering schema management, validation, and a streaming parser. It also expands the shared conformance test suite and updates the Python SDK tests to leverage these new cases. Feedback on the C++ implementation suggests ensuring consistent error handling in catalog selection to match conformance expectations, adopting modern memory management with std::unique_ptr in tests, and using custom delimiters for embedded raw strings to prevent parsing errors. Additionally, improvements were recommended for optimizing component lookup complexity and removing an unused dependency in the CMake configuration.
932a458 to
259a3d8
Compare
259a3d8 to
4b66dc0
Compare
e746d45 to
06efb7e
Compare
jacobsimionato
left a comment
There was a problem hiding this comment.
LGTM! The shared validations approach looks fantastic to me.
More general comment about the project: The agent SDKs are growing and maturing quickly with new languages supported and this sophisticated testing infrastructure. I think we should step back and revisit the design soon to make sure it incorporates most of the pressing needs that we are aware of before before scaling to more languages or deeper infra.
My two main concerns are:
- APIs to model different inference strategies in a somewhat interchangeable way
- Integration with rendering SDKs (e.g. the "core" SDK) for the purposes of:
- Client orchestrated inference
- Server side rendering, validation, compaction of updates
I just shared draft https://docs.google.com/document/d/1VxcAaUh2sMh14soGVQEkRwz5enA6j4tzDPWJZK9tUqA/edit?resourcekey=0-Ban02LRlWm__B7yOkm8gmA&tab=t.0#heading=h.c0uts5ftkk58 describing the latter
06efb7e to
ea458b1
Compare
I see your point that the underlying architecture would change after integration with the rendering SDK, but I'd still want to get this in to unblock the c++ validator. It also stands as an example on how to set up the test harness. |
ea458b1 to
8dbf517
Compare
…rdized conformance test data files
…on file and remove legacy parser tests.
…idate utility tests
… schema manager test file
…ate test loader paths
…nt in conformance tests
… data directories
8dbf517 to
8493bf8
Compare
…ti-step test definitions
8493bf8 to
19b999e
Compare
Description
Replace this paragraph with a description of what this PR is changing or adding, and why. Consider including before/after screenshots.
List which issues are fixed by this PR. For larger changes, raising an issue first helps reduce redundant work.
Pre-launch Checklist
If you need help, consider asking for advice on the discussion board.