Matrix Globe: structured review claims through the Python SDK, checked by failing tests

[osantanaleal-ship-it]

I work on MatrixAgentNet. For the Meta Global AI Developer Hackathon we built Matrix Globe, an MIT-licensed code-review experiment that uses the OpenAI Python SDK's Chat Completions client with Meta's API endpoint. The results below concern Meta models; they are not an OpenAI model evaluation or an endorsement.

The distinction we wanted to make explicit: a schema-valid review is not yet an evidenced defect. Our wrapper requests response_format: json_schema with strict: true, validates the response with Pydantic, and passes each reported defect to a separate sandbox check. A claim only advances as reproduced when its supplied test fails on the candidate code. Collection failures and timeouts have their own invalid outcome.

Some implementation details that may be useful to other SDK users:

• Invalid JSON or a Pydantic validation failure gets one logical repair request. A second parse failure raises our InvalidJSONError.
• An empty response with finish_reason="length" raises our TruncatedResponseError, rather than looking like an empty successful review.
• Usage is recorded before parsing, so an invalid generated response is still counted in the cost ledger. The repair path checks the application's budget again.
• Schema validation and sandbox execution are separate checks; neither a parse success nor a bundle hash establishes that the code is correct.

Wrapper source · Existing wrapper tests.

Before sharing this, I reran those 11 wrapper tests and checked four additional cases through the real SDK with httpx.MockTransport: valid structured response, twice-invalid JSON, empty truncated response, and HTTP 429 followed by success. All passed with Python 3.11.15, openai 3.39.0 and httpx 0.28.1. The transport check set SDK retries to zero so it could observe the wrapper's retry behavior. These were offline checks with synthetic responses and zero model calls; they do not establish live provider compatibility.

The separate published benchmark used the same Muse Spark Open model in all roles: 57/60 buggy variants repaired with the proof rule versus 41/60 for free-text review and 54/60 for self-checking, with all three preserving the 20 clean controls. These are our FinCode-30 results, not a general claim across models. The recorded Matrix MCP demo is explicitly PARTIAL: three defects repaired, a fourth found, then the time limit reached. We operated the demo agents.

The Matrix Globe write-up includes the short demo, full comparisons, costs and 902 inspectable proof bundles. I'd welcome concrete failure cases for the boundary between a structurally valid review and a useful executable counterexample.

Disclosure: I operate MatrixAgentNet; this post is AI-assisted and was checked against the published source, results and the offline runs described above.