Use JSON instead of YAML for typespec-python code model serialization#11178
Use JSON instead of YAML for typespec-python code model serialization#11178timotheeguerin wants to merge 3 commits into
Conversation
…ialization Switch the emitter<->generator code model interchange format from YAML to a graph-preserving JSON codec (flatted). The code model is a cyclic/shared object graph, so a plain JSON format is insufficient; the flatted codec preserves cycles and shared references while being far faster than YAML. - Add pygen/_codemodel_json.py (Python port of the flatted format) - Remove js-yaml (emitter) and PyYAML (generator) dependencies - Rename the emit-yaml-only option to emit-codemodel-only On the Azure network spec, end-to-end serialization drops from ~29s to ~0.2s and the code model file shrinks from 14.4MB to 6.5MB.
microsoft-github-policy-service
Bot
added
the
emitter:client:python
commit: |
|
All changed packages have been documented.
Show changes
|
|
You can try these changes here
|
Replace the flatted wire format with the same $id/$ref reference-preserving
convention the C# emitter uses (System.Text.Json ReferenceHandler.Preserve):
objects become {"$id": "N", ...}, arrays become {"$id": "N", "$values": [...]},
and repeated/cyclic references become {"$ref": "N"}.
Unlike the C# writer, cycles are preserved (not dropped) because the Python
code model genuinely contains them. Validated on the Azure network spec: the
reconstructed graph is structurally identical (35175 objects, 32075 arrays,
3739/2855 shared, 1022 cycle edges) and round-trips stably.
- Add emitter/src/code-model-serializer.ts ($id/$ref serializer)
- Rewrite pygen/_codemodel_json.py loads/dumps for the $id/$ref dialect
- Drop the flatted dependency
# Conflicts: # packages/http-client-python/emitter/src/emitter.ts # packages/http-client-python/emitter/src/external-process.ts
|
This PR will cause some code diff in generated sdk code. Most are just order change but there is one change https://github.com/Azure/azure-sdk-for-python/pull/47940/changes#r3542560807 that may need more investigation. |
|
yeah @msyyc this was just a POC to investigate gains but definitely will need to be designed/implemented more seriously if you want to move in that direction |
2 participants
Summary
Switches the code model interchange format between the emitter and the Python generator in
@typespec/http-client-pythonfrom YAML to a reference-preserving JSON format.To stay consistent with the C# emitter, this uses the same
$id/$refconvention (System.Text.Json'sReferenceHandler.Preserve):{ "$id": "N", ...properties }{ "$id": "N", "$values": [...] }{ "$ref": "N" }The code model is a cyclic / shared object graph (on the Azure network spec: 35,175 objects and 32,075 arrays, of which 3,739 objects and 2,855 arrays are shared, plus 1,022 cycle back-edges). Because
$idis registered before recursing into a node's children, cycles are encoded as a$refback to the enclosing node — so cycles are preserved (the C# writer drops them, which the Python model can't tolerate). The Python decoder reconstructs a graph with identical shared identity and cycles, so the generator's model layer needs zero changes.Changes
emitter/src/code-model-serializer.ts— the$id/$refserializer (cycle-safe).generator/pygen/_codemodel_json.py— matching Pythonloads/dumpsdecoder/encoder.js-yaml/@types/js-yaml(emitter) andPyYAML/types-PyYAML(generator) dependencies.emit-yaml-onlyemitter option toemit-codemodel-only.ARCHITECTURE.md.Performance
Benchmarked on the Azure
networkspec (large real-world spec):$id/$ref)End-to-end serialization: ~29 s → ~0.2 s.
Validation
$id/$refcode model → generator produced 31 correct.pyfiles ✅