Skip to content

Conversation

@Prathamesh9284
Copy link

Which issue does this PR close?

Closes #508

Rationale for this change

Other Substrait producers (Isthmus, DuckDB) support generating JSON-formatted Substrait plans, which makes it much easier to inspect and compare plans across engines. The substrait crate already provides serde support via the pbjson feature, so DataFusion Python can leverage this to expose JSON plan generation and parsing with minimal effort.

What changes are included in this PR?

  • python/datafusion/substrait.py

    • Added to_json() instance method and from_json() static method on the Plan class for converting plans to/from JSON strings.
    • Added serialize_json() and deserialize_json() static methods on the Serde class for writing/reading JSON plan files.
  • src/substrait.rs

    • Implemented the Rust backend for the above Python methods, including error mapping.
    • Added the following functions:
      • fn to_json(&self) -> PyDataFusionResult<String>
      • #[staticmethod] fn from_json(json: &str) -> PyDataFusionResult<PyPlan>
      • #[staticmethod] pub fn serialize_json(sql: &str, ctx: PySessionContext, path: &str, py: Python) -> PyDataFusionResult<()>
      • #[staticmethod] pub fn deserialize_json(path: &str) -> PyDataFusionResult<PyPlan>
  • src/errors.rs

    • Added SerdeJsonError and DecodeError error variants to PyDataFusionError for proper error handling.
  • Cargo.toml

    • Added serde_json as a dependency.
  • python/tests/test_substrait.py

    • Added tests for both in-memory and file-based JSON plan round-trips.

Are there any user-facing changes?

Yes — four new public API methods:

  • Plan.to_json() -> str — convert a Substrait plan to a JSON string
  • Plan.from_json(json: str) -> Plan — create a Substrait plan from a JSON string
  • Serde.serialize_json(sql, ctx, path) — generate a JSON-formatted Substrait plan file from a SQL query
  • Serde.deserialize_json(path) -> Plan — read a Substrait plan from a JSON file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for generating JSON formatted substrait plan

1 participant