Spark Connect extension support in Sail

[SemyonSinchenko]

SparkConnect protocol provides four abstractions to work with extensions and the corresponding protobuf message:

• google.protobuf.Any extension = 998;
• CommandPlugin
• ExpressionPlugin
• MLBackendPlugin
• RelationPlugin

Does the Sail project supports anyhow this plugin system?

For example, there is project for graphs analytics on top of Apache Spark, graphframes. While the core is written is scala, it already supports the SparkConnect via extensions: https://github.com/graphframes/graphframes/blob/master/graphframes-connect/src/main/protobuf/graphframes.proto

If Sail provides an abstraction for extensions, it may be interesting for me to write an alternative implementation of GraphFrames core in Rust and DataFusion and I would like to work on it. So, if extensions are already supported, maybe there are some examples of how to use them? GF is fully relying on the Relation plugin at the moment, but Command plugin may be important too to allow users to set some settings.

Thanks in advance!
P.S. It is very low-priority question and at the moment is mostly about a theoretical possibility.

[linhr]

Thanks for your interest in Sail!

Sail does not support the plugin system yet, but I feel it's possible since all the client-server communication still happens via gRPC. I love your idea of implementing the backend in Rust and DataFusion for GraphFrames. If you're interested in contributing, I'd be happy to chat more and let me know if there is anything I can help with!

[SemyonSinchenko]

Thanks for your interest in Sail!

Sail does not support the plugin system yet, but I feel it's possible since all the client-server communication still happens via gRPC. I love your idea of implementing the backend in Rust and DataFusion for GraphFrames. If you're interested in contributing, I'd be happy to chat more and let me know if there is anything I can help with!

How may such an extension look like if we assume that Sail supports extensions? In Spark it is just calls to DataFrame APIs (with RDDs, custom expressions, etc.), but from what I can understand, Sail uses its own planner, shuffling, etc. Which APIs can I use for a hypothetical planner? For example, would it be enough to just rewrite GraphFrames to pure DataFusion?

For example, in spark the Relation plugin has signature Optional<LogicalPlan> transform(byte[] relation, SparkConnectPlanner planner);

May in that case the Sail's potential plugin has a signature like fn transform(bytes, ...) -> Result<LogicalPlan, Error>?

[linhr]

In Sail, converting from Spark Connect gRPC messages to DataFusion logical plans is a two-step process.

First, the gRPC message is converted to a Sail internal specification ("spec") defined here.

Second, the Sail "spec" is resolved as a DataFusion logical plan by the resolver implemented here. The resolver does the heavy-lifting of semantic analysis etc.

I guess the extension would fit into this process, and the core logic to mimic GraphFrames would be in the second step (plan resolver).

[linhr]

This discussion thread covers extension support for Spark Connect specifically. Please feel free to continue sharing ideas!

For general topics around Sail extensions, we can use this thread: #2001