Initial setup for federated queries #1350
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial setup for #1318
Implementing federated queries will be a large change - instead of batching it up into one big PR that is hard to review properly, I will instead opt for more frequent, smaller PRs.
To ensure trunk stays in a consistent state, the new federation code is gated behind a
federation-experimental
feature flag that is disabled by default. A new Makefile targetinstall-with-federation
is added to build Spice with the federation capability enabled.If the
federation-experimental
flag is enabled, we extend the DataFusion SessionContext with theFederationQueryPlanner
andFederationAnalyzerRule
from thedatafusion-federation
crate. This won't actually do anything yet, since we haven't registered any of our tables as federated (from the point of view of thedatafusion-federation
crate).For now, I'm working out of my own private fork https://github.com/phillipleblanc/datafusion-federation. This is to enable rapid iteration, and facilitate easy PRs to the upstream source. My goal is by the time we mark this feature as non-experimental, this will be pointing to the upstream repo and not to my fork.