Is your feature request related to a problem or challenge?
My ultimate goal is to be able to benchmark the Vortex/DataFusion integration in a consist and reproducible way, especially for ClickBench.
While we have our own tooling, it tends to do through many changes and is not always a perfect match to what DataFusion does, which ends up creating unexpected differences in results.
Describe the solution you'd like
I would like to be able to use the datafusion-cli crate to build our own tool, which should be as close as possible to the existing CLI, just with the ability to modify the session. I think it will be useful for us to include our FileFormat stuff, but I can imagine it being useful to test any other extension point in DataFusion that isn't part of the main repo.
Describe alternatives you've considered
For the time being, I've just created https://github.com/vortex-data/vortex-datafusion-cli which should also help with our versioning, but could be more focused if there's positive response to this idea.
Is your feature request related to a problem or challenge?
My ultimate goal is to be able to benchmark the Vortex/DataFusion integration in a consist and reproducible way, especially for ClickBench.
While we have our own tooling, it tends to do through many changes and is not always a perfect match to what DataFusion does, which ends up creating unexpected differences in results.
Describe the solution you'd like
I would like to be able to use the
datafusion-clicrate to build our own tool, which should be as close as possible to the existing CLI, just with the ability to modify the session. I think it will be useful for us to include ourFileFormatstuff, but I can imagine it being useful to test any other extension point in DataFusion that isn't part of the main repo.Describe alternatives you've considered
For the time being, I've just created https://github.com/vortex-data/vortex-datafusion-cli which should also help with our versioning, but could be more focused if there's positive response to this idea.