Prepare Ballista crates for publishing

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
It is currently not possible to publish the Ballista crates to crates.io from a DataFusion source release because Ballista has dependencies based on relative path only. For example, the ballista-client crate has this dependency:

datafusion = { path = "../../../datafusion" }

We need to update it to include the version number as well:

datafusion = { path = "../../../datafusion", version = "4.0.0-SNAPSHOT" }

We will also need to update the release scripts to update these version numbers as part of the release process.

Describe the solution you'd like

• Add version to each relative dependency
• Implement a release script to update these versions. In the monorepo we used a Python script with regex search and replace. We may want to consider a different solution now? Perhaps cargo has plugins to help with changing dependency versions?
• Update documentation

Describe alternatives you've considered
None

Additional context
None

[andygrove]

@alamb @jorgecarleitao fyi would be good to get your input on this

[alamb]

Sounds reasonable to me.

At some point, we should probably figure out a more regular release cadence for DataFusion (right now, my project is just using the head directly from git, but that isn't appropriate for ballista).

[jorgecarleitao]

Do you have plans about what you would like Ballista to be? I.e. if we release Ballista as a binary (e.g. in artifactory, or in a docker image), we do not need DataFusion released, as we can point to a github hash. The limitation emerges when placing it in crates.io.

I think that the same applies for other consumers of DataFusion. For example, the Python bindings do not require DataFusion in crate.io: we can build the wheels and ship them to pypi.

Regardless, I agree with @alamb with the cadence, as some consumers may want to ship libraries built on top of datafusion.

@alamb , I am curious: do you need datafusion released for iox? I imagine that if you ship it as a binary or docker, you can point to the hash, right?

[alamb]

@alamb , I am curious: do you need datafusion released for iox? I imagine that if you ship it as a binary or docker, you can point to the hash, right?

@jorgecarleitao -- The short answer is no -- we use a git hash in IOx for datafusion

And I don't think we expect anyone to use IOx as a Rust library so I don't think it is likely we would publish it to crates.io -- anything that we think is useful / reusable to others we are trying to contribute back to arrow or datafusion

[andygrove]

@houqp I'm not sure if this issue is still valid or not?

[houqp]

@andygrove i think you already fixed this one :)