Adding a `--pre` option for install/add/sync

[zeroshade]

Similar to the --pre option when using pip/uv we want to allow installing pre-releases of drivers via dbc. Both uv and pip have different behaviors for this so we should decide how we want to handle this.

pip's behavior is very simplistic: Without the --pre option, pre-release versions of packages are never considered and essentially don't exist. The benefit here is that it's very simple to implement via filtering versions when reading package metadata.

uv's behavior provides more options. Via either the --prerelease option or via a UV_PRERELEASE environment variable, one of the following options can be chosen to control the behavior:

• disallow: never allow pre-release versions (the same as pip without the --pre option)
• allow: allow pre-release versions, e.g. treat them as normal versions for consideration. This would mean that the "latest" version could be a pre-release (1.1.0-beta1 > 1.0.0)
• if-necessary: allow pre-release versions if a package only has pre-release versions, ignore them otherwise
• explicit: allow a pre-release version if explicitly asked for, ignore them otherwise
• if-necessary-or-explicit: allow a pre-release version if there are only pre-release versions for a package or if the pre-release version is explicitly asked for

My personal preference is to follow pip's behavior, but I'm not opposed to mirroring the behavior of uv. Thoughts?

[amoeba]

What are the current and future use cases? I think these are current:

1. Let driver authors publish pre-release versions their users can install with dbc
2. Let us completely and seamlessly replace a driver with another using the same path but do it as a pre-release

I can see how just a simple --pre handles (1) nicely. We publish the pre-release versions to the main driver registry index and dbc gains the smarts to handle pre- and non-pre versions.

(2) I'm not so sure about since the metadata changes. Do we have to punt on (2) for today and do something different for that?

[Mandukhai-Alimaa]

Can we combine pip's behavior with uv’s "if-necessary-or-explicit" logic? If we went with a strict pip mirror, dbc install would just error out if only pre-releases existed. By adding this logic, it finds the driver even without the --pre flag if there’s no stable alternative. I think this approach is simple enough for users to control whether they want pre-releases, while also preventing unnecessary errors when a pre-release is the only option for a specific driver.

[ianmcook]

@Mandukhai-Alimaa My thinking is that we should hide pre-release drivers if the user does not specify --pre. dbc install wouldn't install them and dbc search wouldn't show them. That allows us to use --pre as a kind of "early access channel" for drivers that we don't yet have enough confidence in.

In other words: my thinking is that if you do dbc install foo but the only available versions of driver foo are pre-releases, then dbc should fail and not install anything. (But maybe it could print a nice error message saying that a pre-release is available by using --pre.)

[ianmcook]

@amoeba I'm not sure I understand what you mean by this:

2. Let us completely and seamlessly replace a driver with another using the same path but do it as a pre-release

Can you elaborate please? What's the series of steps the user takes?

[amoeba]

If someone writes an SQLite ADBC driver in Rust and we like it enough to make it installable with dbc install sqlite, we could just wholesale replace the sqlite entry in the index with the information for the new driver. But if we wanted to let users try it while in beta, we wouldn't be able to tell them to run dbc install --pre sqlite because of our index structure.

[zeroshade]

I'm not so sure about since the metadata changes. Do we have to punt on (2) for today and do something different for that?

How does the metadata change? It's just a version change?

[zeroshade]

But if we wanted to let users try it while in beta, we wouldn't be able to tell them to run dbc install --pre sqlite because of our index structure.

We could tell them to run dbc install --pre sqlite if we push the new rust version as a pre-release for a new version of the sqlite driver.

[ianmcook]

A few other notes / thoughts:

• From my perspective, I think we can keep this simple and follow pip's behavior and pip's interface (just --pre). I don't see the practical benefit of adding more complexity (like what uv provides) at this stage. Does anyone else?
• If we keep it simple for now, later could we extend this so that:
  • dbc install driver is an alias for dbc install driver --pre disallow (the default behavior)
  • dbc install driver --pre is an alias for dbc install driver --pre allow
  • we also allow users to explicitly specify --pre disallow, --pre allow, --pre if-necessary, etc?
• I don't think dbc sync needs to take --pre. All the information about whether dbc sync can install pre-releases should be encoded in the driver list and lock file. We will need to add an optional field to the driver list to store this.

[ianmcook]

One caveat to my comment above:

I tend to think that

dbc install driver=X.Y.Z-beta.1

(without --pre)

should succeed (like uv's explicit behavior).

not fail (like uv's disallow behavior).

I think maybe explicit is better. If the user is explicitly saying they want version X.Y.Z-beta.1 then surely they know it's a pre-release, so why fail? Thoughts?

[amoeba]

How does the metadata change? It's just a version change?

Every field in the index entry changes except pkg_info unless I'm misunderstanding. (name, description, license, urls, docs_url, etc, etc can change). If this isn't in scope that's fine, I'm not sure it has to be for today.

[ianmcook]

For the new field in the driver list, how about we use uv's option values, like this:

[drivers]

[drivers.mysql]

[drivers.duckdb]
prerelease = 'allow'

[drivers.postgresql]
version = '=1.2.3-beta2'

The default value of prerelease is explicit, so this this list would install pre-releases versions of the drivers for both DuckDB (if it were present and the newest version) and PostgreSQL.

[zeroshade]

Every field in the index entry changes except pkg_info unless I'm misunderstanding. (name, description, license, urls, docs_url, etc, etc can change). If this isn't in scope that's fine, I'm not sure it has to be for today.

In theory the name and description shouldn't change in the scenario we're describing. the urls/docs_url could change, hopefully not the license. If the name would change, then we should do an entirely separate driver, not as a pre-release. That's my opinion at least

[ianmcook]

If someone writes an SQLite ADBC driver in Rust and we like it enough to make it installable with dbc install sqlite, we could just wholesale replace the sqlite entry in the index with the information for the new driver. But if we wanted to let users try it while in beta, we wouldn't be able to tell them to run dbc install --pre sqlite because of our index structure.

I don't think we should ever wholesale replace like this. We should just push new drivers with incremented version numbers—initially prerelease version numbers, and later release versions numbers that are higher than the previous versions' numbers.

Or if we're pushing a driver that needs to update the metadata (e.g. because it's from a different source), then we should do it as a separate driver with a different name (like Matt says above).