Drivers available on PyPI

[erichards97]

Hi,

1. Is installing this package equivalent to dbc install snowflake? Albeit I assume the former installs a copy in your Python env and the latter is system-wide?
2. Why are some drivers, such as Snowflake, available via PyPI, while others, such as MSSQL, are not?
3. Is dbc the most standard method for driver installation?

Thanks

[ianmcook]

Hi @erichards97, thanks for the great questions. I’ll answer in more detail than you might need because this will be useful for others who find this discussion later.

ADBC drivers are fundamentally not language-specific. They are native shared libraries (.so, .dylib, .dll) that can be dynamically loaded by ADBC driver managers. That means the same driver binary can be used from Python, R, Rust, Go, Java, Node.js, and other languages. We've put a lot of effort into making this very easy, for example by canonicalizing the driver install locations so you never need to think about file paths when you're using the driver managers to load drivers. For more context, see the ADBC quickstarts at github.com/columnar-tech/adbc-quickstarts, and the dbc getting-started docs at columnar.tech/dbc. If any of this terminology is confusing, see columnar.tech/glossary which defines all these terms.

To your specific questions:

1. Going forward, dbc install snowflake is the path we recommend. Installing adbc-driver-snowflake from PyPI is a legacy Python-specific distribution path. It installs a copy of the driver into your Python environment, and it exists because that is how some ADBC drivers were distributed historically. dbc install snowflake, on the other hand, installs the driver from the ADBC driver registry. That is the distribution model we are moving toward: drivers are published once as native binaries and can then be used from any ADBC-compatible language/runtime that can load them, rather than being repackaged separately for each language ecosystem.

2. The language-specific driver packages on PyPI, CRAN, and other package repositories are largely a legacy of how ADBC drivers were distributed earlier on. Over time, that patchwork distribution model became difficult to maintain: every driver needed language-specific packaging work, CI, release processes, platform-specific fixes, and package-repository-specific requirements, even though the driver itself is just a native shared library. That maintenance overhead was one of the main reasons we built dbc and the driver registry. Going forward, new drivers generally should not be expected to appear as language-specific packages across all language package managers. The preferred distribution path is the driver registry at dbc-cdn.columnar.tech, with dbc as the main installation tool.

3. Yes, dbc is the standard installation method we recommend. That said, dbc is not magic or mandatory: the registry is a public CDN with a YAML index that points to downloadable driver binaries. The CLI is often the easiest way to install drivers, but we are also working on more programmatic installation options, including dbc libraries, starting with Go, as well as a dbc GitHub Action. See the dbc 0.3.0 announcement for more details: columnar.tech/blog/announcing-dbc-0.3.0.

We are also working on a guide that explains the background and rationale in more detail. The first few paragraphs of this draft are intended to clarify exactly this point: https://github.com/adbc-drivers/docs.adbc-drivers.org/pull/70/changes#diff-f5bc4b80e1897a1c551080bc26b18016ff52c70fc1216090a3cc9fa957df38c3R18. We'd love your feedback on whether that explanation is clear.

[ianmcook]

Another thing to consider is security and provenance. For every driver listed in the ADBC driver registry, we know who the authors are, we vet that they work for the companies they claim to represent, we canonicalize the driver entries, and we coordinate with the relevant vendors whenever possible.

By contrast, anyone, including a bad actor, could upload a package to PyPI with a name like adbc-driver-mydatabase, and we have no way to prevent that. Unfortunately, people and agents may come to blindly trust that this PyPI package naming convention identifies safe, official, or vendor-approved ADBC drivers, when it does not.

[erichards97]

Thank you for the detailed reply! A few suggestions below.

• Perhaps the packages column on the first table on this page should be removed, given what you've said above?

• It might be useful to list the creators/maintainers of each of these drivers in that same table. E.g. I'd be interested if the Snowflake driver has been contributed by Snowflake themselves or a third-party.

• It might be useful to document the repos that dbc connects to for the sake of corporate environments with restricted internet access, where these repos will either need to be allowed or proxied through something like Artifactory. Edit: the docs do mention the default Columnar repo, although I don't think they make it very clear on how to point to a different repo (dbc auth, perhaps?)

• I saw that after having run install, dbc created a folder structure and some yaml config files under my home directory, which wouldn't be immediately obvious without having run dbc in the first place. Therefore:

  a) It might also be useful to add an option to install a local package (i.e. dbc install --local /path/to/snowflake-1.2.3.tar.gz), for those with limited access; or

  b) If the "offline" installation steps given a .tar.gz bundle could be documented, that would work instead. Edit: this may already be covered by dbc init and the docs on driver lists - I'll give it a go

[lidavidm]

I think this all makes sense!

We are working on documenting the CDN structure better. Installing a "local" package already works, though, if you download the package yourself! (Just dbc install the .tar.gz from the CDN.)

We are also working on updating those docs - we kept both the upstream and forked drivers around for a while and will finally fully drop the old packages in the next few releases.