Using entry points for loading backends

[datapythonista]

With entry points, the code is not only cleaner, but this will allow that third-party backends are used in the same exact way as packages deployed with Ibis. For example:

import ibis

ibis.mssql.connect(...)

This will work if ibis-mssql is installed in the environment.

This change also allows to easily move backends out of Ibis, for example OmniSci (see #2356). Having the backend outside of this repo will be backward compatible and no change to the user code will be needed.

[jreback]

can you rebase

[datapythonista]

There is something very strange going on here. Apparently, in some parts of the code, when using import ibis; ibis.<backend>, instead of calling __getattr__ of ibis/__init__.py, the result is the module ibis/<backend>/__init__.py.

I think the responsibly is pytest. I'm not sure, but I think it does a lot of Python internal monkeypatching, and that's the only reason I can find.

Regardless of this error, I don't think it's a great idea that ibis.pandas refers to ibis/pandas/api.py and not the ibis/pandas module. Quite misleading. I'll do a test and see if moving one of the affected backends solves the problem (ibis/pysparks -> ibis/backends/pyspark/). If it does, I'll start moving the backends to that directory. Which I think makes a lot of sense anyway, and I was planning to do at some point.

[jreback]

There is something very strange going on here. Apparently, in some parts of the code, when using import ibis; ibis.<backend>, instead of calling __getattr__ of ibis/__init__.py, the result is the module ibis/<backend>/__init__.py.

I think the responsibly is pytest. I'm not sure, but I think it does a lot of Python internal monkeypatching, and that's the only reason I can find.

Regardless of this error, I don't think it's a great idea that ibis.pandas refers to ibis/pandas/api.py and not the ibis/pandas module. Quite misleading. I'll do a test and see if moving one of the affected backends solves the problem (ibis/pysparks -> ibis/backends/pyspark/). If it does, I'll start moving the backends to that directory. Which I think makes a lot of sense anyway, and I was planning to do at some point.

yep this all sounds good.

[jreback]

@datapythonista looks like needs a rebase.

[datapythonista]

@datapythonista looks like needs a rebase.

Rebased. Let's see if moving the omnisci backend fixed the strange pytest error. If that's the case, we'll still have to move the spark backends (its CI was also red).

[jreback]

dont' we need to rebuild the actual conda packages first?

[jreback]

i think if you sync this now ,then the updates for the other backends can update this as needed

[pep8speaks]

Hello @datapythonista! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-02-24 19:23:33 UTC

[datapythonista]

@jreback this is green now, and should be ready.

[jreback]

cool, this doesn't actually change the user behavior right?

[datapythonista]

cool, this doesn't actually change the user behavior right?

Not at all. It's just an internal change that lets third party packages to also register backends. User API is exactly the same, that's why there are no changes to tests and all is green.

[jreback]

great and actually we dont' import things pre-emptively now, so import times should be faster?

[datapythonista]

great and actually we dont' import things pre-emptively now, so import times should be faster?

We still load all backends at init unfortunately. I wanted to do that, but the problem are the backend specific options. If the backend is loaded when we do ibis.sqlite, that's when the options would be loaded, and if you set them before with a with for example, that would fail.

I think we can find a solution, with small API changes, but will come later.