Circular import when loading external backends

[datapythonista]

Looks like we've got a problem I haven't anticipated (and not sure why I didn't face this before while updating the OmniSci backend). We're currently loading all the backend found on entrypoints on import ibis. But backends also need to import ibis, so, there is a circular import when loading external backends.

The obvious solution would be to implement __getattr__ for the ibis module. So, on ibis.sqlite the backend would be loaded lazily. The problem with that is shown in the next code and there is #2671 for it:

import ibis

ibis.options.impala.temp_db = 'foo'

client = ibis.impala.connect(...)

We need to load the imapala backend on import ibis, if we do in ibis.impala.connect(...) the impala options are not defined, and ibis.options.impala.temp_db = ... fails. Loading the backend on options could be an option but seems too hacky to me.

So, in my opinion, we should take care of #2671 before the release (that would involve breaking changes, not just raising a FutureWarning). And I'd also implement #2532 which completely solves the problem, and backends would only be loaded after they are needed by an ibis.connect(...) call.

@jreback does this make sense to you? Any other idea?

[jreback]

we need to be able to set options w/o the connection parameter, that's sort of the entire point. I think loading on the options is ok here and would solve the problem.

I am not sure we want to break the world before 2.0

[gerrymanoim]

I think I'm running into this same issue trying to path ibis_bigquery to work with the most recent ibis refactor. Very rough WIP where I was just trying to get tests running: gerrymanoim/ibis-bigquery@d98d812.

I end up in weird half initialized ibis import errors.

[datapythonista]

I'm working on it, with the approach that Jeff mentioned. I should have a PR ready later today.