ENH: Load SQL engine backends via entrypoints

[yehoshuadimarsky]

Follow up to #36893. Per @xhochy's suggestion we should use entrypoints as a mechanism for loading alternate SQL engine backends, similar to what we do with plotting backends. Creating a new issue for it per the comment here.

[yehoshuadimarsky]

@xhochy can you please clarify for me, in this scenario that we will load SQL engine backends via entrypoints, what exactly does the SQL backend library need to provide to be compatible? I see you wrote here that it needs to add this to its setup.{py|cfg}

entry_points={"pandas_sql_engine": ["turbodbc = turbodbc.TurbodbcPandasEngine"]}

but what does turbodbc.TurbodbcPandasEngine actually need to implement?

[yehoshuadimarsky]

Actually some good reading here:

• https://packaging.python.org/specifications/entry-points/
• https://packaging.python.org/guides/creating-and-discovering-plugins/
• https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html

Appears that the group portion should be delimited by dots, not underscores, e.g.

entry_points={"pandas.sql.engine": ["turbodbc = turbodbc.TurbodbcPandasEngine"]}

[yehoshuadimarsky]

Looking around, we can potentially reuse some of the plugin approaches that other popular libraries use. Some examples that come to mind:

Pytest

• https://docs.pytest.org/en/6.2.x/writing_plugins.html
• https://github.com/pytest-dev/pytest/search?q=pytest11
• https://pluggy.readthedocs.io/en/latest/
  Looks like they use their own package Pluggy for this, which has a convenient built in method for loading setuptools entrypoints here.

Flake8

• https://flake8.pycqa.org/en/latest/plugin-development/registering-plugins.html

[yehoshuadimarsky]

And maybe we should centralize all plugin loading into a single module with common functions, and that can be invoked from the specific places that need it like plotting, sql engine, etc. Would make adding more plugins easier

[jreback]

happy to get something to work then move to a more formal plugin structure

[yehoshuadimarsky]

Sounds good, working on it. @xhochy how do we enforce that the sql engine has the correct interface? Or do we not do so, we just let the user select an engine and it's up to the engine library to create a function that has the correct signature, and if it fails, it fails? Should an engine library import the base abstract SqlEngine class to subclass from, or is that too strong of a dependency?

Trying to understand what the expectations are for these kinds of loosly coupled dependencies...

[yehoshuadimarsky]

take

[yehoshuadimarsky]

Sounds good, working on it. @xhochy how do we enforce that the sql engine has the correct interface? Or do we not do so, we just let the user select an engine and it's up to the engine library to create a function that has the correct signature, and if it fails, it fails? Should an engine library import the base abstract SqlEngine class to subclass from, or is that too strong of a dependency?

Trying to understand what the expectations are for these kinds of loosly coupled dependencies...

@xhochy surfacing this again, are you able to provide some guidance around this?