feat: a native SQLAlchemy dialect for Superset

[betodealmeida]

SUMMARY

This PR introduces a new SQLAlchemy dialect, superset://, together with a corresponding DB engine spec. With this, users can create a new database using the superset:// SQLAlchemy URI, and use it to write queries like this:

SELECT * FROM "examples.birth_names";

Queries can also join data from multiple databases, or even move data from one to another:

INSERT INTO "db1.table"
SELECT * FROM "db2.table";

The database can even query itself (not that that's useful):

-- these two are identical
SELECT * FROM "examples.bart_lines";
SELECT * FROM "Superset meta database.examples%2Ebart_lines";

The dialect can only be used if the ENABLE_SUPERSET_META_DB feature flag is enabled, otherwise it will be blocked and won't even show up in the list of available databases.

While the dialect can be use to join across databases, users should be careful with big joins and expensive queries. Filtering, sorting, limiting and offsetting are pushed to the corresponding databases, but aggregations and joins happen in memory. For this reason it's recommended to enable asynchronous queries in the superset:// database, so that computations are executed in Celery workers instead of the web workers.

In addition, it's also possible to limit how much data is read from each database via the SUPERSET_META_DB_LIMIT configuration value, set initially to 1000.

The dialect uses Superset's security manager to prevent users from accessing unauthorized databases. DML is supported as long as DML is enabled in the superset:// database and all the related databases. It's also possible to limit the databases that the dialect has access to, by setting allowed_dbs in the engine parameters. Eg:

{"allowed_dbs":["Google Sheets","examples"]}

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Added unit tests.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[rumbin]

Sounds interesting! Could you provide some possible Use Cases? Is this a step towards paving the path to a LookML-like modeling layer?

[betodealmeida]

Sounds interesting! Could you provide some possible Use Cases? Is this a step towards paving the path to a LookML-like modeling layer?

Yeah, that's my vision. Improving the semantic layer in Superset so we can do things like:

1. Define dimension tables, declare the relationship ("foreign keys") in datasets, and have auto-joins in the Explore view, regardless of where the tables live. There's some performance concerns here, so we'd probably want to limit the cardinality of the dimensions.
2. Move data between DBs. This dialect supports DML, so you can SELECT from a table in one DB and INSERT into a table in another. This could also help materializing datasets/dimensions in different databases to improve JOINs.
3. Build connectors to APIs. This dialect is built on top of shillelagh, which helps building SQL connectors to APIs. We can selectively enable in Superset some of the adapters supported by shillelagh. Imagine using a Google Calendar to annotate events on a time series, for example.

[rumbin]

Thanks for the explanation.
I wonder, if all tables of a query built by explore are accessible by the same DB connection -- e.g. different DBs, but the same Snowflake account -- will this method be smart enough to push down the whole query? Our vision is to integrate all relevant sources into Snowflake, in order to have them available for arbitrary client softwares. So, having a modeling layer would be highly welcome, but we would not want to query and merge from different connections.

BTW, is this the right place for such a discussion, or would Slack or the mailing list be more appropriate?

[betodealmeida]

I wonder, if all tables of a query built by explore are accessible by the same DB connection -- e.g. different DBs, but the same Snowflake account -- will this method be smart enough to push down the whole query?

It would not, but that's an interesting use case. Presto has a similar use case, because we don't support multiple catalogs in a single DB, you need to create a DB per catalog.

[betodealmeida]

OK, I think this is ready for it's annual review! 😆

[betodealmeida]

@danilomo I think this shillelagh could be the basis of creating connectors to non-relational databases. This is what I need at the moment. I wanted to plot data from a non-relational source, but Superset only understand SQLAlchemy dialects.

This might interest you: https://preset.io/blog/accessing-apis-with-superset/

[mdeshmu]

@michael-s-molina do we plan to include this in 3.0?

[michael-s-molina]

@michael-s-molina do we plan to include this in 3.0?

Given the experimental nature of this feature, I think it's a good idea to give it some time to mature. It will likely be included in a minor release.

[john-bodley]

@betodealmeida I left one small comment, but otherwise this LGTM.

[john-bodley]

Would dialects be a more appropriate name that metadb?

[betodealmeida]

Hmmm, good point. I'm not sure if we're ever going to have multiple SQLAlchemy dialects defined in Superset, which is why I named it this way.

[betodealmeida]

[danilomo]

@danilomo I think this shillelagh could be the basis of creating connectors to non-relational databases. This is what I need at the moment. I wanted to plot data from a non-relational source, but Superset only understand SQLAlchemy dialects.

This might interest you: https://preset.io/blog/accessing-apis-with-superset/

Thanks a lot, man!

Edit

"merged"

Wait, what? This is fantastic, lol

Congratulations!

[ozbillwang]

It appears that the latest release, v3.0.1 (which was released two weeks ago), still doesn't include this feature.

However, this feature has been mentioned in https://superset.apache.org/docs/databases/meta-database/

When will we have access to this feature?

If I can't wait, how can I build the application with the latest code? Could you please provide instructions?