Skip to content

SQLAlchemy and Snowflake connectors#404

Merged
mildbyte merged 10 commits intomasterfrom
feature/sqlalchemy-connectors
Mar 11, 2021
Merged

SQLAlchemy and Snowflake connectors#404
mildbyte merged 10 commits intomasterfrom
feature/sqlalchemy-connectors

Conversation

@mildbyte
Copy link
Copy Markdown
Contributor

Add some machinery to support SQLAlchemy connectors via a customized version of the FDW (that allows users to get tables to point to subqueries on the remote database to get around pushdown limitations and syntax constructs / operations that aren't available in PostgreSQL).

As a showcase, ship a Snowflake data source in this PR:

$ sgr mount snowflake test_snowflake -o@- <<EOF
{
    "username": "username",
    "password": "password",
    "account": "acc-id.west-europe.azure",
    "database": "SNOWFLAKE_SAMPLE_DATA",
    "schema": "TPCH_SF100"
}
EOF

$ sgr mount snowflake test_snowflake_subquery -o@- <<EOF
{
    "username": "username",
    "password": "password",
    "account": "acc-id.west-europe.azure",
    "database": "SNOWFLAKE_SAMPLE_DATA",
    "tables": {
        "balances": {
            "schema": {
                "n_nation": "varchar",
                "segment": "varchar",
                "avg_balance": "numeric"
            }
            "options": {
                "subquery": "SELECT n_nation AS nation, c_mktsegment AS segment, AVG(c_acctbal) AS avg_balance FROM TPCH_SF100.customer c JOIN TPCH_SF100.nation n ON c_nationkey = n_nationkey"
            }
        }
    }
}
EOF

mildbyte added 10 commits March 11, 2021 17:12
…or PG/MySQL connectors), don't set credentials to None (omit them instead).
…xtracting them from the JSON params in sgr mount (rather than just username/pw).
  * Had to install sqlalchemy-snowflake in the engine outside of the standard Python machinery (not required in sgr)
  * WHERE clauses are pushed down but joins aren't because of Multicorn. Add an escape hatch in the form of a custom `subquery` parameter that turns a remote query to a mounted table.
…he subquery is set, it can have the schema inside of it).
…onversion), remove the CSV FDW from coverage (is tested inside of the engine)
@mildbyte mildbyte merged commit beff612 into master Mar 11, 2021
@mildbyte mildbyte deleted the feature/sqlalchemy-connectors branch March 11, 2021 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant