Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake support #1735

Closed
mbreault opened this issue Aug 6, 2017 · 14 comments
Closed

Snowflake support #1735

mbreault opened this issue Aug 6, 2017 · 14 comments

Comments

@mbreault
Copy link

@mbreault mbreault commented Aug 6, 2017

Snowflake is a hosted data warehousing solution.

Home: https://www.snowflake.net/
Driver: https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc/

Currently get: ERROR: Unsupported Database: Snowflake 2.5

@alexlevene

This comment has been minimized.

Copy link

@alexlevene alexlevene commented Oct 10, 2017

Added support for Snowflake to a fork of v4.2.0 and submitted a pull request #1776. We've put it through its paces in development, but it would be great to get some more feedback. Please review.

@axelfontaine axelfontaine changed the title Feature Request: Support for Snowflake Snowflake support Nov 15, 2017
@axelfontaine axelfontaine added this to the Someday-Maybe milestone Nov 27, 2017
@bobtiernay-okta

This comment has been minimized.

Copy link

@bobtiernay-okta bobtiernay-okta commented Jul 24, 2019

Any update on this? Would be great to have Snowflake support.

@rjungwirth

This comment has been minimized.

Copy link

@rjungwirth rjungwirth commented Sep 5, 2019

I took over ownership of pr #1776 when alex left the company. We've been using this feature branch ever since and it's working flawlessly. Please re-consider it for inclusion.

If I were to port this code to the V6 branch would it have a better chance?

@juliahayward

This comment has been minimized.

Copy link
Member

@juliahayward juliahayward commented Sep 13, 2019

We're going to talk to Snowflake about the commercial details of a testing account for us. I'll let you know the outcome of that; if good, then a port to v6 would be very welcome.

@blian7432

This comment has been minimized.

Copy link

@blian7432 blian7432 commented Sep 20, 2019

Hi @juliahayward Do you have an ETA of Snowflake support?

@juliahayward

This comment has been minimized.

Copy link
Member

@juliahayward juliahayward commented Sep 23, 2019

Not yet, we're still setting up a meeting with them.

@juliahayward

This comment has been minimized.

Copy link
Member

@juliahayward juliahayward commented Nov 7, 2019

We're now working on Snowflake for 6.1. I have a question for people who are using it; in a new database connection without a schema specified in the URL, SELECT CURRENT_SCHEMA() returns null; what would you all expect Flyway's behavior to be:

  • you would specify a current schema in the connection URL
  • you would expect Flyway to default to PUBLIC until specified otherwise by USE SCHEMA xxx
  • you would fully qualify all object names in migration scripts?
@rjungwirth

This comment has been minimized.

Copy link

@rjungwirth rjungwirth commented Nov 7, 2019

@bobtiernay-okta

This comment has been minimized.

Copy link

@bobtiernay-okta bobtiernay-okta commented Nov 7, 2019

This is what we've done to get this to work:

/**
 * Snowflake implementation of a Flyway Connection.
 */
@SuppressWarnings("rawtypes")
@SuppressFBWarnings("SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING")
public class SnowflakeConnection extends Connection<SnowflakeDatabase> {

    private static final Log LOG = LogFactory.getLog(SnowflakeConnection.class);

    private final String originalRole;

    public SnowflakeConnection(Configuration configuration, SnowflakeDatabase database, java.sql.Connection connection, boolean originalAutoCommit) {
        super(configuration, database, connection, originalAutoCommit);
        LOG.debug("Creating new SnowflakeConnection");

        try {
            this.originalRole = jdbcTemplate.queryForString("SELECT CURRENT_ROLE()");
        } catch (SQLException e) {
            throw new FlywaySqlException("Unable to determine current role", e);
        }
    }

    //
    // NOTE: the following methods are overridden to provide an implementation
    //

    @Override
    protected String getCurrentSchemaNameOrSearchPath() throws SQLException {
        String result = jdbcTemplate.queryForString("SELECT CURRENT_SCHEMA()");
        LOG.debug("Current Snowflake schema is " + (result != null ? result : "none"));
        return requireNonNullElse(result, "PUBLIC");
    }

    @Override
    public Schema getSchema(String name) {
        return new SnowflakeSchema(jdbcTemplate, database, name);
    }

    //
    // NOTE: the following methods are overridden to change the implementation
    //

    @Override
    protected void doRestoreOriginalState() throws SQLException {
        // Reset the role to its original value in case a migration or callback changed it
        jdbcTemplate.execute("USE ROLE " + originalRole);
    }

    @Override
    protected void doChangeCurrentSchemaOrSearchPathTo(String schemaNameOrSearchPath) throws SQLException {
        LOG.debug("Switching to Snowflake schema " + schemaNameOrSearchPath);
        jdbcTemplate.execute("USE SCHEMA " + database.quote(schemaNameOrSearchPath));
    }

}

and

/**
 * Snowflake implementation of Flyway Schema.
 */
@SuppressWarnings("rawtypes")
@SuppressFBWarnings({"SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE", "SQL_PREPARED_STATEMENT_GENERATED_FROM_NONCONSTANT_STRING"})
public class SnowflakeSchema extends Schema<SnowflakeDatabase> {

    private static final Log LOG = LogFactory.getLog(SnowflakeSchema.class);

    private final String catalogName;

    public SnowflakeSchema(JdbcTemplate jdbcTemplate, SnowflakeDatabase database, String name) {
        super(jdbcTemplate, database, name);
        LOG.debug("Creating new SnowflakeSchema");
        try {
            catalogName = jdbcTemplate.queryForString("SELECT CURRENT_DATABASE()");
            LOG.debug("Current Snowflake database is " + catalogName);
        } catch (SQLException e) {
            LOG.error("Unable to get current Snowflake database");
            throw new FlywaySqlException("Unable to get current database", e);
        }
    }

    //
    // NOTE: the following are overridden to implement
    //

    @Override
    protected boolean doExists() throws SQLException {
        List<Map<String, String>> objects = getObjects(SCHEMAS, name, "name");
        return !objects.isEmpty();
    }

    @Override
    protected boolean doEmpty() throws SQLException {
        if (doExists()) {
            List<Map<String, String>> objects = getObjects(OBJECTS, "%", "name");
            return objects.isEmpty();
        } else {
            return true;
        }
    }

    @Override
    protected void doCreate() throws SQLException {
        jdbcTemplate.execute("CREATE SCHEMA " + database.quote(catalogName, name));
    }

    @Override
    protected void doDrop() throws SQLException {
        jdbcTemplate.execute("DROP SCHEMA " + database.quote(catalogName, name) + " CASCADE");
    }

    @Override
    protected void doClean() throws SQLException {
        for (String statement : generateDropStatements(VIEWS)) {
            jdbcTemplate.execute(statement);
        }
        for (Table table : allTables()) {
            table.drop();
        }
        for (String statement : generateDropStatements(STAGES)) {
            jdbcTemplate.execute(statement);
        }
        for (String statement : generateDropStatements(FILE_FORMATS)) {
            jdbcTemplate.execute(statement);
        }
        for (String statement : generateDropStatements(SEQUENCES)) {
            jdbcTemplate.execute(statement);
        }
        for (String statement : generateDropStatements(PIPES)) {
            jdbcTemplate.execute(statement);
        }
        for (String statement : generateDropFunctionStatements()) {
            jdbcTemplate.execute(statement);
        }
    }

    @Override
    protected Table[] doAllTables() throws SQLException {
        List<Map<String, String>> objects = getObjects(TABLES, "%", "name");
        List<Table> tables = new ArrayList<>(objects.size());
        for (Map<String, String> object : objects) {
            String tableName = object.get("name");
            tables.add(getTable(tableName));
        }
        return tables.toArray(new Table[0]);
    }

    @Override
    public Table getTable(String tableName) {
        return new SnowflakeTable(jdbcTemplate, database, this, tableName);
    }

    //
    // NOTE: the following overrides are to change the implementation
    //

    //
    // Helper functions...
    //

    /* package */ List<Map<String, String>> getObjects(SnowflakeObjectType type, String filter, String... columns) throws SQLException {
        String inClause;
        if (type == SCHEMAS) {
            inClause = " IN DATABASE " + database.quote(catalogName);
        } else {
            inClause = " IN SCHEMA " + database.quote(catalogName, name);
        }

        String sql = "SHOW " + type.getShowType() + " LIKE '" + filter + "'" + inClause;
        LOG.debug("Executing [" + sql + "]");
        RowMapper<Map<String, String>> mapper = rs -> {
            Map<String, String> result = new HashMap<String, String>();
            for (String column : columns) {
                result.put(column, rs.getString(column));
            }
            return result;
        };
        return jdbcTemplate.query(sql, mapper);
    }

    private List<String> generateDropStatements(SnowflakeObjectType type) throws SQLException {
        List<Map<String, String>> objects = getObjects(type, "%", "name");
        List<String> result = new ArrayList<String>();
        for (Map<String, String> object : objects) {
            String value = object.get("name");
            result.add("DROP " + type.getCreateDropType() + " " + database.quote(name, value));
        }
        return result;
    }

    private List<String> generateDropFunctionStatements() throws SQLException {
        List<Map<String, String>> objects = getObjects(FUNCTIONS, "%", "arguments");
        List<String> result = new ArrayList<String>();
        for (Map<String, String> object : objects) {
            String value = object.get("arguments");
            // remove the RETURN clause from the fuction signature
            value = value.replaceAll("\\sRETURN\\s.*", "");
            result.add("DROP " + FUNCTIONS.getCreateDropType() + " " + database.quote(name, value));
        }
        return result;
    }

}

Also note the need to consider CURRENT_ROLE

@rjungwirth

This comment has been minimized.

Copy link

@rjungwirth rjungwirth commented Nov 7, 2019

@blian7432

This comment has been minimized.

Copy link

@blian7432 blian7432 commented Nov 7, 2019

Agree with option 2

@juliahayward

This comment has been minimized.

Copy link
Member

@juliahayward juliahayward commented Nov 8, 2019

So with CURRENT_ROLE(), is that not a local property of the connection? I'm just working through some test cases and it appears to me that a migration which calls USE ROLE Foo doesn't affect anything outside its own session.

@rjungwirth

This comment has been minimized.

Copy link

@rjungwirth rjungwirth commented Nov 8, 2019

juliahayward added a commit that referenced this issue Nov 15, 2019
@juliahayward

This comment has been minimized.

Copy link
Member

@juliahayward juliahayward commented Nov 25, 2019

Closing this issue as we're releasing imminently. Please start new issues for any problems that arise!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.