-
Notifications
You must be signed in to change notification settings - Fork 271
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Use Flyway to deploy SQL schema to non-prod (#255)
* Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Updated ClaimsList.java so that Hibernate-generated schema would use the right types. Using 'varchar(255)' instead of 'text' for string columns for now. We will need to investigate how to force Hibernate to use the desired types in all cases. * Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Updated ClaimsList.java so that Hibernate-generated schema would use the right types. Using 'varchar(255)' instead of 'text' for string columns for now. We will need to investigate how to force Hibernate to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha using Flyway. Updated ClaimsList.java so that Hibernate-generated schema would use the right types. Using 'varchar(255)' instead of 'text' for string columns for now. We will need to investigate how to force Hibernate to use the desired types in all cases. * Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Corrected the type of ClaimsEntry's revision_id column. It should be plain int8, not bigserial. Make GenerateSqlSchemaCommand use a custom dialect that converts all varchar type to 'text' and timestamp to 'timestamptz'. * Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Use a custome dialect in GenerateSqlSchemaCommand to convert varchar type to 'text' and timestamp to 'timestamptz'. Corrected ClaimsEntry's revision_id column type to int8. This column tracks parent table's primary key and should not be bigserial. * Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Use a custome dialect in GenerateSqlSchemaCommand to convert varchar type to 'text' and timestamp to 'timestamptz'. Corrected ClaimsEntry's revision_id column type to int8. This column tracks parent table's primary key and should not be bigserial. * Use Flyway to deploy SQL schema to non-prod Added Gradle tasks to deploy and drop schema in alpha using Flyway. Use a custome dialect in GenerateSqlSchemaCommand to convert varchar type to 'text' and timestamp to 'timestamptz'. Corrected ClaimsEntry's revision_id column type to int8. This column tracks parent table's primary key and should not be bigserial.
- Loading branch information
Showing
13 changed files
with
319 additions
and
61 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| ## Summary | ||
|
|
||
| This project contains Nomulus's Cloud SQL schema and schema deployment utilities. | ||
|
|
||
| ### Schema Creation DDL | ||
|
|
||
| Currently we use Flywaydb for schema deployment. Versioned migration scripts | ||
| are organized in the src/main/resources/sql/flyway folder. Scripts must follow | ||
| the V{id}__{description text}.sql naming pattern (Note the double underscore). | ||
|
|
||
| The 'nomulus.golden.sql' file in src/main/resources/sql/schema folder is | ||
| mainly informational. It is generated by Hibernate and should not be | ||
| reformatted. We will use it in validation tests later. | ||
|
|
||
| ### Non-production Schema Push | ||
|
|
||
| To manage schema in a non-production environment, use the 'flywayMigration' task. | ||
| You will need Cloud SDK and login once. | ||
|
|
||
| ```shell | ||
| # One time login | ||
| gcloud auth login | ||
|
|
||
| # Deploy the current schema to alpha | ||
| gradlew :db:flywayMigrate -PdbServer=alpha | ||
|
|
||
| # Delete the entire schema in alpha | ||
| gradlew :db:flywayClean -PdbServer=alpha | ||
| ``` | ||
|
|
||
| The flywayMigrate task is idempotent. Repeated runs will not introduce problems. | ||
|
|
||
| The Flyway tasks may also be used to deploy to local instances, e.g, your own | ||
| test instance. E.g., | ||
|
|
||
| ```shell | ||
| # Deploy to a local instance at standard port as the super user. | ||
| gradlew :db:flywayMigrate -PdbServer=192.168.9.2 -PdbPassword=domain-registry | ||
|
|
||
| # Full specification of all parameters | ||
| gradlew :db:flywayMigrate -PdbServer=192.168.9.2:5432 -PdbUser=postgres \ | ||
| -PdbPassword=domain-registry | ||
| ``` | ||
|
|
||
| ### Production Schema Deployment | ||
|
|
||
| Schema deployment to production and sandbox is under development. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| // Copyright 2019 The Nomulus Authors. All Rights Reserved. | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| import com.google.common.collect.ImmutableList | ||
|
|
||
| plugins { | ||
| id "org.flywaydb.flyway" version "6.0.1" | ||
| } | ||
|
|
||
| ext { | ||
| def dbServerProperty = 'dbServer' | ||
| def dbNameProperty = 'dbName' | ||
|
|
||
| def dbServer = findProperty(dbServerProperty) | ||
| def dbName = findProperty(dbNameProperty) | ||
|
|
||
| getAccessInfoByHostPort = { hostAndPort -> | ||
| return [ | ||
| url: "jdbc:postgresql://${hostAndPort}/${dbName}", | ||
| user: findProperty('dbUser'), | ||
| password: findProperty('dbPassword')] | ||
| } | ||
|
|
||
| getSocketFactoryAccessInfo = { | ||
| def cred = getCloudSqlCredential('alpha', 'superuser').split(' ') | ||
| def sqlInstance = cred[0] | ||
| return [ | ||
| url: """\ | ||
| jdbc:postgresql://google/${dbName}?cloudSqlInstance= | ||
| ${sqlInstance}&socketFactory= | ||
| com.google.cloud.sql.postgres.SocketFactory""" | ||
| .stripIndent() | ||
| .replaceAll(System.lineSeparator(), '') , | ||
| user: cred[1], | ||
| password: cred[2]] | ||
| } | ||
|
|
||
| getJdbcAccessInfo = { | ||
| switch (dbServer.toString().toLowerCase()) { | ||
| case 'alpha': | ||
| return getSocketFactoryAccessInfo() | ||
| default: | ||
| return getAccessInfoByHostPort(dbServer) | ||
| } | ||
| } | ||
|
|
||
| // Retrieves Cloud SQL credential for a given role. Result is in the form of | ||
| // 'instancename username password'. | ||
| // | ||
| // The env parameter may be one of the following: alpha, crash, sandbox, or | ||
| // production. The role parameter may be superuser. (More roles will be added | ||
| // later). | ||
| getCloudSqlCredential = { env, role -> | ||
| env = env == 'production' ? '' : "-${env}" | ||
| def command = | ||
| """gsutil cp \ | ||
| gs://domain-registry${env}-cloudsql-credentials/${role}.enc - | \ | ||
| gcloud kms decrypt --location global --keyring nomulus \ | ||
| --key sql-credentials-on-gcs-key --plaintext-file=- \ | ||
| --ciphertext-file=- \ | ||
| --project=domain-registry${env}-keys""" | ||
|
|
||
| return execInBash(command, '/tmp') | ||
| } | ||
| } | ||
|
|
||
| flyway { | ||
| def accessInfo = project.ext.getJdbcAccessInfo() | ||
|
|
||
| url = accessInfo.url | ||
| user = accessInfo.user | ||
| password = accessInfo.password | ||
| schemas = [ 'public' ] | ||
|
|
||
| locations = [ "classpath:sql/flyway" ] | ||
| } | ||
|
|
||
| dependencies { | ||
| runtimeOnly 'org.flywaydb:flyway-core:5.2.4' | ||
|
|
||
| runtimeOnly 'com.google.cloud.sql:postgres-socket-factory:1.0.12' | ||
| runtimeOnly 'org.postgresql:postgresql:42.2.5' | ||
| } | ||
|
|
||
| // Ensure that resources are rebuilt before running Flyway tasks | ||
| tasks | ||
| .findAll { task -> task.group.equals('Flyway')} | ||
| .collect { task -> task.dependsOn('buildNeeded') } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # This is a Gradle generated file for dependency locking. | ||
| # Manual edits can break the build and are not advised. | ||
| # This file is expected to be part of source control. | ||
| gradle.plugin.com.boxfuse.client:gradle-plugin-publishing:6.0.1 | ||
| org.flywaydb.flyway:org.flywaydb.flyway.gradle.plugin:6.0.1 | ||
| org.flywaydb:flyway-core:6.0.1 |
31 changes: 31 additions & 0 deletions
31
db/src/main/resources/sql/flyway/V1__new_claims_list_and_entry.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| -- Copyright 2019 The Nomulus Authors. All Rights Reserved. | ||
| -- | ||
| -- Licensed under the Apache License, Version 2.0 (the "License"); | ||
| -- you may not use this file except in compliance with the License. | ||
| -- You may obtain a copy of the License at | ||
| -- | ||
| -- http://www.apache.org/licenses/LICENSE-2.0 | ||
| -- | ||
| -- Unless required by applicable law or agreed to in writing, software | ||
| -- distributed under the License is distributed on an "AS IS" BASIS, | ||
| -- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| -- See the License for the specific language governing permissions and | ||
| -- limitations under the License. | ||
|
|
||
| create table "ClaimsEntry" ( | ||
| revision_id int8 not null, | ||
| claim_key text not null, | ||
| domain_label text not null, | ||
| primary key (revision_id, domain_label) | ||
| ); | ||
|
|
||
| create table "ClaimsList" ( | ||
| revision_id bigserial not null, | ||
| creation_timestamp timestamptz not null, | ||
| primary key (revision_id) | ||
| ); | ||
|
|
||
| alter table "ClaimsEntry" | ||
| add constraint FKlugn0q07ayrtar87dqi3vs3c8 | ||
| foreign key (revision_id) | ||
| references "ClaimsList"; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.