Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Flyway to deploy SQL schema to non-prod #255

Merged
merged 6 commits into from Sep 6, 2019

Conversation

@weiminyu
Copy link
Collaborator

commented Sep 4, 2019

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.


This change is Reviewable

@mindhog
Copy link
Member

left a comment

Reviewed 4 of 10 files at r1.
Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)


db/build.gradle, line 28 at r1 (raw file):

      .findOptionalProperty(dbServerProperty).orElse('localhost:5432')
  def dbName = rootProject.ext
      .findOptionalProperty(dbNameProperty).orElse('postgres')

Instead of making it optional, can we add this to gradle.properties instead? It would be helpful to have all of our properties in one place instead of strewn around the build files.


db/build.gradle, line 42 at r1 (raw file):

    def dbUser = cred[1]
    def dbPassword = cred[2]
    return ImmutableList.of(

Instead of doing these as immutable lists, you could do:
return [url: ..., user: dbUser, password: dbPassword]

and then access them as:

accessInfo.url
accessInfo.user
accessInfo.password

below


db/build.gradle, line 43 at r1 (raw file):

    def dbPassword = cred[2]
    return ImmutableList.of(
        "jdbc:postgresql://google/${dbName}?cloudSqlInstance=${sqlInstance}&socketFactory=com.google.cloud.sql.postgres.SocketFactory",

Please reformat to at most 100 chars.


db/build.gradle, line 61 at r1 (raw file):

  // - env: alpha, crash, sandbox, or production
  // - role:
  //   * superuser

Formatting of the comments seems weird. I assume "superuser" is the only legitimate value? If so, why not make it a default?

@CydeWeys
Copy link
Member

left a comment

We need to get the TEXT fields working, otherwise we're going to start off with storing bad data in an incorrect format, and need yet another delta later on just to fix the field type.

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, and @weiminyu)

@CydeWeys
Copy link
Member

left a comment

How about @column(columnDefinition = "text") ?

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @weiminyu)

@CydeWeys
Copy link
Member

left a comment

We might also make a simple type translator to do this, that would allow a little bit better abstraction in case someone in the future wants to support other DBs.

E.g. something like @text String columnName;

Where @text is an annotation we've defined that turns it into a TEXT field. Then add a presubmit on the .sql schema files ensuring that we never add a VARCHAR and we're good.

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @weiminyu)

@weiminyu
Copy link
Collaborator Author

left a comment

I'm using a hibernate custom dialect to do wholesale mappings:
varchar -> text
timestamp -> timestamptz

This is only needed in GenerateSqlSchemaCommand, and is safe to tinker with anyway we want.

Reviewable status: 3 of 12 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @mindhog)


db/build.gradle, line 28 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Instead of making it optional, can we add this to gradle.properties instead? It would be helpful to have all of our properties in one place instead of strewn around the build files.

Done.


db/build.gradle, line 42 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Instead of doing these as immutable lists, you could do:
return [url: ..., user: dbUser, password: dbPassword]

and then access them as:

accessInfo.url
accessInfo.user
accessInfo.password

below

Done.


db/build.gradle, line 43 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Please reformat to at most 100 chars.

Done.


db/build.gradle, line 61 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Formatting of the comments seems weird. I assume "superuser" is the only legitimate value? If so, why not make it a default?

Will add more roles, and default will be 'reader'.

@weiminyu weiminyu requested review from CydeWeys and mindhog Sep 6, 2019

@weiminyu weiminyu force-pushed the weiminyu:sql-from-desktop branch from 48d672b to 2e6ae34 Sep 6, 2019

@hstonec
Copy link
Collaborator

left a comment

Reviewable status: 3 of 12 files reviewed, 5 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)


db/build.gradle, line 72 at r3 (raw file):

           --key sql-credentials-on-gcs-key --plaintext-file=- \
           --ciphertext-file=- \
           --project=domain-registry${env}-keys"""

Why do we store the keyring in another project?

@mindhog
Copy link
Member

left a comment

Reviewed 2 of 10 files at r1, 6 of 8 files at r2, 1 of 1 files at r3.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @CydeWeys, @gbrodman, @jianglai, @mindhog, and @weiminyu)


core/src/main/java/google/registry/tools/GenerateSqlSchemaCommand.java, line 192 at r3 (raw file):

      registerColumnType(Types.VARCHAR, "text");
      registerColumnType(Types.TIMESTAMP, "timestamptz");
    }

Please regenerate the generated schema:

java -jar ./core/build/libs/nomulus.jar -e alpha generate_sql_schema -o db/src/main/resources/sql/schema/db-schema.sql.generated


db/build.gradle, line 32 at r3 (raw file):

        'url': "jdbc:postgresql://${hostAndPort}/${dbName}",
        'user': findProperty('dbUser'),
        'password': findProperty('dbPassword')]

Note that you shouldn't need to quote the keys.


db/src/main/resources/sql/flyway/V1__new_claims_list_and_entry.sql, line 15 at r3 (raw file):

-- limitations under the License.

    create table "ClaimsEntry" (

Do the double-quotes have the same semantics as the back-quotes?


db/src/main/resources/sql/schema/nomulus.golden.sql, line 1 at r3 (raw file):

What's the intended purpose of this file?

@hstonec
Copy link
Collaborator

left a comment

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @gbrodman, @jianglai, @mindhog, and @weiminyu)


db/build.gradle, line 72 at r3 (raw file):

Previously, hstonec (Shicong Huang) wrote…

Why do we store the keyring in another project?

ok, I noticed that we already have a section of kms in the yaml file with exact same set up. But I was wondering if it is ok to expose these information to the external? Right now, the keyring name, project id are only stored in the internal file.

weiminyu added 5 commits Sep 4, 2019
Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.
Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Corrected the type of ClaimsEntry's revision_id column.
It should be plain int8, not bigserial.

Make GenerateSqlSchemaCommand use a custom dialect that
converts all varchar type to 'text' and timestamp to
'timestamptz'.
Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.
Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.

@weiminyu weiminyu force-pushed the weiminyu:sql-from-desktop branch from 0ad3e09 to 2e5d513 Sep 6, 2019

Use Flyway to deploy SQL schema to non-prod
Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Use a custome dialect in GenerateSqlSchemaCommand to
convert varchar type to 'text' and timestamp to 'timestamptz'.

Corrected ClaimsEntry's revision_id column type to int8.
This column tracks parent table's primary key and should
not be bigserial.

@weiminyu weiminyu requested review from hstonec and mindhog Sep 6, 2019

@weiminyu
Copy link
Collaborator Author

left a comment

Reviewable status: 10 of 13 files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)


core/src/main/java/google/registry/tools/GenerateSqlSchemaCommand.java, line 192 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

Please regenerate the generated schema:

java -jar ./core/build/libs/nomulus.jar -e alpha generate_sql_schema -o db/src/main/resources/sql/schema/db-schema.sql.generated

Done.


db/build.gradle, line 72 at r3 (raw file):

Previously, hstonec (Shicong Huang) wrote…

ok, I noticed that we already have a section of kms in the yaml file with exact same set up. But I was wondering if it is ok to expose these information to the external? Right now, the keyring name, project id are only stored in the internal file.

I think we should continue keeping such information in internal config.


db/src/main/resources/sql/flyway/V1__new_claims_list_and_entry.sql, line 15 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

Do the double-quotes have the same semantics as the back-quotes?

Back-quote is a mysql thing and is illegal for name quoting in postgresql


db/src/main/resources/sql/schema/nomulus.golden.sql, line 1 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

What's the intended purpose of this file?

We will add test to verify that the final schema produced by the incremental scripts is equivalent to this schema.

@mindhog
mindhog approved these changes Sep 6, 2019
Copy link
Member

left a comment

Reviewed 3 of 3 files at r4.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @CydeWeys, @gbrodman, @jianglai, and @weiminyu)

@hstonec
hstonec approved these changes Sep 6, 2019

@weiminyu weiminyu merged commit 471ed7c into google:master Sep 6, 2019

6 of 7 checks passed

code-review/reviewable 1 discussion left (CydeWeys, gbrodman, jianglai, weiminyu)
Details
LGTM analysis: Java No new or fixed alerts
Details
LGTM analysis: JavaScript No new or fixed alerts
Details
LGTM analysis: Python No new or fixed alerts
Details
cla/google All necessary CLAs are signed
kokoro-foss Kokoro build finished
Details
kokoro-internal Kokoro build finished
Details

@weiminyu weiminyu deleted the weiminyu:sql-from-desktop branch Sep 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.