Use Flyway to deploy SQL schema to non-prod

[weiminyu]

Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.Added Gradle tasks to deploy and drop schema in alpha
using Flyway.

Updated ClaimsList.java so that Hibernate-generated
schema would use the right types.

Using 'varchar(255)' instead of 'text' for string columns
for now. We will need to investigate how to force Hibernate
to use the desired types in all cases.

---

This change is 

[mindhog]

Reviewed 4 of 10 files at r1.
Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)

---

db/build.gradle, line 28 at r1 (raw file):

      .findOptionalProperty(dbServerProperty).orElse('localhost:5432')
  def dbName = rootProject.ext
      .findOptionalProperty(dbNameProperty).orElse('postgres')

Instead of making it optional, can we add this to gradle.properties instead? It would be helpful to have all of our properties in one place instead of strewn around the build files.

---

db/build.gradle, line 42 at r1 (raw file):

    def dbUser = cred[1]
    def dbPassword = cred[2]
    return ImmutableList.of(

Instead of doing these as immutable lists, you could do:
return [url: ..., user: dbUser, password: dbPassword]

and then access them as:

accessInfo.url
accessInfo.user
accessInfo.password

below

---

db/build.gradle, line 43 at r1 (raw file):

    def dbPassword = cred[2]
    return ImmutableList.of(
        "jdbc:postgresql://google/${dbName}?cloudSqlInstance=${sqlInstance}&socketFactory=com.google.cloud.sql.postgres.SocketFactory",

Please reformat to at most 100 chars.

---

db/build.gradle, line 61 at r1 (raw file):

  // - env: alpha, crash, sandbox, or production
  // - role:
  //   * superuser

Formatting of the comments seems weird. I assume "superuser" is the only legitimate value? If so, why not make it a default?

[CydeWeys]

We need to get the TEXT fields working, otherwise we're going to start off with storing bad data in an incorrect format, and need yet another delta later on just to fix the field type.

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, and @weiminyu)

[CydeWeys]

How about @column(columnDefinition = "text") ?

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @weiminyu)

[CydeWeys]

We might also make a simple type translator to do this, that would allow a little bit better abstraction in case someone in the future wants to support other DBs.

E.g. something like @text String columnName;

Where @text is an annotation we've defined that turns it into a TEXT field. Then add a presubmit on the .sql schema files ensuring that we never add a VARCHAR and we're good.

Reviewable status: 4 of 10 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @weiminyu)

[weiminyu]

I'm using a hibernate custom dialect to do wholesale mappings:
varchar -> text
timestamp -> timestamptz

This is only needed in GenerateSqlSchemaCommand, and is safe to tinker with anyway we want.

Reviewable status: 3 of 12 files reviewed, 4 unresolved discussions (waiting on @gbrodman, @hstonec, @jianglai, and @mindhog)

---

db/build.gradle, line 28 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Instead of making it optional, can we add this to gradle.properties instead? It would be helpful to have all of our properties in one place instead of strewn around the build files.

Done.

---

db/build.gradle, line 42 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Instead of doing these as immutable lists, you could do:
return [url: ..., user: dbUser, password: dbPassword]

and then access them as:

accessInfo.url
accessInfo.user
accessInfo.password

below

Done.

---

db/build.gradle, line 43 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Please reformat to at most 100 chars.

Done.

---

db/build.gradle, line 61 at r1 (raw file):

Previously, mindhog (Michael Muller) wrote…

Formatting of the comments seems weird. I assume "superuser" is the only legitimate value? If so, why not make it a default?

Will add more roles, and default will be 'reader'.

[hstonec]

Reviewable status: 3 of 12 files reviewed, 5 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)

---

db/build.gradle, line 72 at r3 (raw file):

           --key sql-credentials-on-gcs-key --plaintext-file=- \
           --ciphertext-file=- \
           --project=domain-registry${env}-keys"""

Why do we store the keyring in another project?

[mindhog]

Reviewed 2 of 10 files at r1, 6 of 8 files at r2, 1 of 1 files at r3.
Reviewable status: all files reviewed, 9 unresolved discussions (waiting on @CydeWeys, @gbrodman, @jianglai, @mindhog, and @weiminyu)

---

core/src/main/java/google/registry/tools/GenerateSqlSchemaCommand.java, line 192 at r3 (raw file):

      registerColumnType(Types.VARCHAR, "text");
      registerColumnType(Types.TIMESTAMP, "timestamptz");
    }

Please regenerate the generated schema:

java -jar ./core/build/libs/nomulus.jar -e alpha generate_sql_schema -o db/src/main/resources/sql/schema/db-schema.sql.generated

---

db/build.gradle, line 32 at r3 (raw file):

        'url': "jdbc:postgresql://${hostAndPort}/${dbName}",
        'user': findProperty('dbUser'),
        'password': findProperty('dbPassword')]

Note that you shouldn't need to quote the keys.

---

db/src/main/resources/sql/flyway/V1__new_claims_list_and_entry.sql, line 15 at r3 (raw file):

-- limitations under the License.

    create table "ClaimsEntry" (

Do the double-quotes have the same semantics as the back-quotes?

---

db/src/main/resources/sql/schema/nomulus.golden.sql, line 1 at r3 (raw file):

What's the intended purpose of this file?

[hstonec]

Reviewable status: all files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @gbrodman, @jianglai, @mindhog, and @weiminyu)

---

db/build.gradle, line 72 at r3 (raw file):

Previously, hstonec (Shicong Huang) wrote…

Why do we store the keyring in another project?

ok, I noticed that we already have a section of kms in the yaml file with exact same set up. But I was wondering if it is ok to expose these information to the external? Right now, the keyring name, project id are only stored in the internal file.

[weiminyu]

Reviewable status: 10 of 13 files reviewed, 8 unresolved discussions (waiting on @CydeWeys, @gbrodman, @hstonec, @jianglai, @mindhog, and @weiminyu)

---

core/src/main/java/google/registry/tools/GenerateSqlSchemaCommand.java, line 192 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

Please regenerate the generated schema:

java -jar ./core/build/libs/nomulus.jar -e alpha generate_sql_schema -o db/src/main/resources/sql/schema/db-schema.sql.generated

Done.

---

db/build.gradle, line 72 at r3 (raw file):

Previously, hstonec (Shicong Huang) wrote…

ok, I noticed that we already have a section of kms in the yaml file with exact same set up. But I was wondering if it is ok to expose these information to the external? Right now, the keyring name, project id are only stored in the internal file.

I think we should continue keeping such information in internal config.

---

db/src/main/resources/sql/flyway/V1__new_claims_list_and_entry.sql, line 15 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

Do the double-quotes have the same semantics as the back-quotes?

Back-quote is a mysql thing and is illegal for name quoting in postgresql

---

db/src/main/resources/sql/schema/nomulus.golden.sql, line 1 at r3 (raw file):

Previously, mindhog (Michael Muller) wrote…

What's the intended purpose of this file?

We will add test to verify that the final schema produced by the incremental scripts is equivalent to this schema.

[mindhog]

Reviewed 3 of 3 files at r4.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @CydeWeys, @gbrodman, @jianglai, and @weiminyu)