Add Meta.migrateTo(Meta):Queries to generate a migration script between two Meta versions

[lukaseder]

org.jooq.Meta instances are becoming increasingly important representations of runtime schema meta information. Historically, they were implemented purely using JDBC's DatabaseMetaData, which queries the actual database dictionary views directly. But we now also have implementations taking:

• Arbitrary jOOQ meta objects, including Catalog, Schema, Table
• The jOOQ InformationSchema object, which can be read from XML
• DDL statements that are parsed and interpreted

The above implementations are all immutable and do not depend on a live database connection. With those immutable implementations, we can now compare two versions of the meta model, and try to derive a migration script between the two versions

Things that can be detected automatically:

• Catalogs (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
• Schemas
  • Schema additions
  • Schema removals
• Tables
  • Table additions
  • Table removals
  • Column data type changes
    • Nullability
    • Default values
    • Precision / scale
    • Length
    • Type
    • Identities (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
    • Collations (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
    • Character sets (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
  • Column additions
  • Column removals
  • Column reorderings (only if respectColumnOrder is set) (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
  • Primary and unique constraints
    • Additions
    • Removals
    • Renames
  • Foreign key constraints
    • Additions
    • Removals
    • Renames
  • Check constraints
    • Additions
    • Removals
    • Renames
  • Index additions
  • Index removals
  • Index renames
• Views
  • View additions
  • View removals
  • View replacements
• Types (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
• Sequences
  • Sequence additions
  • Sequence removals
  • Sequence flag changes (depends on Add support for Sequence flags in meta model #7752)
• Comments
  • Comment additions
  • Comment removals

Things that can be detected using some heuristics (based on likeliness):

• Renaming (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)
• Moving (moved to Continue work on Meta.migrateTo(Meta):Queries #13654)

MigrationConfiguration

Just like the DDLExportConfiguration, we should have a MigrationConfiguration that governs the style of produced queries. Some examples:

• alterTableAddMultiple: To add multiple columns and constraints in one single statement (default: false)
• alterTableDropMultiple: To drop multiple columns and constraints in one single statement (default: false)
• dropSchemaCascade: To use the CASCADE syntax when dropping schemas (default: false)
• dropTableCascade: To use the CASCADE syntax when dropping tables (default: false)
• createOrReplaceView: Whether modified views should be replaced, or dropped and created (default: false)

More have been moved to #13654

Out of scope for 3.13

Some more advanced things are out of scope for 3.13 and will be implemented later, tracked as #9656

[knutwannheden]

I think it could also be interesting to allow the user to "guide" the diff, in case it gets something wrong. E.g. when the diff results in a RENAME rather than in a DROP / CREATE pair or vice versa.

I suppose the user could already achieve this using Queries#concat() (with some calls to ddl() and meta()). But possibly we should also add a Meta#combine(Meta) (or concat()) as a counterpart to diff(), which would make this much easier.

[lukaseder]

I think it could also be interesting to allow the user to "guide" the diff, in case it gets something wrong. E.g. when the diff results in a RENAME rather than in a DROP / CREATE pair or vice versa.

Yes, clearly the "heuristics" I've mentioned should be configurable

I suppose the user could already achieve this using Queries#concat() (with some calls to ddl() and meta()). But possibly we should also add a Meta#combine(Meta) (or concat()) as a counterpart to diff(), which would make this much easier.

Not sure if I understand this?

[knutwannheden]

Not sure if I understand this?

To guide the rename heuristics the user could for instance take the Meta and "manually" rename a table by using the Meta as a baseline with the DDLInterpreter and interpret an ALTER TABLE ... RENAME to produce a new Meta. Then the diff would not have to apply the heuristic anymore.

I was first thinking that a new Meta#combine(Meta) which essentially does a "union" of two Meta would be useful here, but that of course doesn't make sense. Something like Meta#apply(Query...) (or Meta#transform(Query...)) would make more sense.

Possibly Meta#combine(Meta) could also be useful for other use cases, but I guess that remains to be seen.

[knutwannheden]

So a utility method like this is what I had in mind:

    public Meta apply(Query... queries) {
        return dsl().meta(ddl().concat(dsl().queries(queries)).queries());
    }

[lukaseder]

To guide the rename heuristics the user could for instance take the Meta and "manually" rename a table by using the Meta as a baseline with the DDLInterpreter and interpret an ALTER TABLE ... RENAME to produce a new Meta. Then the diff would not have to apply the heuristic anymore.

Ah yes, that's something I also had in mind. In fact, I even thought of the same name as you for the method 👌

public Meta apply(Query... queries) { ... }

I was first thinking that a new Meta#combine(Meta) which essentially does a "union" of two Meta would be useful here, but that of course doesn't make sense.

Well, we already do that when we apply several information_schema XML files, but I agree I'm not convinced of the usefulness of this.

We'll see if we need it for Meta in general.

[lukaseder]

In fact, I had already implemented Meta.apply() in the Diff tests 😉

[lukaseder]

Meta.apply(Queries) was implemented with #9436

[lukaseder]

I'm running a poll about method naming, which is probably too confusing:
https://twitter.com/lukaseder/status/1204689766324854785

A better name for the method will be discussed soon

[lukaseder]

We'll use the term A.migrateTo(B) to describe a migration from A to B. This applies to both Meta.migrateTo(Meta) and Version.migrateTo(Version)

[lukaseder]

After careful thought, I've decided that this is going to be a very central feature of the new migration APIs. Adoption of these APIs depends on the success of this feature, which is why we'll be open sourcing it.

The distinction between editions will be done on a more fine-grained level.

[lukaseder]

Not sure why this issue hadn't been closed at the time. There's still work, yes: #13654, but we've delivered quite a bit of functionality nonetheless.

Closing it now. Continuing work in #13654