Permalink
Switch branches/tags
Commits on Aug 7, 2018
  1. Update dep github.com/skeema/tengo - lowercase information_schema col…

    evanelias committed Aug 7, 2018
    … aliases
    
    This commit brings in the latest Go La Tengo commit, to attempt to fix one
    incompatibility with MySQL 8.0. Skeema still does not yet support MySQL 8.0,
    but this is a step towards supporting it in the near future.
Commits on Aug 1, 2018
  1. Track database flavor (vendor and version) in .skeema files

    evanelias committed Aug 1, 2018
    This commit adds a new "flavor" option, which in Skeema v1.0.x is just
    informational (no functional impact). `skeema init`, `skeema pull`, and
    `skeema add-environment` now automatically populate this option in host-level
    .skeema files, to persist the database's vendor (mysql/percona/mariadb) and
    major.minor versions. This information may be used in new features beginning in
    Skeema v1.1.
Commits on Jul 31, 2018
  1. Update dep github.com/skeema/tengo - more test suite refactors

    evanelias committed Jul 31, 2018
    This brings in cleaner abstractions around Docker sandboxes. No real effect
    outside of the test suite yet, but this support will be leveraged in new
    features in the future (e.g. ability to offload the temporary schema operations
    to a local Docker container matching prod's flavor.)
Commits on Jul 26, 2018
  1. Update dep github.com/skeema/tengo

    evanelias committed Jul 26, 2018
    Functionally, this only immediately affects the test suite:
    
    * Test containers are now given specific names (for example
      "skeema-test-mysql-5.6"), and are stopped instead of destroyed at end of
      tests. This permits re-use across test runs, speeding up tests.
    
    * The TENGO_TEST_IMAGES env variable is no longer supported; use
      SKEEMA_TEST_IMAGES instead.
Commits on Jul 13, 2018
  1. Update dep github.com/skeema/tengo - fixes #31

    evanelias committed Jul 13, 2018
    The latest version of Go La Tengo fixes a bug in the sort order of foreign key
    names that have underscore prefixes, which are common when using pt-osc.
    
    The relevant test case is in skeema/tengo@7776645
    via the change to testdata/integration.sql.
Commits on Jul 11, 2018
  1. travis CI: auto-goreleaser upon tagging

    evanelias committed Jul 11, 2018
  2. Default to using a consistent, strict sql_mode. Fixes #31

    evanelias committed Jul 11, 2018
    Unless overridden by --connect-options, Skeema will now use the following
    session-level sql_mode for its connections:
    
    ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
    
    This provides a consistent baseline for behavior, regardless of what global
    default the server is configured with. This is especially helpful for older
    versions of MySQL, which don't default to strict mode. Additionally, this avoids
    problems with servers that use ANSI_QUOTES, which breaks Skeema's behavior.
    
    This commit also prevents using --connect-options to explicitly enable
    ANSI_QUOTES, or the combination ANSI mode which includes it.
Commits on Jul 9, 2018
Commits on Jul 6, 2018
  1. docs: prep for 1.0.0 release; configure goreleaser

    evanelias committed Jul 6, 2018
Commits on Jul 4, 2018
  1. Add support for MySQL 5.5

    evanelias committed Jul 4, 2018
Commits on Jul 3, 2018
  1. Add support for MariaDB 10.2 and 10.3

    evanelias committed Jul 3, 2018
Commits on Jul 2, 2018
  1. Test suite: skip TestNonInnoClauses for MariaDB backend

    evanelias committed Jul 2, 2018
    MariaDB does not support several of the clauses in this test function.
    
    Tests now pass for mariadb:10.1. Compatibility with MariaDB 10.2 and 10.3 will
    come in a future commit, since 10.2 changes how DEFAULT clauses are displayed by
    SHOW CREATE TABLE in a couple cases.
Commits on Jun 30, 2018
  1. Fix mysql:5.7 test failure of SkeemaIntegrationSuite.TestForeignKeys …

    evanelias committed Jun 30, 2018
    …due to 5.7 strict sql_mode
  2. push: new option --foreign-key-checks

    evanelias committed Jun 30, 2018
    By default, `skeema push` executes DDL in a session with foreign key checks
    disabled. This way, when adding a new foreign key to an existing table, no
    immediate integrity check is performed on existing data. This results in faster
    ALTER TABLE execution, and eliminates one possible failure vector for the DDL.
    
    This behavior may now be overridden by enabling the new --foreign-key-checks
    option. When enabled, `skeema push` enables foreign key checks for any ALTER
    TABLE that adds one or more foreign keys to an existing table. This means the
    server will validate existing data's referential integrity for new foreign
    keys, and the ALTER TABLE will fail with a fatal error if the constraint is not
    met for all rows.
    
    This option does not affect Skeema's behavior for other DDL, including CREATE
    TABLE or DROP TABLE. These statements are always executed in a session with
    foreign key checks disabled, to avoid any potential issues with thorny order-of-
    operations or circular references.
    
    This option has no effect in cases where an external OSC tool is being used via
    --alter-wrapper or --ddl-wrapper.
Commits on Jun 29, 2018
  1. diff/push: unsafe statements now checked in pre-flight

    evanelias committed Jun 29, 2018
    Previously, if unsafe statements were not allowed by the config, they would
    be skipped but any safe statements for the same schema would be allowed. This
    can be problematic with foreign keys, though: we might skip an unsafe ALTER
    that dropped a column but also added an index that is subsequently needed by
    a foreign key created in a separate ALTER.
    
    This situation should be exceedingly rare, but in any case, the correct
    approach seems to be skipping all DDL in a schema if at least one forbidden
    unsafe change is present. So with this commit, we now check for unsafe
    statements prior to executing anything in push. Diff also behaves this way now
    too; even though diff does not execute DDL regardless, it needs to be a "dry
    run" of push that behaves in an equivalent manner.
Commits on Jun 28, 2018
  1. diff/push: apply ADD FOREIGN KEY alters last, after other alters

    evanelias committed Jun 28, 2018
    This commit updates dep github.com/skeema/tengo in order to bring in a fix
    for a foreign key operation edge case. ADD FOREIGN KEY operations now occur
    in separate ALTERs (one per table) which are run after all other ALTERs in
    the same schema. This way, if the new FKs rely on other changes in the same
    diff -- such as referencing a new table, columns, or index -- they will still
    succeed.
    
    This commit adds a test for one specific case that previously would have
    failed, even with foreign_key_checks=0: adding an FK that needs a new index
    on the "parent" (referenced) table, where the parent table name is
    alphabetically after the child table.
    
    This does not yet address the situation of ordering constraints of foreign keys
    that span multiple schemas, which is rare in practice and very difficult to
    solve.
Commits on Jun 27, 2018
  1. Foreign key naming cosmetic by default; override with --exact-match

    evanelias committed Jun 27, 2018
    The --exact-match option now also affects foreign key naming. By default,
    Skeema views foreign key naming to be cosmetic, given two otherwise equivalent
    FK definitions. This provides maximum compatibility with tools like pt-osc
    that need to mangle FK names in order to operate. However, this behavior can
    be overridden with --exact-match now.
    
    This commit also fixes diff/push --verify to correctly handle FK name changes.
    
    Added docs and integration tests for the above.
  2. Docs: update to add contributor and mention FK support

    evanelias committed Jun 27, 2018
  3. support for foreign keys now added - latest tengo installed

    Chris Palmer authored and evanelias committed Jun 27, 2018
  4. pull: de-dupe operations by table, so file only updated once

    evanelias committed Jun 27, 2018
    This has no effect yet, but will matter once the dep github.com/skeema/tengo
    is updated here to bring in foreign key support. With foreign keys, there may
    be two TableDiff values affecting the same table.
Commits on Jun 21, 2018
  1. Support index reordering; add new --exact-match option (#30)

    evanelias committed Jun 21, 2018
    Previously, when a table had multiple secondary indexes, Skeema did not handle attempts to change the order of these indexes in the table definition, or attempts to insert new indexes "before" other existing ones. For InnoDB tables, this is usually just a cosmetic change -- secondary index order only affects tables that have no primary key but multiple unique indexes over non-nullable columns, which is a rare edge case. But despite being cosmetic, inability to reorder indexes was previously breaking Skeema's --verify option, due to the output of SHOW CREATE TABLE not exactly matching Skeema's expectation.
    
    This PR gives Skeema the ability to understand index ordering changes without --verify breaking. By default, Skeema will still nonetheless ignore such changes (in terms of its generated ALTER TABLE expressions), as they are cosmetic and may result in slower ALTER TABLE execution. But if a user truly wants to reorder indexes or position a new index anyway, this PR adds a new option --exact-match which will cause Skeema to include the correct operations in its generated ALTER TABLEs.
    
    Note that in many cases, MySQL also ignores these cosmetic attempts to reorder indexes, unless ALGORITHM=COPY is used. This can be accomplished in Skeema by using --alter-algorithm=COPY along with --exact-match. This may result in much slower ALTER TABLE execution for large tables, which is why it is not enabled by default.
    
    Fixes #29.
Commits on Jun 18, 2018
  1. Remove extraneous debug output in test

    evanelias committed Jun 18, 2018
  2. diff/push: Preserve order of new secondary indexes

    evanelias committed Jun 18, 2018
    When adding new secondary indexes to an existing table, the generated ALTER
    TABLE statement will now properly preserve the requested order in its ADD INDEX
    clauses. Fixes #28.
    
    The main fix and unit test occur at the library (Go La Tengo) level, and are in
    skeema/tengo@ba3e8ce; meanwhile this skeema commit
    simply updates that dep to bring in the fix, and also adds an integration test.
Commits on Jun 16, 2018
  1. Ignore/strip CREATE TABLE clauses that have no effect in InnoDB

    evanelias committed Jun 16, 2018
    There are a few rare CREATE TABLE clauses which have no effect in InnoDB and are
    not reflected anywhere in information_schema, yet MySQL nonetheless "remembers"
    in its *.frm table definition files. These clauses do still appear in SHOW
    CREATE TABLE though, which tricked Skeema into thinking the table is using
    features that it doesn't support, since it can't come up with a matching CREATE
    TABLE.
    
    There are at least 4 types of such clauses, although only the first two are
    realistically ever seen in the wild:
    
    1. USING BTREE and USING HASH index clauses, which may be added by tools like
       phpMyAdmin
    2. KEY_BLOCK_SIZE=n clauses on indexes, which confusingly comes up if you
       compress an InnoDB table and then remove compression (see mysql bug 70534)
    3. STORAGE MEMORY and STORAGE DISK clauses on columns
    4. COLUMN_FORMAT DYNAMIC and COLUMN_FORMAT FIXED clauses on columns
    
    Skeema now strips these clauses from InnoDB tables at all times:
    * `skeema init` omits these clauses from the files it writes
    * `skeema pull` and `skeema lint` will normalize files to remove these clauses
    * `skeema diff` and `skeema push` completely ignore these clauses, meaning that
      they are not treated as differences, and do not break validation logic.
    
    Fixes #13.
Commits on Jun 14, 2018
  1. Increase concurrency further for some temp schema operations

    evanelias committed Jun 14, 2018
  2. Switch from godep to dep; fix support for Golang 1.9 (#27)

    evanelias committed Jun 14, 2018
    Support for Golang 1.9 in Skeema has been broken since master branch of
    transitive dep github.com/docker/docker now requires Golang 1.10. By switching
    to Dep, the dependency chain should now work properly for Golang 1.9.
Commits on Jun 13, 2018
  1. Performance improvements (#26)

    evanelias committed Jun 13, 2018
    This PR improves query patterns by avoiding unnecessary round-trips  and adding concurrency. This makes Skeema faster. The changes help in all cases (including just running the integration tests), but will be most dramatic for users with a larger number of tables and/or high latency between Skeema and the database server.
    
    Specific major improvements:
    * Several query patterns which run 1 query per table are now able to query concurrently:
        * SHOW CREATE TABLE queries run for schema introspection purposes
        * Temporary schema queries, to create and drop the tables represented by *.sql files
        * Various safety check queries
    * `skeema push` only performs checks around table size and presence of rows when needed based on options (--safe-below-size and/or --alter-wrapper-min-size in use, and/or the {SIZE} template variable is used in an --alter-wrapper or --ddl-wrapper command line)
    * the --verify option for `skeema diff` and `skeema push` now adds substantially less overhead, by only interacting with altered tables
Commits on Jun 7, 2018
  1. schema=* now respects ignore-schema properly

    evanelias committed Jun 7, 2018
  2. Minor improvements to `skeema lint`, testing

    evanelias committed Jun 7, 2018
    `skeema lint` now ignores SQL errors in files with filenames that match value
    of --ignore-table.
    
    Added test to confirm `skeema lint` fatals on *.sql files with filename that
    doesn't actually match the corresponding table name.
    
    Command test output now includes log for exit code and message.
  3. Fix 3 minor edge-case bugs around skeema pull --normalize:

    evanelias committed Jun 7, 2018
    1. If a table only had its next-auto-inc change, still normalize its format if
       --normalize is enabled.
    
    2. When normalizing a table's format, strip next-auto-inc, unless the file
       already contained next-auto-inc or --include-auto-inc is enabled.
    
    3. If a table name matches --ignore-table, do not normalize its format.
    
    Also add test coverage around these cases, and a few other related situations
    that were not explicitly covered.
Commits on Jun 6, 2018
  1. Update dep github.com/skeema/tengo, and simplify pull and push

    evanelias committed Jun 6, 2018