Permalink
Commits on Nov 12, 2018
  1. Fix error from foreign key with same name as unique key

    evanelias committed Nov 12, 2018
    Within a single database schema, if a foreign key has the same name as a unique
    index, all Skeema commands would fail with this error message:
    
        Error querying foreign key constraints: sql: Scan error on column index 6:
        unsupported Scan, storing driver.Value type <nil> into type *string
    
    This occurred because an additional join clause is needed to de-dupe this
    situation, as both types of constraints are present in MySQL's
    information_schema.key_column_usage table.
    
    The bug and its fix are both actually in github.com/skeema/tengo, so this commit
    just updates the vendored dep. Test coverage is handled by
    skeema/tengo@c9951fe#diff-ad27291d895bd1679307e9c94c251037
    
    Fixes #46.
Commits on Nov 9, 2018
  1. Options to use local Docker for temporary schema (#45)

    evanelias committed Nov 9, 2018
    BACKGROUND:
    
    Skeema's behavior does not rely on parsing SQL DDL, as this can be too brittle across various MySQL versions and vendors, which have subtle differences in features and functionality. Instead, Skeema uses metadata reported directly from the database to introspect schemas, using information_schema as well as various SHOW commands.
    
    In order to accurately introspect the schemas represented in your filesystem's *.sql files, Skeema actually runs the files' CREATE TABLE statements in a temporary location, now called a "workspace." Previously (and still by default), Skeema creates, uses, and then drops a temporary schema on each database it interacts with.
    
    WHAT'S NEW:
    
    This PR adds the ability to instead use a local Docker container for workspace operations. Two new options control this behavior:
    
    * `workspace=docker` tells Skeema to dynamically manage local Docker container(s) for workspace operations, instead of using a temporary schema on each live DB.
    
    * `docker-cleanup` controls how to manage the container lifecycle as Skeema is exiting. The default, `docker-cleanup=none`, leaves containers running so that subsequent invocations of Skeema are faster. Setting `docker-cleanup=stop` stops containers but does not remove them, and `docker-cleanup=destroy` deletes them entirely.
    
    This functionality is especially useful when running Skeema from a different region/datacenter than your database -- for example, running Skeema on your laptop, when your databases are in AWS. Using `workspace=docker` greatly reduces painful network latency in this scenario, especially if you have a large number of tables. See discussion in #25 for background.
Commits on Nov 2, 2018
  1. Update dep github.com/skeema/tengo for Docker client refactor

    evanelias committed Nov 2, 2018
    This only affects the test suites so far.
Commits on Oct 31, 2018
  1. pull: New option --new-schemas to control new schema handling

    evanelias committed Oct 31, 2018
    Previously, `skeema pull` would always look for schemas (databases) that exist
    on the instance, but have no filesystem representation yet. It will then create
    and populate new directories for these schemas.
    
    This commit adds a new option to control this behavior. It is enabled by
    default, preserving the previous behavior.
    
    However, when using a workflow that involves running `skeema pull development`
    regularly, it may be useful to disable this option. For example, if the
    development environment tends to contain various extra schemas for testing
    purposes, set `skip-new-schemas` in a global or top-level .skeema file's
    `[development]` section to avoid storing these testing schemas in the
    filesystem.
    
    Resolves #38.
  2. push: fix display bug in final error tally (cosmetic only)

    evanelias committed Oct 31, 2018
    When pre-processing errors occurred, such as unsafe statements or bad config,
    these were being double-counted in the final tally of skipped statements. This
    bug was purely cosmetic.
Commits on Oct 30, 2018
  1. workspace: Fix integration test setup for mysql 5.5

    evanelias committed Oct 30, 2018
  2. Fixes for help handler

    evanelias committed Oct 30, 2018
    * Fix `skeema help`, `skeema --help`, etc which were broken by refactor in #44
    
    * Fix `skeema add-environment --help`, which was always broken due to having a
      required positional arg (although other forms like `skeema help
      add-environment` worked previously)
    
    * Add tests to help handlers to ensure no error is returned
  3. init, add-environment: persist more options if specified on CLI

    evanelias committed Oct 30, 2018
    Both `skeema init` and `skeema add-environment` now persist --connect-options
    to the host-level .skeema file, if it was specified on the CLI.
    
    Additionally, `skeema add-environment` now persists --ignore-schema and
    --ignore-table to the host-level .skeema file, matching the existing behavior
    of `skeema init` for these options.
  4. workspace: Refactor locking logic to use sql.Conn instead of sql.Tx

    evanelias committed Oct 30, 2018
    Also, keep the connection holding the lock alive, regardless of wait_timeout,
    by running a trivial SELECT 1 query every 750ms.
Commits on Oct 29, 2018
  1. Fix handling of aggressive server wait_timeout

    evanelias committed Oct 29, 2018
    Previously, if the server had a wait_timeout of 30 seconds or less, Skeema
    may not operate correctly unless --connect-options was used to explicitly set
    a session-level wait_timeout.
    
    Now, Skeema will automatically handle any server-side wait_timeout properly,
    even without --connect-options being used.
    
    See #43 for more background.
  2. Merge pull request #44 from skeema/subpackage-refactor

    evanelias committed Oct 29, 2018
    Major internal refactor to use subpackages
  3. Minor cleanups after reviewing the code in this branch

    evanelias committed Oct 29, 2018
    * When `skeema` exits, gracefully close all connection pools, to avoid aborted
      connection counter/logging in some versions of MySQL
    
    * `skeema diff`: If the only differences for a dir are schema-level DDL, the
      exit code now reflects this as a difference
    
    * applier.TargetGroupChanForDir: skipCount return value is no longer a pointer
    
    * cmd_init.go: Remove unnecessary createOptionFile() function
    
    * cmd_pull.go: Track skipCount by return value, rather than a pointer arg
Commits on Oct 26, 2018
  1. Travis: Add applier package to gofmt check

    evanelias committed Oct 26, 2018
  2. Major internal refactor to use subpackages

    evanelias committed Oct 26, 2018
    This PR moves much of Skeema's logic out of the main package and into several
    new sub-packages, which can be imported by other applications if desired.
    
    Functionality is largely unchanged, and no new features have been added. But a
    few foundational benefits of this work include:
    
    * The codebase no longer assumes a 1:1 mapping between *.sql files and tables.
      This will eventually permit non-table object types (views, procs, grants, etc)
      to be stored in the same repo as schemas, if desired. See #41 for background.
    
    * In the upcoming Skeema 1.1.x series, it will be possible to use a local Docker
      instance for temp schema operations. This performs better in high-latency,
      high-table-count scenarios; see #25 for background.
    
    * The limit on max *.sql file size has been removed. Closes #34.
    
    * `skeema pull` now performs much better than before, as long as --normalize is
      enabled (which it is by default).
    
    * The code supporting `skeema push --concurrent-instances` is now much cleaner
      and more idiomatic.
    
    * Test coverage has been improved.
Commits on Oct 16, 2018
  1. Docs: Escape brackets in ipv6 format example

    evanelias committed Oct 16, 2018
Commits on Sep 4, 2018
  1. diff/push: Fix all remaining bugs in column reordering

    evanelias committed Sep 4, 2018
    This commit updates dep github.com/skeema/tengo, to bring in the new algorithm
    for optimal handling of column reordering. This fixes edge cases where an error
    was generated under certain combinations of column reordering.
    
    This commit also improves error reporting for situations in which verifyDiff
    encounters invalid SQL.
Commits on Aug 29, 2018
  1. password option: if not supplied, check env var MYSQL_PWD

    evanelias committed Aug 29, 2018
    This is now supported for compatibility with the standard MySQL command-line
    client, which also allows the password to be supplied in this fashion. For
    security reasons, this is not recommended though.
Commits on Aug 28, 2018
  1. Dir.InstanceDefaultParams(): Set 3 more session variables

    evanelias committed Aug 28, 2018
    This commit sets three more session variables for Skeema's connections, to
    ensure consistent behavior regardless of what the globals have been set to.
    
    * innodb_strict_mode: now defaults to ON, but may be overridden in
      connect-options to disable if desired. This is typically good to leave
      enabled, which is the global default anyway in MySQL 5.7+.
    
    * sql_quote_show_create: forced to ON, cannot override. This must be enabled
      for Skeema to accurately generate DDL that matches the output of SHOW CREATE
      TABLE.
    
    * default_storage_engine: forced to InnoDB, cannot override. This ensures a
      consistent handling of unlinted *.sql files which do not specify a storage
      engine.
Commits on Aug 24, 2018
  1. init, pull: Always store default-character-set and default-collation

    evanelias committed Aug 24, 2018
    Previously, in schema-level .skeema files, init and pull only populated the
    default-character-set and default-collation options if the schema-level values
    differed from the server's global defaults. This is brittle, as the server
    global default is not tracked by Skeema, and the default server global values
    have changed in MySQL 8.0. As of this commit, the init command will always
    populate the values for each schema; the pull command will now fill in missing
    values for existing schemas.
  2. pull, push, diff: Permit .skeema to override flavor

    evanelias committed Aug 24, 2018
    push/diff: If flavor is set in .skeema, use the value here instead of auto-
    detecting from the target instance. This permits operators to manually configure
    the flavor when needed, for example if running a custom fork/patch-set of the
    mysql-server.
    
    pull: Previously, if a flavor was specified in .skeema but could not be auto-
    detected from the running instance, Skeema would remove the flavor from .skeema.
    Now it leaves it alone, allowing operators to manually configure the flavor
    without risk of `skeema pull` removing it.
  3. MySQL 8.0: fix ability to concurrently drop tables that have foreign …

    evanelias committed Aug 24, 2018
    …keys
    
    This commit updates dep github.com/skeema/tengo, to bring in a fix for an issue
    with MySQL 8.0 compatibility. As of this commit, all existing integration tests
    now pass for MySQL 8.0:
    
      $ SKEEMA_TEST_IMAGES=mysql:8.0 go test -v -cover
      ...
      PASS
      coverage: 84.7% of statements
      ok  	github.com/skeema/skeema	45.545s
  4. Fix several MySQL 8.0 incompatibilities, and fix bug in column reorde…

    evanelias committed Aug 24, 2018
    …ring
    
    This commit updates dep github.com/skeema/tengo, which brings in several MySQL
    8.0 fixes, and also a fix for #36.
    
    Additionally, other Skeema-specific MySQL 8.0 improvements are included in this
    commit:
    
    * The --alter-algorithm option now supports use of the new INSTANT value.
    
    * If the .skeema file includes flavor=mysql:8.0 or flavor=percona:8.0, all
      connections from Skeema will automatically use new session variable
      information_schema_stats_expiry=0. This is necessary to avoid querying stale
      cached data from information_schema, particularly for next auto_increment
      value.
    
    There are still known MySQL 8.0 incompatibilities regarding foreign keys, so
    this commit does not comprehensively consider 8.0 to be supported.
Commits on Aug 7, 2018
  1. Update dep github.com/skeema/tengo - lowercase information_schema col…

    evanelias committed Aug 7, 2018
    … aliases
    
    This commit brings in the latest Go La Tengo commit, to attempt to fix one
    incompatibility with MySQL 8.0. Skeema still does not yet support MySQL 8.0,
    but this is a step towards supporting it in the near future.
Commits on Aug 1, 2018
  1. Track database flavor (vendor and version) in .skeema files

    evanelias committed Aug 1, 2018
    This commit adds a new "flavor" option, which in Skeema v1.0.x is just
    informational (no functional impact). `skeema init`, `skeema pull`, and
    `skeema add-environment` now automatically populate this option in host-level
    .skeema files, to persist the database's vendor (mysql/percona/mariadb) and
    major.minor versions. This information may be used in new features beginning in
    Skeema v1.1.
Commits on Jul 31, 2018
  1. Update dep github.com/skeema/tengo - more test suite refactors

    evanelias committed Jul 31, 2018
    This brings in cleaner abstractions around Docker sandboxes. No real effect
    outside of the test suite yet, but this support will be leveraged in new
    features in the future (e.g. ability to offload the temporary schema operations
    to a local Docker container matching prod's flavor.)
Commits on Jul 26, 2018
  1. Update dep github.com/skeema/tengo

    evanelias committed Jul 26, 2018
    Functionally, this only immediately affects the test suite:
    
    * Test containers are now given specific names (for example
      "skeema-test-mysql-5.6"), and are stopped instead of destroyed at end of
      tests. This permits re-use across test runs, speeding up tests.
    
    * The TENGO_TEST_IMAGES env variable is no longer supported; use
      SKEEMA_TEST_IMAGES instead.
Commits on Jul 13, 2018
  1. Update dep github.com/skeema/tengo - fixes #31

    evanelias committed Jul 13, 2018
    The latest version of Go La Tengo fixes a bug in the sort order of foreign key
    names that have underscore prefixes, which are common when using pt-osc.
    
    The relevant test case is in skeema/tengo@7776645
    via the change to testdata/integration.sql.
Commits on Jul 11, 2018
  1. travis CI: auto-goreleaser upon tagging

    evanelias committed Jul 11, 2018
  2. Default to using a consistent, strict sql_mode. Fixes #31

    evanelias committed Jul 11, 2018
    Unless overridden by --connect-options, Skeema will now use the following
    session-level sql_mode for its connections:
    
    ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
    
    This provides a consistent baseline for behavior, regardless of what global
    default the server is configured with. This is especially helpful for older
    versions of MySQL, which don't default to strict mode. Additionally, this avoids
    problems with servers that use ANSI_QUOTES, which breaks Skeema's behavior.
    
    This commit also prevents using --connect-options to explicitly enable
    ANSI_QUOTES, or the combination ANSI mode which includes it.
Commits on Jul 9, 2018
Commits on Jul 6, 2018
  1. docs: prep for 1.0.0 release; configure goreleaser

    evanelias committed Jul 6, 2018
Commits on Jul 4, 2018
  1. Add support for MySQL 5.5

    evanelias committed Jul 4, 2018