spansql error (want scalar type)

[wyardley]

Describe the Bug
I'm getting an error running the below command with the latest version of migrate

Steps to Reproduce
Steps to reproduce the behavior:

1. My migrations look like
   ALTER TABLE foo ALTER COLUMN createdAt SET OPTIONS (allow_commit_timestamp=true);
   This looks pretty much exactly like the example in https://cloud.google.com/spanner/docs/schema-updates (adding the spaces doesn't make a difference)
2. I ran migrate with the following options '.
   migrate -verbose -path ./spanner/users -database "spanner://projects/fakeproject/instances/test/databases/users?x-clean-statements=True
3. See error

2020/08/15 05:21:25 Read and execute 20200115133304/u xxxx
2020/08/15 05:21:25 error: :1.55: got "SET", want scalar type

(which appears to be a spansql error, so I assume this is coming from migrate vs. the emulator it's talking to)

Expected Behavior
The migration would apply.

Migrate Version
v4.12.2

Loaded Source Drivers
godoc-vfs, gcs, file, s3, github, github-ee, gitlab, go-bindata

Loaded Database Drivers
cassandra, crdb-postgres, firebird, mongodb, mysql, sqlserver, stub, cockroach, neo4j, clickhouse, cockroachdb, firebirdsql, mongodb+srv, postgres, postgresql, redshift, spanner

Go Version
[running linux binary]

Stacktrace
Please provide if available

Additional context
Using emulator to run this in CI

[wyardley]

I tried verifying with an actual spanner instance, just to make sure it's not something wonky with the emulator, and got the same error.

Then downgraded to 4.10.0, and it seems to work there (all the migrations run without the want scalar type error).

Interestingly, the following code seems to work Ok (and error if I put invalid DDL in the same block):

package main

import "cloud.google.com/go/spanner/spansql"
import "fmt"

func main() {

	textString := `ALTER TABLE foo ALTER COLUMN createdAt SET OPTIONS (allow_commit_timestamp=true);`

	_, err := spansql.ParseDDL("", textString)
	if err != nil {
		fmt.Println(err)
	}
}

[wyardley]

Fairly simple / self-contained repo case (again, against real db)

% cat spanner/test/20200815003419_test.up.sql
CREATE TABLE users (
  userId STRING(64) NOT NULL,
  email STRING(256),
  created TIMESTAMP NOT NULL,
) PRIMARY KEY(userId);
% cat spanner/test/20200815003531_alter_set.up.sql
ALTER TABLE users ALTER COLUMN created
  SET OPTIONS (allow_commit_timestamp=true);
% migrate -verbose -path ./spanner/test -database 'spanner://projects/realproject/instances/test/databases/test?x-clean-statements=True' up          

2020/08/15 00:37:45 Start buffering 20200815003419/u test
2020/08/15 00:37:45 Start buffering 20200815003531/u alter_set
2020/08/15 00:37:45 Read and execute 20200815003419/u test
2020/08/15 00:37:49 Finished 20200815003419/u test (read 65.822892ms, ran 4.267760865s)
2020/08/15 00:37:49 Read and execute 20200815003531/u alter_set
2020/08/15 00:37:49 error: :2: got "SET", want scalar type

And, of course, without the x-clean-statements, it barfs completely (odd, though, because Spanner DDL parser does seem to accept semi-colons etc.)

2020/08/15 00:40:31 Start buffering 20200815003419/u test
2020/08/15 00:40:31 Start buffering 20200815003531/u alter_set
2020/08/15 00:40:31 Read and execute 20200815003419/u test
2020/08/15 00:40:31 error: migration failed in line 0: CREATE TABLE users (
  userId STRING(64) NOT NULL,
  email STRING(256),
  created TIMESTAMP NOT NULL,
) PRIMARY KEY(userId);
 (details: rpc error: code = InvalidArgument desc = Error parsing Spanner DDL statement: CREATE TABLE users (\n  userId STRING(64) NOT NULL,\n  email STRING(256),\n  created TIMESTAMP NOT NULL,\n) PRIMARY KEY(userId);\n : Syntax error on line 5, column 22: Expecting \'EOF\' but found an unknown character (\';\').)

I am able to run those DDL statements directly using the CLI

% gcloud spanner  databases ddl update test --instance test --ddl="CREATE TABLE users (userId STRING(64) NOT NULL, email STRING(256), created TIMESTAMP NOT NULL,) PRIMARY KEY(userId);"
Schema updating...done.                                                                                                                                                                
% gcloud spanner  databases ddl update test --instance test --ddl="ALTER TABLE users ALTER COLUMN created SET OPTIONS (allow_commit_timestamp=true);"
Schema updating...done.                                                                                                                                                                
% gcloud spanner  databases ddl describe test --instance test
CREATE TABLE SchemaMigrations (
  Version INT64 NOT NULL,
  Dirty BOOL NOT NULL,
) PRIMARY KEY(Version);

CREATE TABLE users (
  userId STRING(64) NOT NULL,
  email STRING(256),
  created TIMESTAMP NOT NULL OPTIONS (
    allow_commit_timestamp = true
  ),
) PRIMARY KEY(userId);

[wyardley]

Tried adding a fmt.Printf("%q", string(migration)) in database/spanner/spanner.go and got:

"ALTER TABLE users ALTER COLUMN created\n  SET OPTIONS (allow_commit_timestamp=true);\n""

which seems to parse fine if I pass it into the program above.
Commenting out the error check completely allows it to continue, but then bombs out with

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1c7ad0b]

goroutine 1 [running]:
github.com/golang-migrate/migrate/v4/database/spanner.cleanStatements(0xc0003e0000, 0x54, 0x600, 0x600, 0xc000e84960, 0x54, 0x0, 0x0)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/database/spanner/spanner.go:333 +0x13b
github.com/golang-migrate/migrate/v4/database/spanner.(*Spanner).Run(0xc0006b4be0, 0x233d880, 0xc0005f2910, 0xc000790c50, 0x1)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/database/spanner/spanner.go:163 +0x496
github.com/golang-migrate/migrate/v4.(*Migrate).runMigrations(0xc0001fe400, 0xc000c07680, 0xc0001fe400, 0xffffffffffffffff)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/migrate.go:746 +0x685
github.com/golang-migrate/migrate/v4.(*Migrate).Up(0xc0001fe400, 0x2e56cfc8d8fa7, 0x39e08f10)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/migrate.go:284 +0x148
github.com/golang-migrate/migrate/v4/internal/cli.upCmd(0xc0001fe400, 0xffffffffffffffff, 0x2dbc4e0, 0xc0001fe400)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/internal/cli/commands.go:169 +0x11d
github.com/golang-migrate/migrate/v4/internal/cli.Main(0x230b14c, 0x6)
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/internal/cli/main.go:182 +0x523
main.main()
	/Users/wyardley/git/gocode/src/github.com/golang-migrate/migrate/cmd/migrate/main.go:6 +0x39

[wyardley]

Maybe related? googleapis/google-cloud-go#2621

[wyardley]

so, updating to v1.8.1-0.20200815011802-3e0a3eecc468 does fix it. Will work on trying to get them to cut a release.
Not sure if you're willing to update to a non-release version, but will make a PR that does that.