Add actual check of --error command not just that an error exists #52

mjonss · 2022-01-14T16:48:48Z

Currently the --error <ERROR CODE>[,...] only checks if there is an error, not that the error matches, this PR adds such check.

Also since most tests uses error names like ER_NO_SUCH_TABLE a map was needed from name to number, which was generated with perror.

morgo · 2022-01-18T02:23:11Z

I took a look at how many of the existing tests will break: unfortunately it's most :(

I wonder if there is a way to make this behavior transitional? For example the test would need to include the following:

--strict_error_check TRUE

After we've fixed all the tests we remove this flag and have it on all the time. The other option is we divide all the tests by a few people and let them break for a few days while we fix them.

morgo · 2022-01-18T02:30:10Z

Here is the list of failures (58 of 121 fail):

FATA[0000] run test [alter_table1] err: sql:ALTER TABLE t1 CHANGE NAME NAME CHAR(80) not null;: run "ALTER TABLE t1 CHANGE NAME NAME CHAR(80) not null;" at line 43 err Statement succeeded, expected error(s) '8200' 
FATA[0000] run test [alter_table_PK] err: sql:alter table t1 drop i;: run "alter table t1 drop i;" at line 17 err Error 1090: can't drop only column i in table t1 
FATA[0000] run test [alter_table] err: sql:ALTER TABLE warehouse DROP PRIMARY KEY;: run "ALTER TABLE warehouse DROP PRIMARY KEY;" at line 11 err Error 8200: Unsupported drop primary key when the table's pkIsHandle is true 
FATA[0000] run test [auto_increment] err: sql:select last_insert_id();: run "select last_insert_id();" at line 118 err Statement succeeded, expected error(s) 'ER_DUP_ENTRY' 
FATA[0000] run test [builtin] err: sql:SELECT 1, CAST(1234 AS CHAR(3));: run "SELECT 1, CAST(1234 AS CHAR(3));" at line 34 err Statement succeeded, expected error(s) 'Data Too Long.' 
FATA[0000] run test [case] err: sql:SELECT CASE _latin1'a' COLLATE latin1_general_ci
FATA[0000] run test [charset] err: sql:SELECT CONVERT(_latin1'abcd' USING filename);: [parser:1064]You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use [parser:1115]Unknown character set: 'filename' 
FATA[0000] run test [comment_table] err: sql:create table t1 (c1 VARCHAR(10) NOT NULL COMMENT 'c1 comment', c2 INTEGER,c3 INTEGER COMMENT '012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789', c4 INTEGER, c5 INTEGER, c6 INTEGER, c7 INTEGER, INDEX i1 (c1) COMMENT 'i1 comment',INDEX i2(c2)
FATA[0000] run test [composite_index] err: sql:INSERT INTO t (c1, c2, c3) VALUES(11, 22, 33);: run "INSERT INTO t (c1, c2, c3) VALUES(11, 22, 33);" at line 9 err Error 1062: Duplicate entry '11' for key 'PRIMARY' 
FATA[0000] run test [concurrent_ddl] err: run "INSERT INTO t2 VALUES (1);" at line 6 err Error 1146: Table 'concurrent_ddl.t2' doesn't exist 
FATA[0000] run test [create_database] err: sql:CREATE DATABASE create_database2;: run "CREATE DATABASE create_database2;" at line 4 err Error 1007: Can't create database 'create_database2'; database exists 
FATA[0000] run test [create_table] err: sql:INSERT INTO t (c1) VALUES(11);: run "INSERT INTO t (c1) VALUES(11);" at line 20 err Error 1062: Duplicate entry '11' for key 'PRIMARY' 
FATA[0000] run test [datetime_insert] err: sql:create table t (id int, c1 datetime default '12345678');: run "create table t (id int, c1 datetime default '12345678');" at line 48 err Error 1067: Invalid default value for 'c1' 
FATA[0000] run test [daylight_saving_time] err: sql:insert into t value('2011-03-13 02:00:00');: run "insert into t value('2011-03-13 02:00:00');" at line 13 err Error 1292: Incorrect timestamp value: '2011-03-13 02:00:00' for column 'a' at row 1 
FATA[0000] run test [delete] err: sql:DELETE t2 FROM t1 JOIN t2 WHERE t1.a = 10;: run "DELETE t2 FROM t1 JOIN t2 WHERE t1.a = 10;" at line 156 err Statement succeeded, expected error(s) 'ER_UPDATE_WITHOUT_KEY_IN_SAFE_MODE' 
FATA[0000] run test [expression_index] err: sql:INSERT INTO t1 (f2) VALUES ("lorem ipsum");: run "INSERT INTO t1 (f2) VALUES ("lorem ipsum");" at line 75 err Error 1406: Data Too Long, field len 1, data len 11 
FATA[0000] run test [gcol_alter_table] err: sql:alter table t drop c2;: run "alter table t drop c2;" at line 21 err Error 3108: Column 'c2' has a generated column dependency. 
FATA[0000] run test [gcol_blocked_sql_funcs] err: sql:create table t1 (b double generated always as (rand()) virtual);: run "create table t1 (b double generated always as (rand()) virtual);" at line 22 err Error 3102: Expression of generated column 'b' contains a disallowed function. 
FATA[0000] run test [gcol_column_def_options] err: sql:create table t1 (a int, b int generated always as (a+1) stored primary key);: run "create table t1 (a int, b int generated always as (a+1) stored primary key);" at line 106 err Statement succeeded, expected error(s) '1105' 
FATA[0000] run test [gcol_dependenies_on_vcol] err: sql:alter table t1 drop column a;: run "alter table t1 drop column a;" at line 24 err Error 3108: Column 'a' has a generated column dependency. 
FATA[0000] run test [gcol_illegal_expression] err: sql:insert into t (a) value (10);: run "insert into t (a) value (10);" at line 6 err Error 1365: Division by 0 
FATA[0000] run test [gcol_ins_upd] err: sql:insert into t1 (a) values (4);: run "insert into t1 (a) values (4);" at line 268 err Statement succeeded, expected error(s) '1452' 
FATA[0000] run test [grant_dynamic] err: sql:SHOW GRANTS FOR `u1`@`%`;: run "SHOW GRANTS FOR `u1`@`%`;" at line 121 err Statement succeeded, expected error(s) 'ER_NONEXISTING_GRANT' 
FATA[0000] run test [groupby] err: sql:select * from t group by 3 order by c1;: run "select * from t group by 3 order by c1;" at line 31 err Error 1105: Unknown column '3' in 'group statement' 
FATA[0000] run test [insert] err: sql:insert into t values (default(c3), default);: run "insert into t values (default(c3), default);" at line 18 err Error 1054: Unknown column 'c3' in 'field list' 
FATA[0000] run test [insert_update] err: sql:INSERT INTO t1 (id,c1) SELECT 1,NULL
FATA[0000] run test [issue_294] err: sql:insert into t set c1 = 1, c2 = 1, c1 = 2, c2 =2;: run "insert into t set c1 = 1, c2 = 1, c1 = 2, c2 =2;" at line 14 err Error 1110: Column 'c1' specified twice 
FATA[0000] run test [json_functions] err: sql:SELECT JSON_OBJECT( 87, 'name', 'carrot');: run "SELECT JSON_OBJECT( 87, 'name', 'carrot');" at line 133 err Error 1582: Incorrect parameter count in the call to native function 'json_object' 
FATA[0000] run test [json_gcol] err: sql:INSERT INTO t(j) VALUES ('{}');: run "INSERT INTO t(j) VALUES ('{}');" at line 19 err Error 1292: Truncated incorrect INTEGER value: '{}' 
FATA[0000] run test [key] err: sql:alter table t1 add primary key (c1, c2);: run "alter table t1 add primary key (c1, c2);" at line 312 err Error 1068: Multiple primary key defined 
FATA[0000] run test [mariadb_cte_nonrecursive] err: sql:with t(f) as (select * from t1 where b >= 'c')
FATA[0000] run test [mariadb_cte_recursive] err: sql:with recursive
FATA[0000] run test [multi_update] err: sql:revoke all privileges on mysqltest.t1 from mysqltest_1@localhost;: run "revoke all privileges on mysqltest.t1 from mysqltest_1@localhost;" at line 22 err Error 1105: Unknown user: mysqltest_1@localhost 
FATA[0000] run test [mysql_replace] err: sql:alter table t1 engine=heap;: run "alter table t1 engine=heap;" at line 19 err Statement succeeded, expected error(s) 'ErrUnsupportedDDLOperation' 
FATA[0000] run test [partition_innodb] err: sql:create table t1 (a int)
FATA[0000] run test [partition_range] err: sql:create table t1 (a int)
FATA[0000] run test [precedence] err: sql:select FALSE IS NULL IS NULL IS NULL;: run "select FALSE IS NULL IS NULL IS NULL;" at line 109 err Statement succeeded, expected error(s) 'ER_PARSE_ERROR' 
FATA[0000] run test [prepare] err: sql:PREPARE stmt3 FROM 'SELECT id+?, id+? AS id_plus;SELECT id+?, id+? AS id_plus';: run "PREPARE stmt3 FROM 'SELECT id+?, id+? AS id_plus;SELECT id+?, id+? AS id_plus';" at line 7 err Error 8115: Can not prepare multiple statements 
FATA[0000] run test [ps] err: sql:deallocate prepare no_such_statement;: run "deallocate prepare no_such_statement;" at line 29 err Error 8111: Prepared statement not found 
FATA[0000] run test [qualified] err: sql:select *, * from t;: run "select *, * from t;" at line 12 err Error 8122: Wildcard fields without any table name appears in wrong place 
FATA[0000] run test [replace] err: sql:replace into t values (default(c3), default);: run "replace into t values (default(c3), default);" at line 17 err Error 1054: Unknown column 'c3' in 'field list' 
FATA[0000] run test [role] err: sql:GRANT role1 TO peter@clickhost.net;: run "GRANT role1 TO peter@clickhost.net;" at line 58 err Error 1396: Operation GRANT ROLE failed for peter@clickhost.net 
FATA[0000] run test [role] err: sql:GRANT role1 TO peter@clickhost.net;: run "GRANT role1 TO peter@clickhost.net;" at line 58 err Error 1396: Operation GRANT ROLE failed for peter@clickhost.net 
FATA[0000] run test [row] err: sql:SELECT 1 <=>ALL (SELECT 1);: run "SELECT 1 <=>ALL (SELECT 1);" at line 356 err Error 1105: We don't support <=> all or <=> any now 
FATA[0000] run test [sub_query_more] err: sql:SELECT SUM( (SELECT AVG( (SELECT t1.a FROM t2) ) FROM DUAL) ) FROM t1;: run "SELECT SUM( (SELECT AVG( (SELECT t1.a FROM t2) ) FROM DUAL) ) FROM t1;" at line 968 err Statement succeeded, expected error(s) 'ER_INVALID_GROUP_FUNC_USE' 
FATA[0000] run test [sub_query] err: sql:select row(1, 1) + row(1, 1);: run "select row(1, 1) + row(1, 1);" at line 27 err Error 1241: Operand should contain 1 column(s) 
FATA[0000] run test [temp_table] err: sql:SELECT * FROM bogus.t1;: run "SELECT * FROM bogus.t1;" at line 200 err Error 1146: Table 'bogus.t1' doesn't exist 
FATA[0000] run test [timestamp_insert] err: sql:create table t (id int, c1 timestamp default '12345678');: run "create table t (id int, c1 timestamp default '12345678');" at line 55 err Error 1067: Invalid default value for 'c1' 
FATA[0000] run test [time] err: sql:INSERT INTO t (c1) values (-1);: run "INSERT INTO t (c1) values (-1);" at line 60 err Error 1292: Incorrect date value: '-1' for column 'c1' at row 1 
FATA[0000] run test [type_decimal] err: sql:insert into t1 values ("+111111111.11"),("111111111.11"),("-11111111.11");: run "insert into t1 values ("+111111111.11"),("111111111.11"),("-11111111.11");" at line 177 err Statement succeeded, expected error(s) 'checkWarn' 
FATA[0000] run test [type_enum] err: sql:ALTER TABLE t1 CHANGE c1 c1 ENUM('a', '') NOT NULL;: run "ALTER TABLE t1 CHANGE c1 c1 ENUM('a', '') NOT NULL;" at line 214 err Statement succeeded, expected error(s) 'unsupported' 
FATA[0000] run test [type_timestamp] err: sql:SET @@sql_mode='NO_ZERO_IN_DATE,STRICT_ALL_TABLES';: run "SET @@sql_mode='NO_ZERO_IN_DATE,STRICT_ALL_TABLES';" at line 455 err Statement succeeded, expected error(s) 'ER_INVALID_DEFAULT' 
FATA[0000] run test [type_time] err: sql:insert into t1 values("10:22:33"),("12:34:56.78"),(10),(1234),(123456.78),(1234559.99),("1"),("1:23"),("1:23:45"), ("10.22"), ("-10  1:22:33.45"),("20 10:22:33"),("1999-02-03 20:33:34");: run "insert into t1 values("10:22:33"),("12:34:56.78"),(10),(1234),(123456.78),(1234559.99),("1"),("1:23"),("1:23:45"), ("10.22"), ("-10  1:22:33.45"),("20 10:22:33"),("1999-02-03 20:33:34");" at line 11 err Statement succeeded, expected error(s) 'checkWarn' 
FATA[0000] run test [type_uint] err: sql:insert into t1 values (-1);: run "insert into t1 values (-1);" at line 13 err Error 1264: Out of range value for column 'this' at row 1 
FATA[0000] run test [union] err: sql:SELECT 1 UNION SELECT 0, 1;: run "SELECT 1 UNION SELECT 0, 1;" at line 10 err Error 1222: The used SELECT statements have a different number of columns 
FATA[0000] run test [update] err: sql:SET SESSION sql_safe_updates = DEFAULT;: run "SET SESSION sql_safe_updates = DEFAULT;" at line 424 err Statement succeeded, expected error(s) 'ER_UPDATE_WITHOUT_KEY_IN_SAFE_MODE' 
FATA[0000] run test [window_functions] err: sql:SELECT a, SUM(b) OVER () FROM tb GROUP BY a; # illegal b reference
FATA[0000] run test [with_non_recursive] err: sql:with qn as (select * from tmp) select /*+ merge(qn,qn1) */ * from qn, qn qn1;: run "with qn as (select * from tmp) select /*+ merge(qn,qn1) */ * from qn, qn qn1;" at line 694 err Error 1146: Table 'with_non_recursive.tmp' doesn't exist

morgo · 2022-01-18T18:56:04Z

@mjonss I discussed this with others today, and @Alkaagr81 had a suggestion on how we can retrofit this to the current tests a little better. Her proposal was:

We validate that following --error the next line does in fact produce an error.
We print a warning if the error code/error number does not match.

The warnings will be annoying to start with, but we can clean them up and then validate that they match correctly.

mjonss · 2022-01-19T15:31:15Z

@mjonss I discussed this with others today, and @Alkaagr81 had a suggestion on how we can retrofit this to the current tests a little better. Her proposal was:

We validate that following --error the next line does in fact produce an error.

We print a warning if the error code/error number does not match.

The warnings will be annoying to start with, but we can clean them up and then validate that they match correctly.

Thanks for the comments!

I have updated the patch with a --check-error flag (default off) to enable the actual checking and only logging Warnings when not matching.
However currently the mysql-tester is run with --log-level=error which should be changed to --log-level=warn when making use of this PR.

morgo · 2022-01-19T16:47:10Z

Thanks! I've confirmed it works as expected:

morgo@ubuntu:~/go/src/github.com/morgo/tidb-test/mysql_test$ ./run-tests.sh -s ~/go/src/github.com/morgo/tidb/bin/tidb-server -r alter_table1
skip building tidb-server, using existing binary: /home/morgo/go/src/github.com/morgo/tidb/bin/tidb-server
building mysql-test binary: ./mysql_test
build mysql_test successfully
start tidb-server, log file: ./mysql-test.out
tidb-server(PID: 1234183) started
record result for case: "alter_table1"
INFO[0000] recording tests: [alter_table1]              
WARN[0000] --error will simply accept zero or more errors! (i.e. not even check for errors!) 
WARN[0000] Create new db&{0 0xc000010090 0 {0 0} [0xc0001e26c0] map[] 0 1 0xc0000262a0 false map[0xc0001e26c0:map[0xc0001e26c0:true]] map[] 0 0 0 0 <nil> 0 0 0 0 0x49d1a0} 
WARN[0000] alter_table1:43 query succeeded, but expected error(s)! (expected errors: 8200) (query: ALTER TABLE t1 CHANGE NAME NAME CHAR(80) not null;) 
WARN[0000] alter_table1:259 query succeeded, but expected error(s)! (expected errors: ErrUnsupportedDDLOperation) (query: alter table t1 engine = myisam;) 
WARN[0000] alter_table1:267 query succeeded, but expected error(s)! (expected errors: ErrUnsupportedDDLOperation) (query: alter table t1 engine = heap;) 
WARN[0000] alter_table1:529 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE m1 ENABLE KEYS, ALGORITHM= INPLACE;) 
WARN[0000] alter_table1:569 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE t1 ADD COLUMN (c4 INT), ALGORITHM= INPLACE;) 
WARN[0000] alter_table1:591 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED) (query: ALTER TABLE m1 ENABLE KEYS, LOCK= NONE;) 
WARN[0000] alter_table1:593 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED) (query: ALTER TABLE m1 ENABLE KEYS, LOCK= SHARED;) 
WARN[0000] alter_table1:611 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE t1 ADD INDEX i4(b), ALGORITHM= COPY, LOCK= NONE;) 
WARN[0000] alter_table1:613 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE t1 ADD INDEX i5(b), ALGORITHM= COPY, LOCK= SHARED;) 
WARN[0000] alter_table1:615 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE t1 ADD INDEX i6(b), ALGORITHM= COPY, LOCK= EXCLUSIVE;) 
WARN[0000] alter_table1:618 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED) (query: ALTER TABLE m1 ENABLE KEYS, ALGORITHM= INPLACE, LOCK= NONE;) 
WARN[0000] alter_table1:620 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED) (query: ALTER TABLE m1 ENABLE KEYS, ALGORITHM= INPLACE, LOCK= SHARED;) 
WARN[0000] alter_table1:622 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE m1 ENABLE KEYS, ALGORITHM= INPLACE, LOCK= EXCLUSIVE;) 
WARN[0000] alter_table1:624 query succeeded, but expected error(s)! (expected errors: ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (query: ALTER TABLE m1 ENABLE KEYS, ALGORITHM= COPY, LOCK= NONE;) 
WARN[0000] alter_table1:754 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:754 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 AVG_ROW_LENGTH 10;) 
WARN[0000] alter_table1:756 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:756 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE tm1 AVG_ROW_LENGTH 10;) 
WARN[0000] alter_table1:758 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:758 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 CHECKSUM 1;) 
WARN[0000] alter_table1:760 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:760 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE tm1 CHECKSUM 1;) 
WARN[0000] alter_table1:764 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:764 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 MAX_ROWS 100;) 
WARN[0000] alter_table1:766 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:766 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE tm1 MAX_ROWS 100;) 
WARN[0000] alter_table1:768 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:768 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 MIN_ROWS 1;) 
WARN[0000] alter_table1:770 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:770 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE tm1 MIN_ROWS 1;) 
WARN[0000] alter_table1:772 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:772 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 PACK_KEYS 1;) 
WARN[0000] alter_table1:774 Unknown named --error ErrUnsupportedDDLOperation 
WARN[0000] alter_table1:774 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE tm1 PACK_KEYS 1;) 
WARN[0000] alter_table1:822 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: ALTER TABLE db ENGINE=innodb;) 
WARN[0000] alter_table1:824 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: ALTER TABLE user ENGINE=memory;) 
WARN[0000] alter_table1:826 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.proc' doesn't exist) (query: ALTER TABLE proc ENGINE=heap;) 
WARN[0000] alter_table1:828 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.func' doesn't exist) (query: ALTER TABLE func ENGINE=csv;) 
WARN[0000] alter_table1:830 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.event' doesn't exist) (query: ALTER TABLE event ENGINE=merge;) 
WARN[0000] alter_table1:832 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.servers' doesn't exist) (query: ALTER TABLE servers ENGINE=merge;) 
WARN[0000] alter_table1:834 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.procs_priv' doesn't exist) (query: ALTER TABLE procs_priv ENGINE=memory;) 
WARN[0000] alter_table1:836 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: ALTER TABLE tables_priv ENGINE=heap;) 
WARN[0000] alter_table1:838 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: ALTER TABLE columns_priv ENGINE=csv;) 
WARN[0000] alter_table1:840 query failed with non expected error(s)! (ER_NO_SUCH_TABLE != ER_UNSUPPORTED_ENGINE) (err: Error 1146: Table 'mysql.time_zone' doesn't exist) (query: ALTER TABLE time_zone ENGINE=merge;) 
WARN[0000] alter_table1:842 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: ALTER TABLE help_topic ENGINE=merge;) 
WARN[0000] alter_table1:844 query failed with non expected error(s)! (ER_TABLE_EXISTS_ERROR != ER_UNSUPPORTED_ENGINE) (err: Error 1050: Table 'mysql.db' already exists) (query: CREATE TABLE db (dummy int) ENGINE=innodb;) 
WARN[0000] alter_table1:846 query failed with non expected error(s)! (ER_TABLE_EXISTS_ERROR != ER_UNSUPPORTED_ENGINE) (err: Error 1050: Table 'mysql.user' already exists) (query: CREATE TABLE user (dummy int) ENGINE=memory;) 
WARN[0000] alter_table1:848 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE proc (dummy int) ENGINE=heap;) 
WARN[0000] alter_table1:850 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE func (dummy int) ENGINE=csv;) 
WARN[0000] alter_table1:852 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE event (dummy int) ENGINE=merge;) 
WARN[0000] alter_table1:854 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE servers (dummy int) ENGINE=merge;) 
WARN[0000] alter_table1:856 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE procs_priv (dummy int) ENGINE=memory;) 
WARN[0000] alter_table1:858 query failed with non expected error(s)! (ER_TABLE_EXISTS_ERROR != ER_UNSUPPORTED_ENGINE) (err: Error 1050: Table 'mysql.tables_priv' already exists) (query: CREATE TABLE tables_priv (dummy int) ENGINE=heap;) 
WARN[0000] alter_table1:860 query failed with non expected error(s)! (ER_TABLE_EXISTS_ERROR != ER_UNSUPPORTED_ENGINE) (err: Error 1050: Table 'mysql.columns_priv' already exists) (query: CREATE TABLE columns_priv (dummy int) ENGINE=csv;) 
WARN[0000] alter_table1:862 query succeeded, but expected error(s)! (expected errors: ER_UNSUPPORTED_ENGINE) (query: CREATE TABLE time_zone (dummy int) ENGINE=merge;) 
WARN[0000] alter_table1:864 query failed with non expected error(s)! (ER_TABLE_EXISTS_ERROR != ER_UNSUPPORTED_ENGINE) (err: Error 1050: Table 'mysql.help_topic' already exists) (query: CREATE TABLE help_topic (dummy int) ENGINE=merge;) 
WARN[0000] alter_table1:946 query failed with non expected error(s)! (ER_TOO_LONG_KEY != ER_ALTER_OPERATION_NOT_SUPPORTED_REASON) (err: Error 1071: Specified key was too long; max key length is 3072 bytes) (query: ALTER TABLE t1 CHANGE fld1 fld1 VARCHAR(3073), ALGORITHM= INPLACE;) 
WARN[0000] alter_table1:949 Unknown named --error ER_INDEX_KEY_LENGTH_EXCEED_THE_LIMIT 
WARN[0000] alter_table1:949 query failed with non expected error(s)! (ER_TOO_LONG_KEY != ER_INDEX_KEY_LENGTH_EXCEED_THE_LIMIT) (err: Error 1071: Specified key was too long; max key length is 3072 bytes) (query: ALTER TABLE t1 CHANGE fld1 fld1 VARCHAR(3073), ALGORITHM= INSTANT;) 
WARN[0000] alter_table1:1016 query succeeded, but expected error(s)! (expected errors: ER_TOO_LONG_KEY) (query: ALTER TABLE t1 MODIFY COLUMN name TINYTEXT;) 
WARN[0000] alter_table1:1078 query succeeded, but expected error(s)! (expected errors: ER_TRUNCATED_WRONG_VALUE) (query: ALTER TABLE t1 ADD COLUMN fld2 DATETIME NOT NULL, ALGORITHM=INPLACE;) 
WARN[0000] alter_table1:1081 query succeeded, but expected error(s)! (expected errors: ER_TRUNCATED_WRONG_VALUE) (query: ALTER TABLE t1 ADD COLUMN fld4 DATETIME NOT NULL, ALGORITHM=COPY;) 
./t/alter_table1.test: ok! 491 test cases passed, take time 0.714043782 s
INFO[0000] run test [alter_table1] ok                   

Great, All tests passed
mysqltest end
tidb-server(PID: 1234183) stopped

It looks like you might need to import the tidb error code constants into your PR? i.e.

WARN[0000] alter_table1:768 query failed with non expected error(s)! (8200 != ErrUnsupportedDDLOperation) (err: Error 8200: This type of ALTER TABLE is currently unsupported) (query: ALTER TABLE ti1 MIN_ROWS 1;)

Otherwise LGTM

dveeden · 2022-11-08T11:04:20Z

@mjonss could you fix the conflicts?

Makefile

generate_perror/main.go

dveeden · 2023-11-21T16:22:02Z

Makefile

@@ -15,3 +20,5 @@ clean:
 	go clean -i ./...
 	rm -rf mysql-tester

+gen_perror:


Suggested change

gen_perror:

build_gen_perror:

And then maybe something like this:

gen_perror: build_gen_perror ./gen_perror

Also gen_perror (the current one) should be added to .PHONY:

.gitignore

dveeden · 2023-11-21T16:49:44Z

gen_perror failed for me as it ran out of resources. The problem is that cmd.Start() is called many times without a call to cmd.Wait() to release resources. See also https://pkg.go.dev/os/exec#Cmd.Start

If more performance is needed you might want to use a WaitGroup.

The fix:

diff --git a/generate_perror/main.go b/generate_perror/main.go
index 6c0b1f7..f37abc6 100644
--- a/generate_perror/main.go
+++ b/generate_perror/main.go
@@ -140,6 +140,7 @@ func main() {
                                checkNewErr(m[2], i, NameToNum)
                        }
                }
+               cmd.Wait()
        }
        if maxError >= 1000 {
                fmt.Printf("\r")

dveeden · 2023-11-21T20:28:58Z

Do we need both versions? Can't we only use ER_DB_DROP_EXISTS ?

        "ER_DB_DROP_EXISTS": 1008,
        "ErrDBDropExists": 1008,

generate_perror/main.go

src/main.go

mjonss · 2023-11-21T20:44:36Z

Do we need both versions? Can't we only use ER_DB_DROP_EXISTS ?
        "ER_DB_DROP_EXISTS": 1008,
        "ErrDBDropExists": 1008,

Some examples I've seen:

--error ErrUnsupportedDDLOperation
--error ErrAutoincReadFailed
--error ErrUnknownEngine
--error ErrOperandColumns

So allowing both makes less existing tests to be modified.

dveeden · 2023-11-21T20:45:01Z

Could we use something like this?

//go:generate ../gen_perror

https://go.dev/blog/generate

generate_perror/main.go

dveeden · 2023-11-21T21:58:15Z

src/main.go

@@ -1055,6 +1131,16 @@ func main() {
 	flag.Parse()
 	tests := flag.Args()
 	startTime := time.Now()
+	if ll := os.Getenv("LOG_LEVEL"); ll != "" {


Can't we use a flag like -d/-debug or something similar? I think that makes it easier to see from the usage output how to set the loglevel. We could use both if CI or something else depends on this.

there is already a '-log-level' command line option/flag, does that work?

mjonss · 2023-11-21T22:36:19Z

Could we use something like this?
//go:generate ../gen_perror
https://go.dev/blog/generate

Yes, but I don't think it is worth doing it in this PR, I wrote that code long time ago just to avoid a shell script oneliner (see the comment) and I don't expect the generator to be run very often.

src/main.go

Defined2014 · 2023-11-22T03:18:00Z

generate_perror/main.go

+)
+
+func init() {
+	flag.StringVar(&tidbCodePath, "path", "../tidb", "Path to TiDB source code root directory.")


Does this mean we need to rely on tidb?

Yes, in the same way as it relies on perror (a binary from the mysql distribution, to map error numbers to error code names).
If TiDB would have a similar program/binary/utility, to map error codes to names, we could use that instead of reading the source code.

But since we are generating the perror.go file, with the mapping, and that is checked into the mysql-tester repository, it only depends on perror and tidb repository when it needs to be updated. So it does not have any direct dependencies, only needs the dependencies when new error codes needs to be added.

generate_perror/main.go

Co-authored-by: Daniël van Eeden <github@myname.nl>

Defined2014 · 2023-11-23T01:54:55Z

r/example.result

+6
+1 SELECT;
+Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 1 near "1 SELECT;" 
+2 SELECT;


Why this SQL doesn't have any error message? Seems error no begin with 0 will ignore the error msg. Does it same as mysql?

Checked MySQL-Server, they are same.

bb7133

LGTM

Add actual check of --error command not just that an error exists

50cd967

Added --check-error flag (default false) to keep old behavior unless set

503d997

Added tool for generate perror which also includes TiDB error codes

d6aaba0

morgo previously approved these changes Jan 24, 2022

View reviewed changes

Merge remote-tracking branch 'pingcap/master' into check-error

9f216d9

mjonss dismissed morgo’s stale review via 9f216d9 November 9, 2022 07:16

mjonss mentioned this pull request Nov 21, 2023

MySQL Tester Improvement Plan #110

Open

10 tasks

mjonss added 2 commits November 21, 2023 16:10

Merge remote-tracking branch 'pingcap/master' into check-error

9b48712

Cleanup from manual merge (removed commented out code)

f765db4

dveeden reviewed Nov 21, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

dveeden reviewed Nov 21, 2023

View reviewed changes

generate_perror/main.go Outdated Show resolved Hide resolved

dveeden reviewed Nov 21, 2023

View reviewed changes

generate_perror/main.go Outdated Show resolved Hide resolved

removed duplicate perror and updated warning for unset --check-error

a9d9a88

dveeden reviewed Nov 21, 2023

View reviewed changes

.gitignore Show resolved Hide resolved

mjonss added 3 commits November 21, 2023 17:54

Added wait for exec.Command().Start()

c493931

Fixed log levels

bec7b97

Lowered log levels

6a94e90

dveeden reviewed Nov 21, 2023

View reviewed changes

generate_perror/main.go Show resolved Hide resolved

dveeden reviewed Nov 21, 2023

View reviewed changes

generate_perror/main.go Outdated Show resolved Hide resolved

dveeden previously approved these changes Nov 21, 2023

View reviewed changes

dveeden reviewed Nov 21, 2023

View reviewed changes

src/main.go Outdated Show resolved Hide resolved

mjonss added 4 commits November 21, 2023 21:59

Added error check in generate_perror

2e026ea

code cleanup (commented out code only)

dd155fd

Code cleanup

260499d

simplify the logging of gen_perror

555cefe

dveeden previously approved these changes Nov 21, 2023

View reviewed changes

dveeden reviewed Nov 21, 2023

View reviewed changes

generate_perror/main.go Outdated Show resolved Hide resolved

dveeden reviewed Nov 21, 2023

View reviewed changes

Updated to go 1.21

8373650

Used the fileHeader constant

85f2af2

mjonss dismissed dveeden’s stale review via 85f2af2 November 21, 2023 22:43

fixed make test

606e03a

Defined2014 reviewed Nov 22, 2023

View reviewed changes

src/main.go Show resolved Hide resolved

Defined2014 reviewed Nov 22, 2023

View reviewed changes

mjonss added 2 commits November 22, 2023 10:30

Added support for --error with --replace_regex

7e1d9e1

Added support for --error with --replace_regex

2cf23e5

dveeden previously approved these changes Nov 22, 2023

View reviewed changes

dveeden reviewed Nov 22, 2023

View reviewed changes

generate_perror/main.go Outdated Show resolved Hide resolved

Update generate_perror/main.go

4d06c73

Co-authored-by: Daniël van Eeden <github@myname.nl>

mjonss dismissed dveeden’s stale review via 4d06c73 November 22, 2023 21:11

dveeden approved these changes Nov 22, 2023

View reviewed changes

Defined2014 reviewed Nov 23, 2023

View reviewed changes

Defined2014 approved these changes Nov 23, 2023

View reviewed changes

bb7133 approved these changes Nov 24, 2023

View reviewed changes

bb7133 merged commit 17b728e into pingcap:master Nov 24, 2023
1 check passed

mjonss deleted the check-error branch November 24, 2023 08:23

This was referenced Dec 6, 2023

Added --replace_result #119

Open

Make mysql-tester less verbose/confusing #46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add actual check of --error command not just that an error exists #52

Add actual check of --error command not just that an error exists #52

mjonss commented Jan 14, 2022

morgo commented Jan 18, 2022

morgo commented Jan 18, 2022

morgo commented Jan 18, 2022

mjonss commented Jan 19, 2022

morgo commented Jan 19, 2022

dveeden commented Nov 8, 2022

dveeden Nov 21, 2023

dveeden Nov 21, 2023

dveeden commented Nov 21, 2023 •

edited

Loading

dveeden commented Nov 21, 2023

mjonss commented Nov 21, 2023

dveeden commented Nov 21, 2023 •

edited

Loading

dveeden Nov 21, 2023

mjonss Nov 21, 2023

mjonss commented Nov 21, 2023

Defined2014 Nov 22, 2023

mjonss Nov 22, 2023

Defined2014 Nov 23, 2023

Defined2014 Nov 23, 2023 •

edited

Loading

Defined2014 Nov 23, 2023

bb7133 left a comment

Add actual check of --error command not just that an error exists #52

Add actual check of --error command not just that an error exists #52

Conversation

mjonss commented Jan 14, 2022

morgo commented Jan 18, 2022

morgo commented Jan 18, 2022

morgo commented Jan 18, 2022

mjonss commented Jan 19, 2022

morgo commented Jan 19, 2022

dveeden commented Nov 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dveeden commented Nov 21, 2023 • edited Loading

dveeden commented Nov 21, 2023

mjonss commented Nov 21, 2023

dveeden commented Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjonss commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Defined2014 Nov 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bb7133 left a comment

Choose a reason for hiding this comment

dveeden commented Nov 21, 2023 •

edited

Loading

dveeden commented Nov 21, 2023 •

edited

Loading

Defined2014 Nov 23, 2023 •

edited

Loading