1.39.4

Merged PRs

dolt

• 7979: Allow pulling from a remote if the only changes are to ignored tables.
  This loosens the restrictions for pulling from remotes: instead of requiring that there are no working changes, we allow working changes but only to ignored tables.
  If there are conflicting changes to ignored tables (which is rare, since ignored tables shouldn't be pushed to remotes in the first place), this will still abort the pull later when it's computing the new root hash.
  This is, IMO, better than what git does. Git will just overwrite the ignored files without warning.
• 7977: integration-tests/bats: Add some waits on the remotesrv_pid exit for bats tests which spawn a background remotesrv.
  Attempts to fix some observed flakiness in bats tests.
• 7975: implemented dolt_hashof_db function
  Implemented dolt_hashof_db() function which returns the root hash of a database.
• 7967: go/libraries/doltcore/remotesrv: grpc.go: Respect X-Forwarded-Proto when generating HTTP download links in the gRPC server.
  Fixes #7961.
• 7965: Bug fix: Allow unresolved FKs to merge with resolved FKs
  Dolt foreign keys can be in a "resolved" or "unresolved" state. A resolved FK has resolved the table and columns it references, and contains unique identifiers for the referenced columns. An unresolved FK only knows the table and column names that it references. Because of these two states, the way Dolt matches FKs differs depending on whether each key is resolved or unresolved.
  Dolt has logic (ForeignKeyCollection.GetMatchingKey()) to match a resolved FK with an unresolved FK, but this function didn't support matching an unresolved FK with a resolved FK. That code assumed that the ForeignKeyCollection would always be from an ancestor root value and therefore it wasn't valid for the ancestor to be resolved, while a more recent root value was unresolved. However, since then, we have used this logic in our root merging logic that breaks that assumption.
  In a multi-session environment, one client can create a table with an unresolved FK, then a second session can load that table, resolve the FK, and commit the changes to disk. If the first session still contains references to the unresolved FK, then when it goes to commit, Dolt's merge logic wasn't able to match the unresolved FK in the session with the resolved FK that was written to disk, and the FK constraints were silently dropped from the new table version.
  This PR adds a new parameter to ForeignKeyCollection.GetMatchingKey() to allow the caller to control whether a resolved FK should match with unresolved FKs or not. This means ForeignKeyCollection.GetMatchingKey() doesn't have to assume its receiver instance is a ForeignKeyCollection from an ancestor root value, and instead the caller is responsible for specifying which behavior is needed.
  Related to #7956
• 7955: [types] cache frequently read value store chunks (like working set roots)
• 7952: Made drop table work with search path

go-mysql-server

• 2537: Update generated index names to match MySQL
  A customer pointed out that when we add indexes with generated names, we don't generate the same names as MySQL. Specifically:
  • When a FK is added with an explicit constraint name, that name should be used to name the automatically created index, if one is created.
  • Secondary indexes are named after the first column in the index in MySQL, not by joining all the columns together.
  
  Customer issue: https://github.com//issues/7960 Dolt PR with test fixes: https://github.com//pull/7974
• 2536: Rename generated FK names when their table is renamed
  Updates our rename table logic to match MySQL's behavior of updating auto-generated foreign key names to match the new table name.
  Customer issue: #7959
  Dolt companion PR: #7968
• 2535: Fix UPDATE JOIN matchedRows
  fixes: #7957
  Main question is how thorough we want to make the child iter check. Should all iterators implement a ChildIter interface?
• 2534: Implement row alias expressions (INSERT ... VALUES (...) AS new_tbl ON DUPLICATE x = new_tbl.x)
  When inserting values, the user can specify names for both the source table and columns which are used in ON DUPLICATE expressions. It looks like either of the below options:

  INSERT INTO tbl VALUES (1, 2) AS tbl_new ON DUPLICATE KEY b = tbl_new.b;
  INSERT INTO tbl VALUES (1, 2) AS tbl_new(a_new, b_new) ON DUPLICATE KEY b = b_new;

  This replaces the previous (now-deprecated) syntax:

  INSERT INTO tbl VALUES (1, 2) ON DUPLICATE KEY b = VALUES(b);

  Supporting both syntaxes together was non-trivial because it means there's now two different ways to refer to the same column. While he had an existing way to "redirect" one column name to another, this only worked for unqualified names (no table name), and it overrode the normal name resolution rules, which meant we would fail to detect cases that should be seen as ambiguous.
  Previously, we would implement references to the inserted values by using a special table named "__new_ins". I implemented this by keeping that as the default, but using the row alias instead of one was provided. We then create a map from the destination table names to column aliases, and use that map to rewrite expressions that appear inside the VALUES() function.
• 2533: Table name validation folds strings
  fixes: #7958
• 2532: Move json_function_tests.go and json tests that depend on it to their own package.
  This ensures that non-test code in sql/expression/function/json doesn't depend on testify, which is a library that we only want to depend on for tests.
• 2530: Bug Fix: Index name case-insensitivity
  MySQL index names are case-insensitive, but GMS' memory implementation wasn't handling them that way. This makes index names case-insensitive.
  Related to #7945

Closed Issues

• 7970: Add support for DOLT_HASHOF_DB()
• 7957: Dolt returns wrong number of affected rows for UPDATE ... JOIN with clientFoundRows=true
• 7958: UPDATE ... JOIN fails for tables containing capital letters
• 7945: Dolt panics when renaming index containing capital letters
• 7944: Dolt panics on subquery in IF statement in procedure

1.39.3

Merged PRs

dolt

• 7947: Bug Fix: Index name case-insensitivity
  MySQL index names are case-insensitive, but Dolt's index implementation wasn't handling them that way. This makes index names case-insensitive.
  Customer issue: #7945
  New enginetests added in GMS PR: dolthub/go-mysql-server#2530
• 7941: Bug fix for AllSchemas method for schemas
• 7940: dolt sql slash cmds
  Add the ability to run some dolt commands directly from the dolt sql shell.
  Fixes: #6874
• 7933: Update get-mysql-dolt-job-json.sh
  TPS comparison is inverted compared t latency_p95 comparison.
• 7931: Update get-mysql-dolt-job-json.sh

go-mysql-server

• 2531: Bug Fix: Finalize subqueries in IfConditionals when applying stored procedures
  When applying a stored procedure to a CALL statement, we weren't calling finalizeSubqueries() on any subqueries in IfConditional expressions, which caused the subquery to not have a NodeExecBuilder populated.
  Customer issue: #7944
• 2529: Fix global decimal.MarshalJSONWithoutQuotes overwrite
  The decimal.MarshalJSONWithoutQuotes is a global variable.
  By setting this value then this can cause problems with any other code that does not expect this value to be changed.
  Instead using a custom encoder to ensure that the marshalling behaviour is as expected without changing the global value ensure that this will not cause compatibility issues with other projects.
  This code is covered both by existing tests, and an additional one in this PR.
  (if the custom encode switch case is not added, but the global variables are, then the tests fail).
• 2528: Bug fix for unwrapping a privileged db
• 2524: Adding @@max_binlog_size system variable
  https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_max_binlog_size
• 2523: Added additional analyzer hooks for integrators
• 2522: More INSERT short-circuits
  Only run an "on update" code block when expressions are non-nil. Directly compare sql mode default string, rather than lowercasing every time.
• 2519: IndexedTableAccess gets indexing fast path

vitess

• 350: Refactoring BinlogStream type into BinlogMetadata
  The mysql.BinlogStream type from Vitess was a little awkward to use, and seems to have been mostly intended as test code. This gives it a more descriptive name and makes it a little easier to pass around struct copies without concurrency issues from a shared instance.
• 349: Fixed timestamp bindvar formatting to match MySQL string expectation
• 348: Allowing caching plugin to be specified in string quotes
  The CREATE USER ... IDENTIFIED WITH syntax (MySQL ref) allows the caching plugin to be specified in string quotes, but our parser only supported identifier quotes.
  This came up as part of binlog replication testing – MySQL was sending a CREATE USER statement from the primary to a Dolt replica, but Dolt wasn't able to parse the statement because of the use of string quotes around the caching plugin name.

Closed Issues

• 6874: Embed cli command in dolt sql
• 2289: First Unique Key in a keyless table should be represented as a primary key

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	2.97	1.4
groupby_scan	13.22	17.01	1.3
index_join	1.34	5.28	3.9
index_join_scan	1.27	2.22	1.7
index_scan	34.95	52.89	1.5
oltp_point_select	0.18	0.5	2.8
oltp_read_only	3.49	8.13	2.3
select_random_points	0.34	0.81	2.4
select_random_ranges	0.39	0.95	2.4
table_scan	34.95	54.83	1.6
types_table_scan	75.82	137.35	1.8
reads_mean_multiplier			2.1

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.21	0.8
oltp_insert	3.82	3.07	0.8
oltp_read_write	8.58	14.73	1.7
oltp_update_index	3.89	3.19	0.8
oltp_update_non_index	3.89	3.13	0.8
oltp_write_only	5.37	6.55	1.2
types_delete_insert	7.7	6.79	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	99.71	25.77	4.2
tpcc_tps_multiplier			3.9

Overall Mean Multiple	2.33

1.39.2

Merged PRs

dolt

• 7930: Bump mysql2 from 3.9.7 to 3.9.8 in /integration-tests/mysql-client-tests/node
  Bumps mysql2 from 3.9.7 to 3.9.8.
• 7929: dolt fetch default spec from empty repo should return silently
  Git fetch returns without error when you fetch the default refspec. When you fetch a specific ref you get an error. Dolt now matches this behavior.
  Fixes: #7928
• 7925: apply filter-branch changes to working/staged changes
  This PR adds support for a --apply-to-uncommitted option to dolt filter-branch, which applies the filter-branch changes to the working and staged roots.
  fixes #7902
• 7923: [dsess] Cache checks lookup for TPC-C update
• 7922: [writer] skip more deserialization steps in getTableWriter
• 7900: prevent dolt filter branch when it would overwrite unchecked branch's working set
  Turns out other branches can have working sets, and dolt-filter branch would drop those. PR prevents that from happening.
  adding tests to comments I missed here:
  #7895
• 7898: Added workflow for checking DoltgreSQL
  This adds a new workflow that runs a subset of the tests in DoltgreSQL to check for any major integration errors. The workflow does not fail if errors are encountered. Instead, it creates a comment stating that failures were found. If no failures were found, then no comment is made.
• 7892: dolt admin archive
  This hidden admin command will convert the table files in oldgen into archive files, then update the manifest so that you can run queries against the archive for performance testing. Currently we assume that dolt gc has been run immediately prior to using this command.
  After the build is complete, we lookup every chunk in the archive using the index of the originating table file. We then verify each chunk's key checks out. If this verification fails, exit status 1.
  Lot of rough edges still:
  • Currently no feedback as the build progresses. This is annoying because it can take a fair amount of time
  • ChunkSource interface is single threaded, so getMany and hasMany are not going to perform well.
  • Lacking checks to ensure that the server isn't running and we have the LOCK on oldgen.
  • No bats tests, and this is kind of a temporary thing. There are go tests on key bits.
• 7863: Use the search path to resolve table names in Doltgres
  Doltgres enables the UseSearchPath global at startup, which triggers this behavior.
  This is a shim to get a proof of concept of this behavior working faster. A better solution, coming next, involves making this behavior pluggable and putting this logic in the Doltgres package, not in Dolt.
  Companion PRs:
  dolthub/go-mysql-server#2498
  dolthub/doltgresql#269

go-mysql-server

• 2520: Default sql mode for common path
  Bit strange & verbose, but has a noticeable effect for small queries.
  perf here: #7915
• 2519: IndexedTableAccess gets indexing fast path
• 2518: Short circuit for update/delete
  Simple updates and deletes skip most of analysis.
  perf here: #7907
• 2517: Improve correctness and error messages for JSON functions.
  MySQL doesn't do this and neither should we.
  MySQL:

  mysql> select JSON_INSERT("null", "$.a", 1);
  +-------------------------------+
  | JSON_INSERT("null", "$.a", 1) |
  +-------------------------------+
  | null                          |
  +-------------------------------+
  1 row in set (0.00 sec)
  mysql> select JSON_INSERT("null", "$.a", 1) is null;
  +---------------------------------------+
  | JSON_INSERT("null", "$.a", 1) is null |
  +---------------------------------------+
  |                                     0 |
  +---------------------------------------+
  

  The only time we should be coercing a JSON-null document into SQL-null is for JSON_EXTRACT (for paths other than "$") and JSON_VALUE (for all paths). But these are already handled separately.
• 2515: Zachmu/schemas2 merge
• 2513: Added workflows for checking integrators
  This adds a new workflow that runs a subset of tests in Dolt and DoltgreSQL to check for any major integration errors. The workflows do not fail if errors are encountered. Instead, they'll create a comment stating which projects had failures. If no failures were found, then no comment is made.
• 2498: New interfaces for resolving table names for databases with schemas
  This is a proof of concept to get schema resolution working quickly, and I'm not super happy with the separation of concerns. A better solution would implement table name resolution in the Catalog directly, rather than in the integrator. That effort is significantly hindered by the Catalog being a concrete analyzer implementation with many analyzer-specific details that can't be easily substituted for another implementation. The longer term plan is to perform the extensive refactoring necessary to make the relevant parts of the Catalog swappable, rather than (effectively) having to swap only DatabaseProvider and friends.

Closed Issues

• 7902: filter-branch option to apply query to WORKING and STAGED roots
• 7928: CLI dolt fetch <remote> failed to use the default ref spec
• 7897: Pomelo Entity Framework connector is not able to commit changes
• 7909: [Question] How to init Dolt database programatically?

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.11	2.97	1.4
groupby_scan	13.22	17.32	1.3
index_join	1.34	5.18	3.9
index_join_scan	1.27	2.18	1.7
index_scan	33.72	52.89	1.6
oltp_point_select	0.17	0.5	2.9
oltp_read_only	3.36	8.13	2.4
select_random_points	0.32	0.8	2.5
select_random_ranges	0.38	0.95	2.5
table_scan	34.33	54.83	1.6
types_table_scan	73.13	137.35	1.9
reads_mean_multiplier			2.2

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.21	0.8
oltp_insert	3.75	3.07	0.8
oltp_read_write	8.43	15.0	1.8
oltp_update_index	3.82	3.19	0.8
oltp_update_non_index	3.82	3.13	0.8
oltp_write_only	5.37	6.55	1.2
types_delete_insert	7.7	6.91	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	101.2	25.57	4.1
tpcc_tps_multiplier			0.3

Overall Mean Multiple	1.17

1.39.1

Merged PRs

dolt

• 7901: Zachmu/schemas2 merge
• 7899: Properly add database collation change when using -a option in dolt commit
  This PR fixes a case where we don't properly handle database collation changes with the -a option in dolt commit.
  fixes #7897
• 7888: [sort] index build streams sorted edits
  Use sorting to skip more steps building a prolly map. Shaves maybe 20-25% off of external index rebuilds.
  This also fixes a bug where we were incorrectly using only the prefix descriptor to sort secondary index keys.

go-mysql-server

• 2512: Spooling shortcut for one/zero return schemas
  Nodes that return zero or one row don't need a beefy channel/wait group setup to execute. They just need to grab the first row and close the iterator. There are several nodes that incorrectly reported their schemas previously, which I've updated to be more accurate. There are some nodes that optionally return rows, which I've simplified to return an empty schema that can be differentiated from the nil schema. We could make the distinction more explicit, also.
  bump with perf here: #7894

vitess

• 348: Allowing caching plugin to be specified in string quotes
  The CREATE USER ... IDENTIFIED WITH syntax (MySQL ref) allows the caching plugin to be specified in string quotes, but our parser only supported identifier quotes.
  This came up as part of binlog replication testing – MySQL was sending a CREATE USER statement from the primary to a Dolt replica, but Dolt wasn't able to parse the statement because of the use of string quotes around the caching plugin name.
• 347: Added InjectedStatement
  This is the same as InjectedExpr, except for statements instead of expressions.

Closed Issues

• 7891: filter-branch destroys working and staged roots
• 7897: Pomelo Entity Framework connector is not able to commit changes
• 7890: Pomelo Entity Framework connector is not able to recreate database.

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	2.97	1.4
groupby_scan	13.22	17.32	1.3
index_join	1.34	5.28	3.9
index_join_scan	1.27	2.22	1.7
index_scan	34.33	53.85	1.6
oltp_point_select	0.17	0.51	3.0
oltp_read_only	3.36	8.28	2.5
select_random_points	0.33	0.81	2.5
select_random_ranges	0.39	0.97	2.5
table_scan	34.33	55.82	1.6
types_table_scan	74.46	137.35	1.8
reads_mean_multiplier			2.2

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.21	0.8
oltp_insert	3.75	3.07	0.8
oltp_read_write	8.43	15.27	1.8
oltp_update_index	3.82	3.25	0.9
oltp_update_non_index	3.82	3.19	0.8
oltp_write_only	5.37	6.79	1.3
types_delete_insert	7.7	7.04	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	101.76	24.53	4.4
tpcc_tps_multiplier			4.4

Overall Mean Multiple	2.53

1.39.0

Merged PRs

dolt

• 7895: prevent filter-branch when there are local changes
  This PR changes filter-branch to detect any local changes so working/staged changes aren't lost.
  A future PR should include changes to have the working set just be applied over the result of dolt filter-branch.
  partially addresses: #7891

go-mysql-server

• 2512: Spooling shortcut for one/zero return schemas
  Nodes that return zero or one row don't need a beefy channel/wait group setup to execute. They just need to grab the first row and close the iterator. There are several nodes that incorrectly reported their schemas previously, which I've updated to be more accurate. There are some nodes that optionally return rows, which I've simplified to return an empty schema that can be differentiated from the nil schema. We could make the distinction more explicit, also.
  bump with perf here: #7894
• 2511: Adding mapping to error code 1049 for ErrDatabaseNotFound errors
  When a database doesn't exist, MySQL returns error code 1049. This change adds a mapping to error code 1049 for ErrDatabaseNotFound errors, and updates our handler so that ComInitDB messages will map errors to MySQL error codes.
  This is needed because tooling (e.g. Pomelo EntityFramework MySQL library) can rely on this error code in application logic.
  Related to #7890

Closed Issues

• 7890: Pomelo Entity Framework connector is not able to recreate database.

1.38.3

Merged PRs

dolt

• 7884: [tree] return blob builders to pool after use
  I added a builder pool and never returned the objects, this adds the Put().
• 7882: Bug fix: no-op dolt_pull() was leaving working set dirty
  Customer-reported bug. Two dolt_pull() operations on two branches in the same session when local branches are already up to date, with @@autocommit off, leave the session unable to commit because two branch heads are considered dirty. See new bats test for details on reproducing.
  The issue is that DoltSession.SetWorkingSet() marks that branch head dirty until the transaction is committed. Most merge code paths used by pull involve performing a dolt_commit(), which has the side effect of zeroing out the current transaction, meaning the next statement would get a new transaction and fresh working sets loaded from disk, avoiding the dirty state problem. Only the code path where the branch head is already up to date is affected by this bug. All the merge library code that actually needs to call DoltSession.SetWorkingSet() (only necessary before a dolt_commit happens, or in the case of a squash where changes should remain in the working set) already does so, making the additional call in dolt_pull.go redundant and leading to this buggy behavior in the no-change case.
  There are probably still related bugs for session state management during pull and merge operations, but I want to keep this fix narrow to address the customer issue while I build up more robust (non-bats) tests for pull.
• 7878: Move sql patch statement generation APIs to the sqlfmt package
  We have a few different APIs scattered around for generating SQL patch statements. I needed to make some functions from dolt_patch_table_function.go public to generate DDL statements for binlog support, so I moved them into the sqlfmt package and cleaned up some package import cycles along the way.
• 7872: Various test utils and small fixes
  As part of the work for binlog source support (on fulghum/binlog_prototype branch), these are various smaller changes to tidy up docs, packaging, small bug fixes, and add new test utils that I've pulled out into this PR to review separately.
  Notable changes:
  • Adds the third version component to the go version in our go.mod. Two component versions indicate a development version, not a release version and cause an error about not being able to download a toolchain.
  • Allows Dolt binlog replicas to accept the SOURCE_AUTO_POSITION config parameter, and errors if a user attempts to disable GTID auto positioning.
  • Adds several new test util functions specific to binlog testing.
• 7870: go/utils/publishrelease: Bump MUSL toolchains used for cutting releases.
  The new toolchain uses MUSL + mimalloc.
  Include the mimalloc license in our released LICENSES notice.
• 7859: Cache table and schema indexes on schema address
  The bulk of ~1ms read and write TPC-C queries benefit from caching table and index schemas, which have a lifecycle between schema migrations/alter statements/new table additions. This is in contrast to how we've typically cached objects using the root value hash, which is great for read-only workflows, but has a much shorter half-life.

go-mysql-server

• 2511: Adding mapping to error code 1049 for ErrDatabaseNotFound errors
  When a database doesn't exist, MySQL returns error code 1049. This change adds a mapping to error code 1049 for ErrDatabaseNotFound errors, and updates our handler so that ComInitDB messages will map errors to MySQL error codes.
  This is needed because tooling (e.g. Pomelo EntityFramework MySQL library) can rely on this error code in application logic.
  Related to #7890
• 2510: Fix race errors with memory tables
  We use this library for running our tests. These are run with the -race flag - and we are seeing some errors related to concurrency and updating of the tables map.
  I've added a sync.Mutex to all the places where this map is updated - our tests are now passing :)
• 2504: Added InjectedStatement as an AST node
  This is the same as InjectedExpr, except for statements instead of expressions.
• 2502: Use Uint32 for SEQ_IN_INDEX in 'SHOW INDEXES' queries.
  This is seemingly the correct type for this field.
  MySQL Connector/NET expects this for servers >8.0.1: https://github.com/mysql/mysql-connector-net/blob/8.4.0/MySQL.Data/src/SchemaProvider.cs#L298-L300
  Fixes dolthub/go-mysql-server#2501

vitess

• 347: Added InjectedStatement
  This is the same as InjectedExpr, except for statements instead of expressions.
• 346: support DATE, TIME, and TIMESTAMP literal parsing
  The SQL standard has special syntax for parsing date, time, and timestring literals.
  https://dev.mysql.com/doc/refman/8.0/en/date-and-time-literals.html
  This PR adds support for that.
  Code was mostly taken from vitessio.
  The types are still left as string types, as type conversion later on handles it just fine.
• 345: parse type aliases in cast
  add support for statements like:
  • select cast(<str> as character)
  • select cast(<str> as double precision)
  • select cast(<str> as read)

Closed Issues

• 2501: Problems with MySQL Connector/NET (Mysql.Data) and go-mysql-server
• 2503: "ON UPDATE CURRENT_TIMESTAMP" not come into effect

1.38.2

Merged PRs

dolt

• 7880: Migrate dolt remote to SQL
  Fixes: #7622
• 7879: avoid NewEmptyIndex when table does not exist
  It would be better to have NewEmptyIndex not write to chunkstore, but I'm not sure if it's possible right now.
  Workaround is to just avoid calling it altogether in this particular case.

Closed Issues

• 7622: Migrate dolt remote to SQL

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	3.02	1.5
groupby_scan	13.22	17.32	1.3
index_join	1.34	5.18	3.9
index_join_scan	1.27	2.18	1.7
index_scan	35.59	53.85	1.5
oltp_point_select	0.17	0.51	3.0
oltp_read_only	3.36	8.28	2.5
select_random_points	0.33	0.8	2.4
select_random_ranges	0.39	0.95	2.4
table_scan	35.59	55.82	1.6
types_table_scan	75.82	137.35	1.8
reads_mean_multiplier			2.1

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.67	0.8
oltp_insert	3.75	3.25	0.9
oltp_read_write	8.43	15.83	1.9
oltp_update_index	3.82	3.49	0.9
oltp_update_non_index	3.82	3.43	0.9
oltp_write_only	5.37	7.56	1.4
types_delete_insert	7.7	7.56	1.0
writes_mean_multiplier			1.1

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	101.66	22.89	4.4
tpcc_tps_multiplier			4.4

Overall Mean Multiple	2.53

1.38.1

Merged PRs

dolt

• 7877: keep sql.Schema for conflicts table schema
  Previously, we would create a new schema of NomStringKind for every column. Now, we just reuse the underlying sql.Schema.
  #7874
• 7876: Improve error messages for CLI commands when a sql-server is running
  Related to #7873
  Resolves #7875
• 7866: update release process for new config refactor
• 7864: move config
• 7862: Bug fix: sql-server should initialize persisted global vars
  The local config store (.dolt/config.json) can store persisted global variable values, but when --data-dir is used when starting a sql-server, the local configuration doesn't get loaded properly.
• 7860: Bug fix: load local config when using --data-dir
  When using the --data-dir flag to work on a Dolt directory outside of the current working directory, the local configuration in the Dolt directory wasn't getting correctly loaded. This change evaluates the --data-dir parameter earlier, so that the first time we load the Dolt environment, we can pass the data directory and get the local configuration loaded correctly.
• 7858: [nbs] safer peek root hash record
• 7848: Added additional function to RootValue
  This just adds a function to the RootValue for special merge logic, which is used by Doltgres.
• 7846: [dsess] session trigger cache

go-mysql-server

• 2499: fix LIKE NULL edge case
  This PR fixes an edge case where SELECT <str> LIKE NULL should return NULL instead of false.
• 2497: trim whitespace when converting strings to numbers
  fixes #7854
• 2495: fix panic in VALUES constructor
  When the number of rows in a ... VALUES ROW(...), ROW(...) statement were not equal, we would throw a panic.
  This PR also unskips some tests that are now fixed.
  Companion PR: #6849
  fixes: #6849
• 2494: Replace count star also matches single column pk
• 2493: Implement status variables for Slow_queries, Max_used_connections, Com_select, and Connections
  Adds support for four new status variables:
  • Slow_queries
  • Max_used_connections
  • Com_select
  • Connections
    Note that Connections currently only reports the successful connection attempts, but MySQL includes all connection attempts in that status variable. To capture the failed attempts, we'll need to expose that information from the Vitess layer.
    Also removes a mutex that was covering the whole scope over all status variables. Now that each individual status variable has a value that uses an atomic instance, we don't need to synchronize at a larger scope.
    Related to #7646
• 2492: skip source values analyze when it only contains simple types

vitess

• 345: parse type aliases in cast
  add support for statements like:
  • select cast(<str> as character)
  • select cast(<str> as double precision)
  • select cast(<str> as read)
• 344: make row optional in VALUES constructor and insert statement
  This PR adds additional syntax support for VALUE constructor.
  fixes #6849
  fixes #7853
• 338: Add a schema qualifier to table names

Closed Issues

• 7873: Running sql-server from an empty state make inconsistent repository
• 7874: Failed to write conflicts table
• 7875: Confusing error messages when using Dolt CLI from within a running Dolt sql-server directory
• 7854: trim whitespace when casting from string
• 7853: make ROW keyword optional in VALUES statement
• 6849: support INSERT INTO ... (VALUES ROW(...)) statement
• 7845: Make sure deleting a branch behaves similarly to drop database

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	3.02	1.5
groupby_scan	13.22	17.63	1.3
index_join	1.34	5.18	3.9
index_join_scan	1.27	2.22	1.7
index_scan	34.33	54.83	1.6
oltp_point_select	0.17	0.51	3.0
oltp_read_only	3.36	8.43	2.5
select_random_points	0.32	0.8	2.5
select_random_ranges	0.39	0.95	2.4
table_scan	34.33	55.82	1.6
types_table_scan	73.13	137.35	1.9
reads_mean_multiplier			2.2

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.67	0.8
oltp_insert	3.75	3.25	0.9
oltp_read_write	8.28	15.83	1.9
oltp_update_index	3.82	3.49	0.9
oltp_update_non_index	3.82	3.43	0.9
oltp_write_only	5.28	7.56	1.4
types_delete_insert	7.56	7.56	1.0
writes_mean_multiplier			1.1

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	101.35	22.45	4.9
tpcc_tps_multiplier			4.9

Overall Mean Multiple	2.73

1.38.0

Merged PRs

This minor release includes a new entry in the dolt_status and dolt_diff system tables for database collation changes, making these tables backwards incompatible for some select statements. Changes to a dolt database collation will show up as table changes with the name __DATABASE__<db>. Additionally, tables starting with this prefix are not allowed.

dolt

• 7823: handle database charset/collation changes
  This PR makes dolt aware of database collation changes.
  We treat database collation changes similarly to a collation change to a table.
  To properly show a dolt diff we need to add support for show create database as of ..., which would require changes to vitess and gms. For now, we just show the new create statement.
  Addtionally, we should add support to resolve database collation merge conflicts.
  Affected functions are:
  • dolt add
  • dolt commit
  • dolt status
  • dolt diff
  • dolt merge
    Addresses: #7815
• 7819: use parser interface in engine
• 7803: Avoid escaping HTML characters when displaying them to the user.
  This fixes an issue where if a JSON document in the storage layer contains escaped characters, those escape sequences could end up being displayed to the user via the dolt sql -r json command.
• 7764: Bump golang.org/x/net from 0.17.0 to 0.23.0 in /go
  Bumps golang.org/x/net from 0.17.0 to 0.23.0.

go-mysql-server

• 2492: skip source values analyze when it only contains simple types
• 2491: ValidateInsertColumns avoids allocating hash map
• 2490: Avoid escaping HTML when Marshalling JSON
  Due to a misconfiguration, HTML characters were being escaped when marshaling JSON. This is unnecessary, and since we now potentially display marshalled JSON to the user, we shouldn't be doing this.
• 2488: System Variables: Add log_bin and change the default of performance_schema
  The log_bin system variable controls whether a MySQL server logs to the binary log or not.
  The performance_schema system variable was previously defaulted to 1, to match MySQL's default, but this can cause tools (e.g. Datadog) to believe that the performance_schema system tables are available, and then error out when trying to query them. Since we don't provide a performance_schema database, the new default for the performance_schema system variable is 0.
• 2487: Expand literals in comparisons when safe
• 2486: add parser interface in engine
  This PR creates sql.Parser interface. This interface is defined in the engine and it should be used rather than using mysql parser directly.
  Added GlobalParser variable to expose Doltgres parser for parsing view definition for now. It can also be used in places that needs doltgres-specific syntax parsing.

Closed Issues

• 7812: Entity Framework updating working set when there are no changes
• 7815: dolt workflows for manipulating the collation of a DB needed
• 6161: Optimizer statistics v1

1.37.0

The previous (now deleted) release 1.36.1 had a start up time issue for databases > 10GB. We patched it with this one. That release was only up for an hour or so, so it is unlikely anyone got it. Thus, we moved this to 1.37.0 to warn people, just in case.

This minor release includes an internal interface change to the chunk journal index. The first startup process for a database with the old index format will perform a rewrite. This rewrite is a one-time penalty that in testing is <5% of the time it would take to reimport the database.

Merged PRs

dolt

• 7833: Bug fix: Apply replication settings for newly cloned databases
  Dolt SQL servers using remote-based replication will pull new databases if the @@dolt_replication_remote_url_template system variable is configured, but those new databases weren't getting configured to continue pulling updates from the remote.
  This change registers the newly cloned databases as ReadReplicaDatabase instances, so that they will poll their remote and pull new commits. It also adds some additional logging to help debug issues with remote-based replication.
• 7829: Changed RootValue into an interface
  Companion:
  • dolthub/doltgresql#232
    This changes the RootValue into an interface. Every function that seems unique to Dolt's RootValue has been changed into a function variable, with the variable being overwritten from Doltgres to point to a different function.
• 7799: Archive Serialization and Deserialization
  This PR doesn't direclty change any Dolt behavior. It just lays the groundwork for archive creation and reading. Currently, no file is materialized by this code as unit tests exercise it with ByteSinks.
• 7780: Reformat journal index
  Change the way we write journal index lookups. Each write appends a lookup to a bufio.Writer that lazily writes to disk. And after some increment we flush a CRC/root value record for consistency checking the index during bootstrap. This avoids big stalls for flushing a batch of index records. We also only write an addr16 now, because that's what we load into the default chunk address map.
  Databases with the older format will pay a one-time startup penalty to rewrite the journal index. In testing this appears to be 5-10% of the import time for the database.
• 7836: Journal index offset 8bytes
  On >10GB datasets, offsets overflow uint32. Bug from previous PR #7780
• 7834: minver refactor to be used by doltgres
• 7821: Prevent panic when dropping columns in schema merge
  Fixes #7762
  In certain cases, performing a schema merge when the merged schema had fewer columns than the base schema would cause a panic.
  We actually had a test for this, but the test was disabled because a limitation in how the test harness generated column tags was causing incorrect detection of merge conflicts.
  To re-enable these tests, this PR slightly relaxes the logic for merge conflicts wrt column tags. This is safe to do because column tags shouldn't influence the result of merges outside of helping to identify renamed columns, so long as the merge behaves the same in both directions.

Closed Issues

• 7762: Panic during schema merge

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	2.97	1.4
groupby_scan	13.22	17.63	1.3
index_join	1.37	5.18	3.8
index_join_scan	1.27	2.22	1.7
index_scan	34.33	53.85	1.6
oltp_point_select	0.17	0.51	3.0
oltp_read_only	3.36	8.43	2.5
select_random_points	0.33	0.8	2.4
select_random_ranges	0.39	0.95	2.4
table_scan	34.33	54.83	1.6
types_table_scan	74.46	134.9	1.8
reads_mean_multiplier			2.1

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	7.98	6.91	0.9
oltp_insert	3.75	3.43	0.9
oltp_read_write	8.43	16.12	1.9
oltp_update_index	3.82	3.55	0.9
oltp_update_non_index	3.82	3.43	0.9
oltp_write_only	5.37	7.84	1.5
types_delete_insert	7.7	7.56	1.0
writes_mean_multiplier			1.1

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	102.12	22.29	4.6
tpcc_tps_multiplier			4.6

Overall Mean Multiple	2.60