2.2.4

Merged PRs

dolt

• 11528: branch: fix -d when the upstream does not resolve
  Deleting a branch with dolt branch -d could fail with branch not found when deleted from a remote. Dolt now compares against the local snapshot of the upstream, not the remote repository, and falls back to the current branch when its copy is gone.
  • Add env.UpstreamRef to resolve a branch's upstream local ref from BranchConfig.
  • Add upstreamOrHead to decide when to use the upstream branch or switch to current working branch.
  • Upstream is searched by the configured merge ref in the BranchConfig to support different names.
  • Merged predicate reads the remote-tracking ref instead of the live remote state.
  • RemoteDbProvider parameter dropped because it goes unused.
    Fix #11450
• 11520: Look up table names directly
  Change table name lookups to try directly looking up the table name, instead of always iterating over all table names in the database.
• 11515: Use sql.ErrDuplicateCheckName
  We should use the same error in GMS in Dolt.
  Depends on: dolthub/go-mysql-server#3691
• 11486: Bug fixes for binlog replication with virtual columns
  Adds support for correctly handling virtual columns and functional indexes in binlog replication, both when Dolt is acting as the replication primary and when Dolt is acting as a replica.
  Fixes: #11475
• 11443: Fix dump of tables with a generated column
  Fixes a panic in dolt dump on tables with a generated column which affected sql, csv, and parquet output. The generated INSERT statements also were updated to omit generated columns, addressing STORED generated columns non-restorable dumps.
  • mvdata/engine_table_reader.go: NewSqlEngineReader took its rows from a SELECT statement but its schema from a separate SHOW CREATE TABLE plan that was built but never ran. When a table has a generated column, the planbuilder replaces it with plan.VirtualColumnTable that does not implement sql.PrimaryKeyTable, the interface that plan uses to fill its PrimaryKeySchema. The schema now comes from the same query as the rows to avoid a zero value.
  • NewSqlEngineTableReaderWithEngine became same as the above and now delegates.
  • sqlfmt/row_fmt.go: InsertStatementPrefix and SqlRowAsTupleString both skip generated columns, keeping the column list and the value tuple the same length.
  • SqlRowAsTupleString returns an error on a row and schema width mismatch.
    Fix #11439
• 11430: go: store/nbs: reject a file manifest update that references a missing table file
  A bare local store takes no cross-process lock while a table file is landing, so between the moment a writer opens or renames a table file and the moment it commits a manifest naming it, another process can unlink it. The writer does not notice, since its own file descriptor keeps working, and it publishes a manifest that is broken for everyone else.
  Add a sanity check where we Stat the newly added files under the manifest LOCK. This allows other Dolt processes which want to safely unlink unreferenced files from the directory to do so under the same LOCK. While a write may transiently fail if an about-to-be-referenced file is deleted out from under it, it won't cause database corruption by writing a manifest which references a non-existant table file.
  This PR adds the add-files sanity check. It does not add the machinery to safely remove any files yet.
  Journaled stores are unaffected: ChunkJournal goes through journalManifest, not fileManifest.
• 11406: bug fix for nonlocal tables to work in Doltgres
• 11400: go: remotesrv: Move away from deprecated http2/h2c package. Use SetUnencryptedHTTP2 instead.
  Also adds "h2" as a supported ALPN next protocol to the remotesrv server listener when configuring with TLS.
  Fixes #11386
• 11385: /go/store/{datas,nbs}: cleanup byte sink files on error
  Currently there are some paths where temporary byte sink files may persist on disk despite an error. This PR aims to ensure these are always deleted in such cases.
• 11377: Enable signed.bats with isolated GNUPGHOME
• 11374: bug fixes for schema names in table statistics
  This is necessary to get doltgres to use table statistics.
  Also a bug fix for the declared column type of the dolt_statistics table, to match what GMS actually returns.
• 11368: Bug fix for lookup joins with extended types
  Regression tests included in: dolthub/doltgresql#2988
• 11337: Refactor AutoIncrementTracker into generitc type with interfaces
  The purpose of this PR is to enable Doltgres to create a global state for Sequences that behaves similarly to Dolt's global state for auto increment columns. In order to allow for code reuse, we have to do the following:
  • Create a generic type containing the behavior common to both auto-increment and sequences.
  • Create new interfaces that can be implemented to describe the behavior unique to each feature.
    (Bikeshedding note: I use the term "Relation" to describe a top-level object in the database, and encompasses both tables and root objects. I'm open to alternative names.)
    The new interfaces can be found in go/libraries/doltcore/sqle/globalstate/sequences/state.go
  • SequenceState is an interface for a state machine that can be incremented. Each time it is incremented, it produces a new SQL value. SequenceStates can also be compared to each other to determine which one is "further along" and "merged" to create a single state that is further along in its sequence than any of the input states. The only implementation in Dolt is AutoIncrementState, which is just a uint64. Doltgres sequences will have a more complicated implementation, since they have more parameters that affect how the state is incremented.
  • SequencedRelation is a relation that holds a SequenceState. The only implementation in Dolt is doltdb.Table, which can hold an auto-increment value. Doltgres Sequences will also implement this interface, and possibly Doltgres tables with a SERIAL column.
    The common logic is in the SequenceTracker type, which has an interface in go/libraries/doltcore/sqle/globalstate/sequence_tracker.go and an implementation in go/libraries/doltcore/sqle/dsess/sequence_tracker.go
    AutoIncrementTracker is now a type alias for a specialization of SequenceTracker.
    Finally, while GlobalState previously only held an AutoIncrementTracker, it now contains a map from arbitrary keys to SequenceTrackers. This will allow Doltgres to add an additional SequenceTracker for global tracking of Doltgres sequences.
    There are a couple places in this PR that are TODOs, because they are corner cases that aren't reachable for AutoIncrement but are reachable for Doltgres sequences. They mostly concern how to handle when branches contain incompatible Sequences. I haven't decided exactly how this should be handled, and this PR will not be merged until we know what to do in that case, but it shouldn't affect the review of the rest of the PR.
• 11329: Reject branch names that differ only by case
  Branch names are matched case-insensitively causing branches that only differ by case to resolve to the same data (e.g., main and Main).
  • Refuse to create branch whose names differs from existing one by case on checkout, branch, and other variants

  dolt branch Main
  dolt checkout -b Main
  dolt branch -c main Main
  dolt branch -f Main

  • Allow case-only rename to remove ambiguity of conflicting branch names
    Fix #11270
• 11279: Remove no-op AutoIncrement statement
  A simple cleanup: this line always errors, but the error isn't being checked. It effectively does nothing and isn't needed because:
  • the auto increment state is automatically updated by the AutoIncrement node in the plan tree
    -as long as the auto_increment as long as w.aiSet is true, the correct auto_increment value will be read from the AutoIncrementTracker.

go-mysql-server

• 3693: fix panic for GeometryFromWKT and various GeoSpatial bugs
  We added support for "pure empty geometry forms", which MySQL doesn't support, except for "GEOMETRYCOLLECTION EMPTY".
  Changes:
  • fix panics for pure empty geometry forms
  • fix panics for invalid axis-order options
  • fix ordering for axis-order options
  • expose st_latitude function
  • fix geospatial srid for st_x() and st_y() functions
    Fixes: #11492
• 3691: block certain statements from PREPARE
  This PR blocks CREATE EVENT and CREATE PROCEDURE from getting created.
  Additionally, adds tests for a variety of DDL statements like DELETE, CREATE, and ALTER.
  Fixes:
  #11417
  #11451
• 3690: Check for null before casting LAST_INSERT_ID input
  fixes #11454
• 3689: Refactor window function tests into ScriptTests
  Converts existing window function tests to use ScriptTest and moves those ScriptTests into a new window_function_queries.go file.
• 3687: SECURITY: Document security stance on panics generated by GMS.
  Add some defensive recover() calls at places where GMS itself spawns goroutines.
• 3677: Bump jsonpath and add new test for parsing JSON null.
  Fixes #11394
  Our jsonpath dependency wasn't properly parsing paths when the input object was a JSON null value.
• 3676: Update various window aggregation functions to expect the correct number of children
  Fixes #11428
  Some window functions were expecting the wrong number of child expressions, thus incorrectly throwing an error. However, this error was getting swallowed and going unnoticed during fixExprToScope, which is intentional due to errors being falsely triggered for dual tables or subqueries but does lead to unintentional consequences. As a result, child expressions were not getting properly updated during fixExprToScope, leading to a panic.
  This PR updates WithChildren for those window functions to expect the correct number of children.
• 3674: Error out for undefined window names
  fixes #11426
  ErrUnknownWindowName was defined but never actually used anywhere. This PR updates the error message to better match MySQL and throws the error when a referenced window name has not actually be defined.
• 3672: Pass context through aggregate/window check funcs
• 3671: Handle negative start positions in LOCATE function
  fixes #11393
• 3670: Fix panic when a RANGE window frame offset pushes a SET/ENUM order-by value outside its valid domain
  Also stops swallowing non-EOF framer errors that let it turn into an out-of-bounds index.
  Fixes: #11397
• 3669: Combine assignmentExprsToUpdateExprs and buildOnDupUpdateExprs
  Fixes #11389
  This PR addresses a TODO that I had added to combine assignExprsToUpdateExprs and buildOnDupUpdateExprs, since these two functions did pretty much the same thing with a lot of repeated code. buildOnDupUpdateExprs was missing the step to resolve DEFAULT so it made more sense to combine the two functions that it did to rewrite that step.
  This PR also replaces the panic in DefaultColumn.Eval with an error in an effort to reduce the number of panics in GMS and Dolt (#11299).
• 3667: Clear autocommit transactions on err on all paths
• 3666: Include orderBy in various window functions to return correct default framer
  fixes #11381
  baseWindowFunction was created to avoid duplicating shared code and is intended to be extended to fully eliminate duplicated code for window functions.
  Also updates tests that were asserting incorrect behavior (verified to match MySQL)
  Some new test cases are skipped for Doltgres (see dolthub/doltgresql#3036 and dolthub/doltgresql#3038)
• 3665: Adding a new TimeDeltaExpression interface
  Allows window interval arithmetic to work properly with Doltgres' implementation.
• 3664: Check if SubqueryAlias itself is lateral when prepending rows
  Fixes #11378
• 3663: bug fix: infer a bindvar's column type
  Fix untyped bind-var type inference to use the comparison expression's own declared type instead of digging into its children. This means operators like #>> (jsonb→text) (from Doltgres) don't leak their operand's type onto the parameter.
  Related to: dolthub/doltgresql#3012
• 3662: add case for injected exprs in node batch
• 3661: create schema from inscope columns of the table being parsed as column
• 3660: bug fixes for schema in table statistics
  As well as other bugs that were preventing table statistics from being used in doltgres
• 3659: fallback to gms types for extended type histograms
  benchmarks: dolthub/doltgresql#3001 (comment)
• 3658: When comparing tuples for equality, widen each pair of input values.
  Previously, when comparing tuples, we would cast the right tuple to the type of the left tuple. This is inconsistent with MySQL, which considers each pair of values and converts them both to a widened type.
  This resulted in us returning the incorrect result for the simple query:

  SELECT (1, 1) = (1.1, 1.1);
  

  Dolt would round both of the values on the right to integers prior to comparing them, evaluating to true. The correct behavior is to apply MySQL's type conversion rules to each pair, which in this case would result in both integers being converted to decimals, and the tuples comparing unequal.
• 3656: Bug fix for CASE expressions with ExtendedType instances
  Related to: dolthub/doltgresql#2980
  Doltgres fix: dolthub/doltgresql#2987
• 3651: Correctly do NullUnsafe comparisons with tuples.
  Previously, all comparisons with tuples acted as though the comparison was null-safe. But the correct behavior is more subtle. Essentially, any comparison operation involving NULL should return NULL if substituting different values for the NULL could result in different outputs.
  This PR fixes most of the incorrect comparisons. Some stragglers remain involving WHERE x IN (SELECT ...) subqueries.
  This PR is inspired by dolthub/go-mysql-server#3640 but doesn't use any of the code from that PR.
• 3611: avoid fmt.Sprintf in groupby hash

Closed Issues

• 11475: INSERT INTO a table with functional index enabled panics when binlog replication is active on dolt sql-server
• 11450: dolt branch -d does not find branch that exists when doing dolt branch [-a]
• 11503: ORDER BY + huge LIMIT (≥ 2^62) panics
• 11457: BIN_TO_UUID panics for a NULL swap flag
• 11492: Dolt panics on empty POLYGON WKT
• 11451: Dolt panics on an unchecked AST cast within prepared CREATE PROCEDURE
• 11417: Dolt panics when preparing/executing CREATE EVENT.
• 11394: Dolt panics when JSON_TABLE is evaluated over JSON null.
• 11454: LAST_INSERT_ID(NULL) panics on a nil result
• 11397: Dolt panics for a RANGE BETWEEN CURRENT ROW AND 1 FOLLOWING window over a SET order key.
• 11181: noms chunk journal unrecoverable after unclean shutdown ('journal index is malformed' / 'corrupted journal at offset N') — request repair tooling
• 11439: dump panics with "index out of range" on a table with a generated column
• 11428: Dolt panics for FIRST_VALUE with an empty OVER () spec.
• 11426: Dolt panics when a named window inherits from an undefined window.
• 11386: sql-server: cluster remotesapi TLS listener does not advertise ALPN "h2", breaking gRPC through L7 proxies
• 11270: Dolt allows the creation of branches of the same name with different cases, but dolt checkout does not support this
• 11429: Dolt aggregate numeric coercion over TIME returns zero.
• 11424: Dolt ignores LIMIT inside a correlated EXISTS subquery.
• 11415: Dolt ONLY_FULL_GROUP_BY recognizes functional dependency through a primary key but not through a UNIQUE NOT NULL key.
• 11393: Dolt panics when evaluating LOCATE with a negative start position.
• 11389: Dolt panics when DEFAULT is used in an ON DUPLICATE KEY UPDATE assignment.
• 11381: Aggregate window function with OVER (ORDER BY <col>) has inconsistent behavior
• 11378: Correlated subquery over a column-split-rejoin view raises internal "unable to find field with index" planner error