Skip to content

2.7.4

Latest

Choose a tag to compare

@github-actions github-actions released this 16 Nov 18:15

Merged PRs

dolt

  • 12497: Avoid panicking when a tree is more than 16 levels deep
  • 12495: add DumboRootValueFileID for dumbo databases
  • 12490: /go/store/blobstore: disable auto-maintenance on fetch
    Fixes #12488
    git fetch into the git remote cache ends by starting git's automatic maintenance, which detaches and can repack and delete packs after the fetch has returned. A later read could then get a warning on stderr from a git cat-file -s that still succeeded, and Dolt parsed the warning as the size.
    This PR updates Runner.Run to return only stdout on success, so stderr warnings can't corrupt parsed output. Failures still carry both streams. It also changes FetchRef to disable auto-maintenance (maintenance.auto=false, gc.auto=0). Dolt's own periodic git gc and the end of the process already compacts the cache.
  • 12489: Add DOLT_DISABLE_VERSION_CHECKS to disable release checks via the environment
    Fix #12479
  • 12470: Plan fewer foreign keys for UPDATE and DELETE
    Bumps go-mysql-server to plan fewer foreign keys for UPDATE and DELETE.
    Companion PRs: dolthub/go-mysql-server#4151, dolthub/doltgresql#3841
  • 12469: Fix merges that combine a column drop and row deletion
    When one branch drops a column and the other deletes a row, compare the surviving row with the ancestor using the surviving branch’s schema. Use stored column positions for type lookups and name-based column matches so virtual columns cannot shift those lookups.
    Adds regression coverage for merge and preview in both directions, including a table with a virtual column.
    Fixes #12274
  • 12468: Make ForeignKeyCollection copy-on-write and share each root's collection
    Fixes super-linear UPDATE planning cost as foreign keys referencing a table increase.
    Companion PR: dolthub/doltgresql#3840
  • 12467: Respect partial index predicates when merging secondary indexes
    Fixes merges and index rebuilds adding rows that don't satisfy a partial index predicate to the index.
    Companion PR: dolthub/doltgresql#3839
  • 12466: go: sql: Visit a pending writer's root's children for GC, not the root, which is stored inline.
    A table's primary index root is stored inline in the table's chunk, so asking session-aware GC to keep it by address failed the GC with dangling references requested during GC. This was the recent TestConcurrentGC failure on macOS CI. TestGCWithPendingEdits reproduces it deterministically.
    Also, disables auto GC in TestConcurrentGC's KillConnections subtests, since it forces session-aware GC.
  • 12463: Answer view lookups for names that aren't views from the session cache instead of rescanning dolt_schemas, and key the view cache by schema.
    Answer view lookups for names that aren't views from the session cache instead of rescanning dolt_schemas, and key the view cache by schema.
  • 12443: Fix repeated recursive CTE rows with a schema override
    A schema override can expose a join that reuses stale recursive CTE rows. Pin the GMS fix so recursive joins return the same rows with the override enabled or cleared.
    Fixes #12309
    Depends on: GMS #4148
  • 12433: sqle, dsess: key database and revision maps with typed names
    Adds doltdb.QualifiedDbName and the case-normalized DbKey, RevisionKey and QualifiedDbKey, and uses them for the provider, session, transaction and cache maps instead of ad hoc split-and-lowercase strings. This fixes the revision database cache, which never hit for mixed-case revisions because it stored lower-cased keys but looked them up with the revision's original case.
  • 12427: store/types: prevent lost GC admission cancellation wakeups
    Fix a dropped cancellation signal in GC, because the broadcast could land before the listener registration. Guard the broadcast with gcMu as well.
  • 12424: Reduce the per-foreign-key cost of commits and transaction merges
    Bundle of performance fixes to dramatically speed up hot path database operations when there are a large number of foreign keys. Foreign keys were being iterated on every merge regardless of whether they had changed, which happens on every concurrent transaction. Lookups by name were quadratic time. And we were copying a collection N times during a diff operation.
    Also fixes unrelated bugs discovered during testing:
    • Fixes ALTER TABLE ... DROP CONSTRAINT sometimes dropping another table's foreign key with the same name in Doltgres.
    • Fixes merges dropping foreign keys whose parent table doesn't exist yet.
      Companion PRs:
      dolthub/go-mysql-server#4140
      dolthub/doltgresql#3820
  • 12418: Add option and context parameters to store and database creation
    Lets Dolt Enterprise implement encryption at rest with fewer edits to Dolt's files, so its merges of Dolt conflict less. Replaces #12350, #12351 and #12352. It includes #12429, merged into this branch, which replaced #12419.

    What changes

    • nbs:
    • Every store constructor takes a trailing ...StoreOption. A StoreOption fills the store's options, kept in a new opts field, as the store is created.
    • store_options.go has a no-op method for each place in store.go where a store's options take effect, and store.go calls them:
    • manifest checks in rebase and updateManifest, and a check before writes in addChunk;
    • a check of added files in updateManifestAddFiles, plus the open of added files;
    • the journaling store's format and table persister;
    • the GC check and GC copier;
    • the table file split offsets in WriteTableFile and newTableFile, and whether addTableFilesToManifest can read added chunks to check their refs.
      Enterprise replaces store_options.go with its own version.
    • The persister literals in store.go are keyed.
    • Archive getRecordRanges names its context.
    • remotestorage: NewDoltChunkStoreFromPath takes a trailing ...ChunkStoreOption.
    • env/actions: EnvForCloneOf takes the source database and configure steps that run on the new environment before its database is created. A configure step's error fails the clone, and the clone cleans up as for any other failure. EnvForClone calls it.
    • sqle: internally, the provider's getRemoteDB takes the dialer and createDatabase takes load params.
    • CLI:
    • dolt clone and dolt read-tables use EnvForCloneOf.
    • dolt transfer's resolveDatabaseChunkStore returns the DBCache it serves.
      Dolt defines no StoreOption or ChunkStoreOption, every store hook does nothing, and getRemoteDB and createDatabase get only today's values. Existing exported signatures are kept, so other callers compile unchanged.

    Changes for Dolt users

    • dolt_clone gives the new database the provider's load params before creating it, as CREATE DATABASE already does, rather than after.

    Size

    176 changed lines in 8 existing non-test files (40 of them in store.go for the hooks), and 80 lines in two new files.
    Dolt Enterprise built on this passes its encryption tests. Weighting each hunk by how often Dolt changed its file in the past year, its risk of a merge conflict falls by 54%.

    Testing

    • New tests in env/actions check that configure steps run before the database is created, and that a failing step leaves no directory behind.
    • Existing tests pass, with keyed literals in one test.
      🤖 Generated with Claude Code

go-mysql-server

  • 4149: Fix two event executor timing races
    • Fixes the event executor going idle without loading an event change that was made through the EventScheduler but not yet committed.
    • Fixes recurring events being skipped when the executor woke just over a polling period after they were due; the executor now wakes when the next event is due.
      These made the dolt events.bats tests "out-of-band event changes are detected" and "recurring event with ENDS defined" flaky.
  • 4148: Prevent joins from caching recursive CTE working rows
    Joins can repeat or omit rows when they cache inputs that read a recursive CTE’s working table. Skip hash join build plans and derived-table result caches that depend on those changing rows.
    Part of Dolt issue #12309.
  • 4146: sql: Remove ContextAwareSystemVariableRegistry and AssignValuesWithContext.
    SystemVariableRegistry.AssignValues replaces both; no callers remain.
  • 4145: sql/analyzer: Build only the triggers of the tables a statement writes
    Build only triggers that match the tables being mutated, instead of building all triggers in the schema.
  • 4144: Prevent columns in top-most node from getting pruned
    Fixes #12366
    #3250 replaced SQAs that referenced entire tables with a TableAlias node, removing redundant SQA and Project nodes; it added guards to prevent TableAlias and ResolvedTable nodes from getting pruned. However, it did not consider when those TableAlias nodes were wrapped inside Limit, Distinct, or Offset nodes.
    This PR makes sure that the columns in the schema of the top-most node are not pruned by adding them to the parent columns set.
  • 4141: Fix StoredProcParam referencing stale values
    Stored Procedure parameters were incorrectly getting preserved across CALLs.
    This PR makes it so new parameters properly override any existing values in the session.
  • 4140: Don't use partial indexes for lookup joins or sort replacement unless the query implies the predicate
    Fixes lookup joins and ORDER BY index scans that returned too few rows by using a partial index the query does not imply.
    Tests are in the linked Doltgres PR, since partial indexes can't be declared in the MySQL dialect, except for the right semi lookup join case, which those queries don't reach.
    Companion PRs: #12424 dolthub/doltgresql#3820
  • 4139: Fix JSON string datetime conversion
    Fix the NULL results when casting JSON date strings to DATETIME or passing them to DAY(). Apply JSON-specific suffix validation, fractional-second truncation, and timezone-offset handling while preserving ordinary SQL-string conversion.
    Builds on the customer's contribution in #4111, retaining Samuel Cormier-Iijima's commit authorship. Coverage uses SQL engine scripts, including stored JSON and writes into DATE/DATETIME/TIMESTAMP columns.
    Fixes #12341.
  • 4137: sql: keep TIMESTAMPDIFF unit a keyword in stored text
    Fix #12340
  • 4134: Report NOT_FIXED_DEC decimals for float result columns
    PHP apps read FLOAT columns from Dolt rounded to whole numbers: a stored 4.2 comes back as 4. Laravel uses server-side prepared statements, and over the binary protocol PHP's mysqlnd rounds FLOAT values to the column's reported decimals. Dolt reports 0 for every float and double result column. MySQL reports 31 (NOT_FIXED_DEC) when there's no fixed scale.
    rating FLOAT   CAST(4.25 AS FLOAT)
    MySQL 8.0  (decimals)  4.2   (2)      4.25  (31)
    Dolt       (decimals)  4     (0)      4     (0)
    
    Since Dolt doesn't keep a scale on float types, this reports 31 for all float32/float64 result columns, as MySQL does for unscaled floats. TestSchemaToFields is updated, and with the patch the PDO prepared-statement values above match MySQL.
    Created with Claude Opus 5.5 in Claude Code.
  • 4133: Add ST_DISTANCE_SPHERE
  • 4124: Use every ORDER BY key for RANGE CURRENT ROW peer groups
    RANGE BETWEEN CURRENT ROW AND CURRENT ROW and RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING both go through rangeFramerBase, which only takes the first ORDER BY expression. So with ORDER BY k, id, rows that share k but have different id were treated as peers. The repro in the issue returns c = 2 for the first two rows; MySQL returns 1.
    UNBOUNDED PRECEDING AND CURRENT ROW already moved to PeerGroupFramer in 8dbc053 ("Fix implicit window peer frames"), and that framer compares every ORDER BY key. I did the same for the two remaining CURRENT ROW-only range frames:
    • CURRENT ROW AND CURRENT ROW now uses NewPeerGroupFramer directly.
    • CURRENT ROW AND UNBOUNDED FOLLOWING uses a new NewPeerGroupToUnboundedFollowingFramer, which is the peer-group framer with the end pinned to the partition end. I added a peerEnd field so it still knows where the current peer group ends even though frameEnd is the partition end.
      The change is in framer_gen.go, and window_framer.og.go was regenerated with go generate. Frames with numeric offsets still require a single ORDER BY key, same as before.
      While testing I also saw that ORDER BY k DESC, g DESC RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING returned the whole partition for every row before this change. The new tests cover that case too.
      Testing:
    • added a script test to enginetest/queries/window_functions_queries.go covering the issue's query, ties on the full key, DESC ordering, and a partitioned case
    • go test ./enginetest/ -run 'TestWindow|TestNamedWindows'
    • go test ./sql/expression/function/aggregation/
      The new assertions fail on main and pass with this change.
      Fixes #11423
  • 4120: sql/expression: range-check unsigned DIV results
    A signed value DIV an unsigned value previously went through the decimal path, which returned the truncated quotient as an int64 even though IntDiv reports a BIGINT UNSIGNED type, so SELECT -1 DIV CAST(1 AS UNSIGNED) returned 18446744073709551615 instead of MySQL's error 1690, and CAST(18446744073709551615 AS UNSIGNED) DIV 1 errored. When the result type is unsigned, that path now checks the truncated quotient the way MySQL does: a negative result returns sql.ErrIntegerOutOfRange (the error from #3823 for #11411), a truncated zero like -1 DIV CAST(2 AS UNSIGNED) still returns 0, and results up to max uint64 come back as uint64. Results past max uint64 keep the existing ErrIntDivDataOutOfRange, so the div 0.1 error queries in queries.go are unchanged. The script test after the #11411 case in numeric_queries.go covers the regression.
    Mixed signed and unsigned operands now use native integer division when their evaluated Go values match their SQL types. The synthetic 16-level mixed DIV benchmark dropped from 256 to 32 allocations per evaluation. The full Go suite passed.
    Fixes #12311
  • 4055: Create ValueRowWriter interface for safer, faster plan evaluation
    Dolt PR: #12190
    Traditionally, we evaluate a plan by constructing a tree of iterators. This has turned out to be a common source of bugs:
    • In order to resolve references in expressions, we must compute the offset within a given iterator where the resolved column is expected to appear. The logic for this appears in the assignExecIndexes and is extremely difficult to get correct. Getting it wrong results in incorrect plans.
    • The is no obvious way in the nested iterator model to make columns from other scopes visible, which has resulted in hacks like the PrependNode iterator type that gets injected into the plan tree. This is also a source of bugs, especially when combined with assignExecIndexes.
      To avoid these issues, this PR introduces an alternate interface for resolving queries called ValueRowWriter. It has the following interface:
    type ValueRowWriter interface {
    RowIter
    // WriteNextValueRow produces the next row by writing its values into the provided ValueRow.
    // It will return io.EOF if it's the last ValueRow.
    // After retrieving the last ValueRow, Close will be automatically called.
    WriteNextValueRow(ctx *Context, rows ValueRow) error
    // IsValueRowWriter checks whether this implementor and all its children support ValueRowWriter.
    IsValueRowWriter(ctx *Context) bool
    // Columns is the set of ColumnIds represented by this iterator's output schema
    Columns() ColSet
    Closer
    }
    
    The ValueRow passed into WriteNextValueRow contains an element for every columnId in the whole query. All values, both inner and outer scopes, both reads and writes, are accessed by using columnIds as indexes. This eliminates the need for calculating indexes for field accesses. This ValueRow can also be reused for every iteration, eliminating allocations.
    The server Handler checks if the top level iterator is a ValueRowWriter, and if it is, copies only the selected columns into an output ValueRow.
    Currently this exists alongside ValueRowIter, since not every node that implements ValueRowIter is currently capable of implementing ValueRowWriter. But it should be possible to migrate all remaining uses to the new interface.
    This PR more than doubles the number of QueryTests that use ValueRow from 50 to 110. Queries matching these tests should see a speedup because they no longer need to deserialize-then-box all database values.
    Follow-up tasks:
    • The implementation of ColSet.Max
    • The name ValueRowWriter only makes sense in the context of the ValueRow type, which is an alternative to the Row type. None of these names are very clear. We may want to rename these types to make their purpose more obvious.
    • We should remove the remaining implementations of ValueRowIter to have them use ValueRowWriter instead. To do this, ProjectIter and prollyRowIter need a way to know the ColumnIds of their output schema. This must be able to handle cases where the order of the columns changes in the projection (example: SELECT a, b FROM (SELECT b, a ...)
    • Subqueries always allocate a new ValueRow for their child. This is currently necessary because the child might be a ProjectIter with projections, and because in the case of views or common table expressions with multiple references in the same query, each of those references might re-use the same ColumnIds. We can do more work to detect when this is actually necessary, and make it necessary in fewer situations.
    • When we must allocate a separate ValueRow for a subquery, we may be able to allocate a smaller one than currently. Currently, if the child query has a max ColumnId of N, we allocate a ValueRow with N columns. Instead, we could also look at the minimum ColumnId and only create a backing array for the range min...max.

Closed Issues

  • 12488: git remote cache: detached auto-maintenance repack -d races later reads ("packfile ... index unavailable", then a strconv.ParseInt error)
  • 12479: Request: Add the DOLT_DISABLE_VERSION_CHECKS environment variable
  • 10365: dolt_tags system table missing index on tag_name - causes O(n) lookup
  • 11205: Pre-2.1 ALTERed TEXT columns: mixed-row 'invalid hash length: 19' panics on 2.1.4; schema-encoding-drift check reports clean; rows not repairable via SQL
  • 11244: Support refreshable custom gRPC metadata for remotes
  • 12236: Reading a view inside a transaction commits the transaction; a later ROLLBACK does not undo earlier writes
  • 12274: dolt_merge fails when one branch dropped a column and the other deleted a row
  • 12241: COM_BINLOG_DUMP_GTID ignores the executed GTID set when BINLOG_THROUGH_GTID is unset
  • 12309: Recursive CTE returns duplicate rows when @@dolt_override_schema is set to HEAD
  • 12340: Generated TIMESTAMPDIFF quotes its unit and leaves an unusable stored expression
  • 11014: ROW_COUNT function returns 0 on INSERT statement
  • 12311: signed DIV unsigned wraps around instead of raising an out of range error
  • 12366: Query error - Nested SQL query returns no rows when it should - adding a single table alias gives expected