Repository navigation
Merged PRs
dolt
- 12497: Avoid panicking when a tree is more than 16 levels deep
- 12495: add DumboRootValueFileID for dumbo databases
- 12490: /go/store/blobstore: disable auto-maintenance on fetch
Fixes #12488
git fetchinto the git remote cache ends by starting git's automatic maintenance, which detaches and can repack and delete packs after the fetch has returned. A later read could then get a warning on stderr from agit cat-file -sthat still succeeded, and Dolt parsed the warning as the size.
This PR updatesRunner.Runto return only stdout on success, so stderr warnings can't corrupt parsed output. Failures still carry both streams. It also changesFetchRefto disable auto-maintenance (maintenance.auto=false,gc.auto=0). Dolt's own periodicgit gcand the end of the process already compacts the cache. - 12489: Add DOLT_DISABLE_VERSION_CHECKS to disable release checks via the environment
Fix #12479 - 12470: Plan fewer foreign keys for UPDATE and DELETE
Bumps go-mysql-server to plan fewer foreign keys for UPDATE and DELETE.
Companion PRs: dolthub/go-mysql-server#4151, dolthub/doltgresql#3841 - 12469: Fix merges that combine a column drop and row deletion
When one branch drops a column and the other deletes a row, compare the surviving row with the ancestor using the surviving branch’s schema. Use stored column positions for type lookups and name-based column matches so virtual columns cannot shift those lookups.
Adds regression coverage for merge and preview in both directions, including a table with a virtual column.
Fixes #12274 - 12468: Make ForeignKeyCollection copy-on-write and share each root's collection
Fixes super-linear UPDATE planning cost as foreign keys referencing a table increase.
Companion PR: dolthub/doltgresql#3840 - 12467: Respect partial index predicates when merging secondary indexes
Fixes merges and index rebuilds adding rows that don't satisfy a partial index predicate to the index.
Companion PR: dolthub/doltgresql#3839 - 12466: go: sql: Visit a pending writer's root's children for GC, not the root, which is stored inline.
A table's primary index root is stored inline in the table's chunk, so asking session-aware GC to keep it by address failed the GC withdangling references requested during GC. This was the recentTestConcurrentGCfailure on macOS CI.TestGCWithPendingEditsreproduces it deterministically.
Also, disables auto GC in TestConcurrentGC's KillConnections subtests, since it forces session-aware GC. - 12463: Answer view lookups for names that aren't views from the session cache instead of rescanning dolt_schemas, and key the view cache by schema.
Answer view lookups for names that aren't views from the session cache instead of rescanning dolt_schemas, and key the view cache by schema. - 12443: Fix repeated recursive CTE rows with a schema override
A schema override can expose a join that reuses stale recursive CTE rows. Pin the GMS fix so recursive joins return the same rows with the override enabled or cleared.
Fixes #12309
Depends on: GMS #4148 - 12433: sqle, dsess: key database and revision maps with typed names
Adds doltdb.QualifiedDbName and the case-normalized DbKey, RevisionKey and QualifiedDbKey, and uses them for the provider, session, transaction and cache maps instead of ad hoc split-and-lowercase strings. This fixes the revision database cache, which never hit for mixed-case revisions because it stored lower-cased keys but looked them up with the revision's original case. - 12427: store/types: prevent lost GC admission cancellation wakeups
Fix a dropped cancellation signal in GC, because the broadcast could land before the listener registration. Guard the broadcast withgcMuas well. - 12424: Reduce the per-foreign-key cost of commits and transaction merges
Bundle of performance fixes to dramatically speed up hot path database operations when there are a large number of foreign keys. Foreign keys were being iterated on every merge regardless of whether they had changed, which happens on every concurrent transaction. Lookups by name were quadratic time. And we were copying a collection N times during a diff operation.
Also fixes unrelated bugs discovered during testing:- Fixes
ALTER TABLE ... DROP CONSTRAINTsometimes dropping another table's foreign key with the same name in Doltgres. - Fixes merges dropping foreign keys whose parent table doesn't exist yet.
Companion PRs:
dolthub/go-mysql-server#4140
dolthub/doltgresql#3820
- Fixes
- 12418: Add option and context parameters to store and database creation
Lets Dolt Enterprise implement encryption at rest with fewer edits to Dolt's files, so its merges of Dolt conflict less. Replaces #12350, #12351 and #12352. It includes #12429, merged into this branch, which replaced #12419.What changes
- nbs:
- Every store constructor takes a trailing
...StoreOption. AStoreOptionfills the store's options, kept in a newoptsfield, as the store is created. store_options.gohas a no-op method for each place instore.gowhere a store's options take effect, andstore.gocalls them:- manifest checks in
rebaseandupdateManifest, and a check before writes inaddChunk; - a check of added files in
updateManifestAddFiles, plus the open of added files; - the journaling store's format and table persister;
- the GC check and GC copier;
- the table file split offsets in
WriteTableFileandnewTableFile, and whetheraddTableFilesToManifestcan read added chunks to check their refs.
Enterprise replacesstore_options.gowith its own version. - The persister literals in
store.goare keyed. - Archive
getRecordRangesnames its context. - remotestorage:
NewDoltChunkStoreFromPathtakes a trailing...ChunkStoreOption. - env/actions:
EnvForCloneOftakes the source database and configure steps that run on the new environment before its database is created. A configure step's error fails the clone, and the clone cleans up as for any other failure.EnvForClonecalls it. - sqle: internally, the provider's
getRemoteDBtakes the dialer andcreateDatabasetakes load params. - CLI:
dolt cloneanddolt read-tablesuseEnvForCloneOf.dolt transfer'sresolveDatabaseChunkStorereturns theDBCacheit serves.
Dolt defines noStoreOptionorChunkStoreOption, every store hook does nothing, andgetRemoteDBandcreateDatabaseget only today's values. Existing exported signatures are kept, so other callers compile unchanged.
Changes for Dolt users
dolt_clonegives the new database the provider's load params before creating it, asCREATE DATABASEalready does, rather than after.
Size
176 changed lines in 8 existing non-test files (40 of them instore.gofor the hooks), and 80 lines in two new files.
Dolt Enterprise built on this passes its encryption tests. Weighting each hunk by how often Dolt changed its file in the past year, its risk of a merge conflict falls by 54%.Testing
- New tests in
env/actionscheck that configure steps run before the database is created, and that a failing step leaves no directory behind. - Existing tests pass, with keyed literals in one test.
🤖 Generated with Claude Code
go-mysql-server
- 4149: Fix two event executor timing races
- Fixes the event executor going idle without loading an event change that was made through the EventScheduler but not yet committed.
- Fixes recurring events being skipped when the executor woke just over a polling period after they were due; the executor now wakes when the next event is due.
These made the doltevents.batstests "out-of-band event changes are detected" and "recurring event with ENDS defined" flaky.
- 4148: Prevent joins from caching recursive CTE working rows
Joins can repeat or omit rows when they cache inputs that read a recursive CTE’s working table. Skip hash join build plans and derived-table result caches that depend on those changing rows.
Part of Dolt issue #12309. - 4146: sql: Remove ContextAwareSystemVariableRegistry and AssignValuesWithContext.
SystemVariableRegistry.AssignValues replaces both; no callers remain. - 4145: sql/analyzer: Build only the triggers of the tables a statement writes
Build only triggers that match the tables being mutated, instead of building all triggers in the schema. - 4144: Prevent columns in top-most node from getting pruned
Fixes #12366
#3250 replaced SQAs that referenced entire tables with a TableAlias node, removing redundant SQA andProjectnodes; it added guards to preventTableAliasandResolvedTablenodes from getting pruned. However, it did not consider when thoseTableAliasnodes were wrapped insideLimit,Distinct, orOffsetnodes.
This PR makes sure that the columns in the schema of the top-most node are not pruned by adding them to the parent columns set. - 4141: Fix StoredProcParam referencing stale values
Stored Procedure parameters were incorrectly getting preserved acrossCALLs.
This PR makes it so new parameters properly override any existing values in the session. - 4140: Don't use partial indexes for lookup joins or sort replacement unless the query implies the predicate
Fixes lookup joins and ORDER BY index scans that returned too few rows by using a partial index the query does not imply.
Tests are in the linked Doltgres PR, since partial indexes can't be declared in the MySQL dialect, except for the right semi lookup join case, which those queries don't reach.
Companion PRs: #12424 dolthub/doltgresql#3820 - 4139: Fix JSON string datetime conversion
Fix the NULL results when casting JSON date strings to DATETIME or passing them to DAY(). Apply JSON-specific suffix validation, fractional-second truncation, and timezone-offset handling while preserving ordinary SQL-string conversion.
Builds on the customer's contribution in #4111, retaining Samuel Cormier-Iijima's commit authorship. Coverage uses SQL engine scripts, including stored JSON and writes into DATE/DATETIME/TIMESTAMP columns.
Fixes #12341. - 4137: sql: keep TIMESTAMPDIFF unit a keyword in stored text
Fix #12340 - 4134: Report NOT_FIXED_DEC decimals for float result columns
PHP apps readFLOATcolumns from Dolt rounded to whole numbers: a stored4.2comes back as4. Laravel uses server-side prepared statements, and over the binary protocol PHP's mysqlnd roundsFLOATvalues to the column's reporteddecimals. Dolt reports0for every float and double result column. MySQL reports31(NOT_FIXED_DEC) when there's no fixed scale.Since Dolt doesn't keep a scale on float types, this reportsrating FLOAT CAST(4.25 AS FLOAT) MySQL 8.0 (decimals) 4.2 (2) 4.25 (31) Dolt (decimals) 4 (0) 4 (0)31for all float32/float64 result columns, as MySQL does for unscaled floats.TestSchemaToFieldsis updated, and with the patch the PDO prepared-statement values above match MySQL.
Created with Claude Opus 5.5 in Claude Code. - 4133: Add ST_DISTANCE_SPHERE
- 4124: Use every ORDER BY key for RANGE CURRENT ROW peer groups
RANGE BETWEEN CURRENT ROW AND CURRENT ROWandRANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWINGboth go throughrangeFramerBase, which only takes the first ORDER BY expression. So withORDER BY k, id, rows that sharekbut have differentidwere treated as peers. The repro in the issue returnsc = 2for the first two rows; MySQL returns 1.
UNBOUNDED PRECEDING AND CURRENT ROWalready moved toPeerGroupFramerin 8dbc053 ("Fix implicit window peer frames"), and that framer compares every ORDER BY key. I did the same for the two remaining CURRENT ROW-only range frames:CURRENT ROW AND CURRENT ROWnow usesNewPeerGroupFramerdirectly.CURRENT ROW AND UNBOUNDED FOLLOWINGuses a newNewPeerGroupToUnboundedFollowingFramer, which is the peer-group framer with the end pinned to the partition end. I added apeerEndfield so it still knows where the current peer group ends even thoughframeEndis the partition end.
The change is inframer_gen.go, andwindow_framer.og.gowas regenerated withgo generate. Frames with numeric offsets still require a single ORDER BY key, same as before.
While testing I also saw thatORDER BY k DESC, g DESC RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWINGreturned the whole partition for every row before this change. The new tests cover that case too.
Testing:- added a script test to
enginetest/queries/window_functions_queries.gocovering the issue's query, ties on the full key, DESC ordering, and a partitioned case go test ./enginetest/ -run 'TestWindow|TestNamedWindows'go test ./sql/expression/function/aggregation/
The new assertions fail on main and pass with this change.
Fixes #11423
- 4120: sql/expression: range-check unsigned DIV results
A signed valueDIVan unsigned value previously went through the decimal path, which returned the truncated quotient as an int64 even thoughIntDivreports aBIGINT UNSIGNEDtype, soSELECT -1 DIV CAST(1 AS UNSIGNED)returned 18446744073709551615 instead of MySQL's error 1690, andCAST(18446744073709551615 AS UNSIGNED) DIV 1errored. When the result type is unsigned, that path now checks the truncated quotient the way MySQL does: a negative result returnssql.ErrIntegerOutOfRange(the error from #3823 for #11411), a truncated zero like-1 DIV CAST(2 AS UNSIGNED)still returns 0, and results up to max uint64 come back as uint64. Results past max uint64 keep the existingErrIntDivDataOutOfRange, so thediv 0.1error queries inqueries.goare unchanged. The script test after the #11411 case innumeric_queries.gocovers the regression.
Mixed signed and unsigned operands now use native integer division when their evaluated Go values match their SQL types. The synthetic 16-level mixedDIVbenchmark dropped from 256 to 32 allocations per evaluation. The full Go suite passed.
Fixes #12311 - 4055: Create ValueRowWriter interface for safer, faster plan evaluation
Dolt PR: #12190
Traditionally, we evaluate a plan by constructing a tree of iterators. This has turned out to be a common source of bugs:- In order to resolve references in expressions, we must compute the offset within a given iterator where the resolved column is expected to appear. The logic for this appears in the
assignExecIndexesand is extremely difficult to get correct. Getting it wrong results in incorrect plans. - The is no obvious way in the nested iterator model to make columns from other scopes visible, which has resulted in hacks like the
PrependNodeiterator type that gets injected into the plan tree. This is also a source of bugs, especially when combined withassignExecIndexes.
To avoid these issues, this PR introduces an alternate interface for resolving queries calledValueRowWriter. It has the following interface:
The ValueRow passed intotype ValueRowWriter interface { RowIter // WriteNextValueRow produces the next row by writing its values into the provided ValueRow. // It will return io.EOF if it's the last ValueRow. // After retrieving the last ValueRow, Close will be automatically called. WriteNextValueRow(ctx *Context, rows ValueRow) error // IsValueRowWriter checks whether this implementor and all its children support ValueRowWriter. IsValueRowWriter(ctx *Context) bool // Columns is the set of ColumnIds represented by this iterator's output schema Columns() ColSet Closer }WriteNextValueRowcontains an element for every columnId in the whole query. All values, both inner and outer scopes, both reads and writes, are accessed by using columnIds as indexes. This eliminates the need for calculating indexes for field accesses. This ValueRow can also be reused for every iteration, eliminating allocations.
The server Handler checks if the top level iterator is a ValueRowWriter, and if it is, copies only the selected columns into an output ValueRow.
Currently this exists alongside ValueRowIter, since not every node that implements ValueRowIter is currently capable of implementing ValueRowWriter. But it should be possible to migrate all remaining uses to the new interface.
This PR more than doubles the number ofQueryTeststhat use ValueRow from 50 to 110. Queries matching these tests should see a speedup because they no longer need to deserialize-then-box all database values.
Follow-up tasks:- The implementation of ColSet.Max
- The name
ValueRowWriteronly makes sense in the context of theValueRowtype, which is an alternative to theRowtype. None of these names are very clear. We may want to rename these types to make their purpose more obvious. - We should remove the remaining implementations of ValueRowIter to have them use ValueRowWriter instead. To do this, ProjectIter and prollyRowIter need a way to know the ColumnIds of their output schema. This must be able to handle cases where the order of the columns changes in the projection (example:
SELECT a, b FROM (SELECT b, a ...) - Subqueries always allocate a new ValueRow for their child. This is currently necessary because the child might be a ProjectIter with projections, and because in the case of views or common table expressions with multiple references in the same query, each of those references might re-use the same ColumnIds. We can do more work to detect when this is actually necessary, and make it necessary in fewer situations.
- When we must allocate a separate ValueRow for a subquery, we may be able to allocate a smaller one than currently. Currently, if the child query has a max ColumnId of N, we allocate a ValueRow with N columns. Instead, we could also look at the minimum ColumnId and only create a backing array for the range min...max.
- In order to resolve references in expressions, we must compute the offset within a given iterator where the resolved column is expected to appear. The logic for this appears in the
Closed Issues
- 12488: git remote cache: detached auto-maintenance
repack -draces later reads ("packfile ... index unavailable", then astrconv.ParseInterror) - 12479: Request: Add the
DOLT_DISABLE_VERSION_CHECKSenvironment variable - 10365: dolt_tags system table missing index on tag_name - causes O(n) lookup
- 11205: Pre-2.1 ALTERed TEXT columns: mixed-row 'invalid hash length: 19' panics on 2.1.4; schema-encoding-drift check reports clean; rows not repairable via SQL
- 11244: Support refreshable custom gRPC metadata for remotes
- 12236: Reading a view inside a transaction commits the transaction; a later ROLLBACK does not undo earlier writes
- 12274: dolt_merge fails when one branch dropped a column and the other deleted a row
- 12241: COM_BINLOG_DUMP_GTID ignores the executed GTID set when BINLOG_THROUGH_GTID is unset
- 12309: Recursive CTE returns duplicate rows when @@dolt_override_schema is set to HEAD
- 12340: Generated TIMESTAMPDIFF quotes its unit and leaves an unusable stored expression
- 11014: ROW_COUNT function returns 0 on INSERT statement
- 12311: signed
DIVunsigned wraps around instead of raising an out of range error - 12366: Query error - Nested SQL query returns no rows when it should - adding a single table alias gives expected