You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
11302: /go/libraries/doltcore/sqle/{dsess,enginetest}: add regression test and lazy load db
Creating databases and then querying them across multiple sessions intermittently fails with could not resolve initial root for database /.
A DoltTransaction snapshots the noms root of every known database when it begins. If one session creates a database after another session's transaction has started, the second session can resolve that database from the shared provider, but its transaction has no start-point for it, so TransactionRoot errors. The only code that added late-appearing databases to a transaction was the read-replica clone path; nothing handled the concurrent-create case.
This PR makes it so that in DoltSession.addDB, if a database isn't in the transaction's snapshot, lazily register it (tx.AddDb) with its current root on first reference — same behavior the clone path already relies on. Also normalized AddDb's map key to the base name to match NewDoltTransaction/GetInitialRoot.
11298: fsck should accept stash objects
fixes: #11292
11296: go: jwtauth: When using auth plugin authentication_dolt_jwt, validate the Subject claim against the expected claims.
A bug in JWT token authentication meant that an incoming token's Subject claim was not validated against the set of expected claims listed in the User record.
The end result is that a given, valid JWT token was not scoped to authenticating a single user account. Any valid JWT token bearing the correct signature, audience and issuer could be used to connect as any user account which was configured to accept that JWKS source, issuer and audenience field.
The fungible token behavior is too big of a footgun to leave in place as the default. To restore fungible token behavior, leave the sub= claim off the expected claims in the User Identity field.
This is a backward incompatible change. Existing workflows expecting fungible tokens to work may need to change token issuance or User record Identity fields to upgrade.
11286: New engine init interface
This lets integrators register engine initialization logic before connections are accepted.
11285: Correctly convert foreign key lookups across encoding type boundaries
Tests are in Doltgres, should be a no-op for Dolt
Companion PR: dolthub/doltgresql#2914
11283: go: sqle: Cleanup comments and function names in DatabaseProvider around creatingDatabases.
Restructure the regression test to be a go-sql-server-driver test and to run a self-clone.
11281: go: sqle: Fix concurrent-clone deadlock between two remotesapi sql-servers DoltDatabaseProvider.CloneDatabaseFromRemote held the provider RWMutex write lock across the entire clone, including the network fetch of all chunks from the remote. When a server serves a --remotesapi-port, that fetch hits a peer's remotesapi whose handler needs a read lock on this same provider (via SessionDatabase). Two sql-servers that CALL DOLT_CLONE from each other's remotesapi at the same time therefore deadlock: each holds its own provider write lock while blocked on the peer, and neither peer can take the read lock needed to answer. DOLT_PULL is immune because it operates on an already-registered database and does not hold the write lock across its fetch. This is the lock the existing // TODO: This holds the database provider lock across the entire duration of the clone ... was warning about.
The fix narrows the lock scope so the network fetch does not run under the provider write lock:
Reserve the database name under a brief lock acquisition (a new creatingDatabases set), release the lock for the fetch, then re-acquire it only to register the finished database (registerNewDatabase still requires the lock).
The reservation keeps concurrent CREATE DATABASE / dolt_clone / undrop of the same name mutually exclusive, so those races behave as before even though the lock is dropped for the fetch.
A shared databaseNameUnavailableLocked helper performs the availability checks for all creation paths and preserves the interrupted-create in-progress marker behaviour (ErrIncompleteDatabaseDir / dbfactory.MarkDatabaseInProgress). The in-memory reservation and the on-disk marker are complementary: the marker guards crash recovery across processes, while creatingDatabases serialises concurrent in-process operations while the lock is dropped.
Names in creatingDatabases and deletingDatabases are normalised with formatDbMapKeyName (Dolt database names are case-insensitive), matching how p.databases is keyed. Unlike deletingDatabases, creatingDatabases deliberately does not gate database enumeration: an in-progress clone is simply invisible until it registers, so enumeration / remotesapi reads never block on it.
Tests:
integration-tests/bats/sql-server-remotesrv.bats: a regression test that starts two --remotesapi-port servers cloning from each other concurrently. It hangs (clones time out) on the buggy build and passes on the fixed build; it also fails if a clone errors for any non-deadlock reason.
go/libraries/doltcore/sqle/database_provider_test.go: TestCreatingDatabaseReservation covering the reservation semantics — conflict, case-insensitivity across create/clone/undrop, release, and that a reservation does not block AllDatabases.
Verified with go test -race ./libraries/doltcore/sqle/ and the new bats test; gofmt / goimports clean.
Fixes #11280
11269: Fix spurious unique key error when staging workspace rows
Staging a delete and an insert that reuse the same unique key via dolt_workspace_* failed with a spurious duplicate unique key error, because rows were applied in workspace order and checked per row.
Updater now applies all deletes first and defers the puts to statement completion
Fix #11195
11267: go: grpcreresolve: Add a load balancer implementation which requests ResolveNow for endpoints when we see Unavailable or DeadlineExceeded.
When running Dolt within a k8s cluster with a service mesh like Istio, the L4 connection to the first-hop load balancer can stay open even when the endpoint associated with that name moves on. gRPC's default pick_first load balancer will not reresolve the endpoints associated with the name unless the L4 connection is lost, which causes persistent failures instead of quick recovery.
Here we implement a new load balancer strategy which just wraps pick_first. It listens for Done on the RPCs, and it calls resolver.ResolveNow if it sees Unavailable or DeadlineExceeded. The DNS resolver in gRPC already debounces resolve requests, so this does not spam reresolves too quickly in the case of true upstream unavailability.
11266: Bug fix:adaptive encoding inserts to a unique index
Uses the ValueStore from the ExtendedTupleComparator instance when set, so that out-of-band data can be correctly loaded.
Related: dolthub/doltgresql#2886
11247: Fix slow clone/fetch from git remotes
Reading a git-backed table file went chunk by chunk, and because git cat-file cannot serve a byte range, each read re-streamed the whole object from the start. Fetching a large table spawned thousands of git processes and stalled.
Spool a git-backed table file to a local temp file on open and serve its chunk reads locally
Look up the sizes of a chunked object's parts in a single git cat-file --batch-check call
Detect an early reader close by end of stdout, so a partial read is torn down the same way on every platform
Fix #11236
11246: Skip incomplete database directories during discovery
Dolt can leave a half-written directory on disk when creating or cloning a database, affecting INFORMATION_SCHEMA and similar queries to error. Dolt now places a marker file in the directory before creation starts and durably removes it once the database is complete.
Dolt storage without a repo state file is now skipped with a warning at load time
Write .dolt_safe_to_ignore marker file when creation a database
Skip marked files or incomplete directories with a warning
CREATE DATABASE or clone over a leftover directory now returns an error asking for removal
Fix #11206
11052: Support covering secondary indexes on DOLT_DIFF() table function.
DOLT_DIFF provides a way to efficiently compare the changes to a table at two commits. If the user only cares about changes to rows within a range of key values, DOLT_DIFF exposes an index that wraps the primary index on the underlying table.
This PR adds additional indexes to DOLT_DIFF that similarly wrap the underlying table's secondary indexes. This allows a user to efficiently compare changes to rows within a range of secondary key values.
This only works if the secondary index is covering, since otherwise the index can't be used to detect modified rows if the only modified columns are outside the index.
go-mysql-server
3619: sql: Fix ALTER USER to allow the Identity field of the User record to be updated. CREATE USER ... IDENTITY WITH <plugin> AS '<identity>' correctly parsed and persisted the Identity field. ALTER USER correctly parsed the Identity field but failed to persist the changes. The end result is that GMS had a bug where attempt to alter the identity field on an existing user seemed to succeed but was not reflected in the data going forward.
Fix the bug so that updates to Identity are reflected and persisted going forward.
3615: Bump golang.org/x/crypto from 0.46.0 to 0.52.0
Bumps golang.org/x/crypto from 0.46.0 to 0.52.0.
Commits
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/dolthub/go-mysql-server/network/alerts).
3614: Report Sum/Avg's true float64 type; add window-func override hook
Various tweaks to go-mysql-server's unary aggregate function support to enable Doltgres window function support:
fixed agg_gen template to be in-sync with generated code after manual changes were applied recently.
changed Avg() and Sum()'s definition to always return float64 and removed analyzer code in aggregates.go that override return types for Avg() and Sum() to float64.
added a window function override hook (IsWindowFunc).
3613: Apply IndexSearchable optimization to sql.Equality expressions
3612: Bug fix: allow running analysis on indexed join queries with bindvars
To support Doltgres, the analyzer in GMS needs to support running on queries that contain bindvars. An assumption in indexed join costing was causing a panic for Doltgres when a bindvar appeared in the join condition on an indexed column.
Doltgres needs to run the analyzer when preparing a statement because of the Postgres wire protocol, which requires more detailed schema information to be returned before the prepared statement is executed.
3610: pushdown inequalities to merge joins
Before, only equality filters would be pushed down into the IndexedTableAccess for MergeJoins.
This PR makes it so all static comparisons are pushed down.
TODO:
The Filter over these IndexedTableAccess nodes are useless and we should drop them
The coster needs to properly account for the pushed down filters.
We should be able to push down more than just one expression over a column
We should be able to push more than just the prefix into the index
3609: server: Add client-disconnect watching to more platforms and more query types.
Previously GMS had rudimentary logic on Linux to scan the socket state list and watch for a TCP connection associated with a running query to transition to a non-connected state. If that happened, GMS would cancel a Context associated with the query so that it could terminate in a timely manner --- the client was no longer waiting for the response.
The old implementation had a few downsides:
The scan itself periodically scanned a large table in /proc for every in-flight query. This was relatively expensive and somewhat wasteful.
It only worked on Linux and only for TCP listeners.
It only worked for some queries. In particular, it was never enabled for OkResult queries, or queries that were knowingly going to return only one result. This was an optimization because even setting up the connection logic scanning was somewhat expensive.
This PR changes GMS to take a different approach. We added (*mysql.Conn).WaitForClientActivity(context.Context) to the vitess layer, and now the server handler uses that to get a quick notification the the client connection received readable activity (probably an EOF) when it was not expecting.
We still have to be careful to not needless add overhead to very fast-running queries. The approach we take is to have a single goroutine periodically scanning the connection state. Connections themselves register when they start a query and when they end it. The scanning goroutine will start the disconnect watcher on the connection only after its query has been running for a bit of time.
3608: Rewrite pushFilters to use single top-down node traversal
Split off from #3591 pushFilters was also previously written with two nested transform.NodeWithCtx (a bottom-up traversal). The outer traversal was done to identify filter nodes. The inner traversal "pushed down" filter expressions to the table level - this was not a true "pushdown" because it was a bottom-up traversal. An additional InspectUp traversal was done for each filter node to collect all the filter expressions in the node tree -- this was unnecessary since any child filter nodes would've already had their filter expressions handled and this resulted in chains of identical filter nodes that later needed to be condensed. An additional Inspect traversal was done per filter node to find projection expressions -- this only needed to be done once with the root node instead of per filter node. pushFilters also checks if the node tree is unresolved at the beginning, adding another traversal. This meant a node tree with n filters would be traversed 2+3n times. #3603 reduced the number of traversals done to find projection expressions to 1, instead of once per filter node. This PR rewrites the pushFilters rule to only use a single node traversal that collects filters as the traversal moves down the node tree. It also removes the check for unresolved nodes. These changes reduce the number of node tree traversals to a constant 2.
This also makes it easier to later combine the pushFilters and moveJoinConditionsToFilter rule (which is done in #3591).
Query plans have changed because repeated filter expressions (previously from traversing the entire child tree to find all filters) have been de-duped and filter expressions are ANDed slightly differently.
3600: Allow filter pushdown in subqueries when analyzing insert sources
fixes #11232
3591: Combine pushFilters rule with moveJoinConditionsToFilter
Fixes #10899
This PR rewrites the pushFilters rule by combining it with the moveJoinConditionsToFilter. Background
Previously, if a join condition only referenced one side of a join, moveJoinConditionsToFilter would wrap that side with a filter. The idea was that the filter would later get pushed down to the table during pushFilters, but this caused an issue with nested joins where join/filter conditions that referred to multiple tables that were all part of the same child join would get orphaned in the middle of the join tree and then dropped during reorderJoin (#9868). This was partially fixed with #3231, where filters could get pushed into the join condition. However, this didn't work for all join types and also introduced the problem in #10899, where the filter expression pushed down into the join condition was being treated as an edge during reorderJoin, leading to incorrect join plans. Why query plan tests changed
Filters are no longer automatically getting pushed into join conditions, which affects the join type and join reordering. There are cases where this could still be allowed, but for now, it's safer to not since doing so was causing incorrect join reorders.
More CrossJoins. Previously, joins that had all their join conditions pushed down into a filter were getting TRUE assigned as their join condition. This was preventing InnerJoins from being turned into CrossJoins because they had a non-nil join condition, even though nil join conditions are treated as TRUE. Additional Work
There are cases where parent filter expressions or join conditions can be pushed down into a join condition. This could be achieved by extending filtersByTable to use a FastIntSet to store expressions that reference more than one TableId and for pushdownFiltersAboveTables to propagate up a FastIntSet of child TableIds.
There are also join types, such as Left Outer and Anti joins, where a parent filter expression should not be evaluated as a join condition; however, it may be performant to evaluate the filter expression as part of the join iterator to reduce the need for an additional loop through a filter iterator.
The replaceCrossJoins may not be necessary and could probably combined into pushFilters as well
Closed Issues
11195: Spurious UNIQUE key failure when staging rows in workspace (fixed by staging individually)
11292: dolt fsck misclassifies stash list as a commit
11280: Deadlock when two sql-servers CALL DOLT_CLONE from each other's --remotesapi-port concurrently
10899: LEFT JOIN + IS NULL filter returns wrong rows when combined with another INNER JOIN (1.86.2)
11206: sql-server: one broken database dir in the data dir fails every INFORMATION_SCHEMA query server-wide
11232: Dolt Predicate Pushdown Bug: INSERT…SELECT Loses Outer WHERE Pushdown
