Skip to content

fix(perf): open tables without waiting behind background schema introspection#1483

Merged
datlechin merged 14 commits into
mainfrom
fix/metadata-connection-contention
May 29, 2026
Merged

fix(perf): open tables without waiting behind background schema introspection#1483
datlechin merged 14 commits into
mainfrom
fix/metadata-connection-contention

Conversation

@datlechin
Copy link
Copy Markdown
Member

Problem

On connections with many tables (300+), opening a table tab showed the executing spinner for 3-4 seconds while the toolbar reported the query itself took ~100ms.

Root cause

Background schema introspection ran on the same physical database connection as user queries. Plugin connections serialize every operation through one queue, so a user SELECT waited behind whatever introspection was in flight. The largest offender is the autocomplete column preload (fetchAllColumns over every table) that fires on connect, database switch, and refresh. The toolbar only measured the SELECT's own time once it finally ran, so the spinner covered the whole wait.

Fix

Route background introspection onto the existing MetadataConnectionPool instead of the main query connection, so user queries never wait behind it.

  • New DatabaseManager.withMetadataDriver(connectionId:workload:) borrows a pooled connection for the connection's active database.
  • The pool now separates bulk introspection (autocomplete preload, ER diagram) from interactive metadata (table open, switchers) onto different pooled connections, so a long bulk scan never blocks a quick lookup.
  • Autocomplete column loading, per-table schema/foreign-key/row-count fetches, structure view, view definitions, ER diagram, AI chat schema context, the database and quick switchers, and the export dialog now fetch metadata through the pool.
  • User-visible queries, data export, row edits, and schema mutations stay on the main connection. The data grid keeps correct primary keys and editability from the first frame.

Tests

Added coverage that the schema provider's column loading uses the injected metadata source. Existing provider tests cover the driver fallback path.

How to verify

Open a table on a connection with 300+ tables. The spinner should clear at about the query time, even right after connecting while autocomplete is still loading. Watch the gap between the tab-select log line and [executeUserQuery].

Based on #1473 (sidebar database tree), which introduced MetadataConnectionPool.

@datlechin datlechin force-pushed the fix/metadata-connection-contention branch from 4ef3399 to 749fa67 Compare May 29, 2026 10:11
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4ef3399098

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread TablePro/Core/Database/DatabaseManager+Metadata.swift
Signed-off-by: Ngô Quốc Đạt <datlechin@gmail.com>
@datlechin datlechin force-pushed the fix/metadata-connection-contention branch from 749fa67 to c0505c0 Compare May 29, 2026 10:24
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0505c0a20

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread TablePro/Core/Services/Query/MetadataConnectionPool.swift
@datlechin datlechin force-pushed the fix/metadata-connection-contention branch 2 times, most recently from faf4792 to fa994a3 Compare May 29, 2026 10:31
@datlechin datlechin force-pushed the fix/metadata-connection-contention branch from fa994a3 to e058031 Compare May 29, 2026 11:06
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e05803197c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread TablePro/Core/Services/Query/MetadataConnectionPool.swift Outdated
@datlechin datlechin force-pushed the fix/metadata-connection-contention branch from e058031 to 3056d9c Compare May 29, 2026 11:39
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3056d9c2fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +66 to +67
try await DatabaseManager.shared.withMetadataDriver(connectionId: connId) { driver in
let cols = try await driver.fetchColumns(table: tableName)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep session-local metadata on the main driver

When the editable table comes from a session-local object (for example a PostgreSQL/MySQL temporary table created on the current connection, or a temp table shadowing a permanent table), this metadata fetch now runs on a separate pooled connection that cannot see the main connection's temp schema. The user query still succeeds on the main driver, but the schema task will either return no metadata or metadata for a different permanent table, so primary keys/editability and phase-2 metadata can be wrong for those results; keep this path on the active driver or fall back when the table is connection-local.

Useful? React with 👍 / 👎.

Base automatically changed from feat/139-sidebar-database-tree to main May 29, 2026 13:29
datlechin added 3 commits May 29, 2026 20:38
# Conflicts:
#	TablePro/Core/Database/DatabaseManager+Health.swift
#	TablePro/Core/Database/DatabaseManager+Sessions.swift
#	TablePro/Core/Services/Query/DatabaseTreeMetadataService.swift
#	TablePro/Core/Services/Query/MetadataConnectionPool.swift
#	TablePro/Views/Main/MainContentCoordinator.swift
#	TablePro/Views/Sidebar/DatabaseTreeView.swift
@datlechin datlechin merged commit e1d88fb into main May 29, 2026
2 checks passed
@datlechin datlechin deleted the fix/metadata-connection-contention branch May 29, 2026 13:53
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46726a6f63

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +380 to +381
count = try await DatabaseManager.shared.withMetadataDriver(connectionId: parent.connectionId, workload: .bulk) { driver in
let result = try await driver.execute(query: sql)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep row counts on the session connection

When an editable table is connection-local (for example a PostgreSQL/MySQL temporary table, or a temp table shadowing a permanent table), this new count path runs SELECT COUNT(*) on a pooled metadata connection instead of the session connection that loaded the grid. That separate connection cannot see the same temp objects/session state, so the row-count refresh can fail or report a different table’s count even though the data grid query itself succeeded; the prior implementation executed this count on the active driver.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant