fix: Skip mark_databases_as_deleted for single-database connectors#27333
Open
fix: Skip mark_databases_as_deleted for single-database connectors#27333
Conversation
…d fix Azure SQL query Azure SQL Database (single database tier) does not support cross-database references like `master.sys.databases`, causing the metadata agent post-process step to fail even when `ingestAllDatabases` is false. Two fixes: 1. Changed `AZURE_SQL_GET_DATABASES` query to use `sys.databases` instead of `master.sys.databases` for Azure SQL Database compatibility. 2. Added a guard in `mark_databases_as_deleted` that skips the post-process step when the source is a `MultiDBSource` with a configured single database (`get_configured_database() is not None`). This prevents unnecessary queries when only one database is being ingested. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
github-actions
bot
added
Ingestion
safe to test
When `ingestAllDatabases` is false, the `mark_databases_as_deleted` post-process step would still query all databases from the source to mark deletions. On Azure SQL Database (single database tier), this fails because `master.sys.databases` is not supported. Added a guard in `mark_databases_as_deleted` that skips the post-process step when the source is a `MultiDBSource` with a configured single database (`get_configured_database() is not None`). When only ingesting one database, querying all databases to mark deletions is unnecessary. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Code Review ✅ ApprovedSingle-database connectors now skip the mark_databases_as_deleted operation, optimizing resource usage for connectors that manage only one database. No issues found. OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prevents database-deletion post-processing from running in “single database” ingestion mode for multi-database connectors (notably Azure SQL Database), and fixes the Azure SQL database listing query to be compatible with Azure SQL Database’s single-database tier.
Changes:
- Update Azure SQL
AZURE_SQL_GET_DATABASESto querysys.databases(nomaster.prefix). - Add a guard in
mark_databases_as_deletedto skip deletion marking when aMultiDBSourceis configured for single-db ingestion (get_configured_database() != None). - Add Azure SQL topology unit tests and extend Postgres topology tests to cover the new guard behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| ingestion/tests/unit/topology/database/test_postgres.py | Updates existing deletion-marking tests to explicitly simulate “ingest all DBs” and adds coverage for the new single-db skip guard. |
| ingestion/tests/unit/topology/database/test_azuresql.py | Adds topology unit tests validating Azure SQL’s configured-db behavior, deletion skip guard, and the updated databases query string. |
| ingestion/src/metadata/ingestion/source/database/database_service.py | Implements the guard to skip mark_databases_as_deleted when running a MultiDBSource in single-db mode. |
| ingestion/src/metadata/ingestion/source/database/azuresql/queries.py | Fixes Azure SQL database listing query to use sys.databases without master. prefix. |
Contributor
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
|
Contributor
🟡 Playwright Results — all passed (29 flaky)✅ 3596 passed · ❌ 0 failed · 🟡 29 flaky · ⏭️ 207 skipped
🟡 29 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AZURE_SQL_GET_DATABASESquery to usesys.databasesinstead ofmaster.sys.databases— themaster.prefix is not supported on Azure SQL Database (single database tier), causingmark_databases_as_deletedpost-process to crash.mark_databases_as_deletedthat skips the entire post-process step when the source is aMultiDBSourcewithingestAllDatabases=false. When only ingesting a single configured database, querying all databases to mark deletions is unnecessary and can fail on platforms like Azure SQL Database.Test plan
mark_databases_as_deletedskips whenget_configured_database()returns a value (single-db mode)mark_databases_as_deletedproceeds whenget_configured_database()returnsNone(all-db mode)None, so deletion always runs)🤖 Generated with Claude Code