Fix SQLite migration compatibility and idempotency issues - #1006
Fix SQLite migration compatibility and idempotency issues#1006Bornunique911 wants to merge 4 commits into
Conversation
Summary by CodeRabbit
WalkthroughThree Alembic migrations conditionally add nullable document and embedding columns. They also create artifact ingestion tables with foreign keys and unique constraints, add missing constraints to existing tables, and remove schema objects during downgrade. ChangesMigration persistence changes
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py (1)
11-21: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
sa.inspect(conn)instead ofInspector.from_engine(conn).The unpinned runtime SQLAlchemy dependency can resolve to SQLAlchemy 2.x, where
Inspector.from_engine()is deprecated. Replace the duplicated calls in the referenced migrations with the supported inspection entry point.Proposed change
-from sqlalchemy.engine.reflection import Inspector ... - inspector = Inspector.from_engine(conn) + inspector = sa.inspect(conn)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py` around lines 11 - 21, Replace the deprecated Inspector.from_engine(conn) usage with sa.inspect(conn) in the column_exists helper in migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py (lines 11-21), migrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.py (lines 11-21), and migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py (lines 11-21). Preserve the existing column inspection behavior and remove any now-unused Inspector imports.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py`:
- Around line 39-43: The downgrade in
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py at
lines 39-43 must remove document_metadata from both cre and node using the
existing SQLite batch-table migration pattern, replacing the no-op downgrade.
Apply the same reversible downgrade change in
migrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.py at
lines 31-33 to remove embedding_vec from embeddings, and add downgrade coverage
for both revisions.
In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py`:
- Around line 24-71: Update the migration logic around artifact_ingest_event and
ingest_chunk so existing tables are inspected for
uq_artifact_ingest_event_run_artifact and uq_ingest_chunk_artifact_chunk. When
either constraint is missing, rebuild or otherwise alter the table to add it
before the migration completes; do not silently skip DDL, and fail the migration
if the constraint cannot be repaired.
---
Nitpick comments:
In `@migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py`:
- Around line 11-21: Replace the deprecated Inspector.from_engine(conn) usage
with sa.inspect(conn) in the column_exists helper in
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py (lines
11-21), migrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.py
(lines 11-21), and
migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py (lines
11-21). Preserve the existing column inspection behavior and remove any
now-unused Inspector imports.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0b0c9beb-ffbf-47a6-a28f-116a34aae300
📒 Files selected for processing (3)
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.pymigrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.pymigrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py
…ints - Add downgrades to 055dbd9f8bfe and 967016ee10fa to drop added columns - Replace deprecated Inspector.from_engine(conn) with sa.inspect(conn) - In 9f1a2b3c4d5e, verify that existing tables have the required unique constraints; add them via batch_alter_table if missing - Ensure migration fails if constraints cannot be added Addresses PR review comments OWASP#1006
Before change on the main website opencre.org/node/standard/CWE :
After change locally on 127.0.0.1:5000/node/standard/CWE :
|
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py (1)
94-97: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftTrack table ownership before dropping in
downgrade().
upgrade()may only add the missing unique constraints to existingartifact_ingest_eventoringest_chunktables, whiledowngrade()always runsop.drop_table()for both. If a pre-existing table was adopted instead of created here, downgrade destroys its rows and schema. Track ownership or downgrade only constraint-alterations made by this revision.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py` around lines 94 - 97, Update the migration’s upgrade/downgrade flow to track whether artifact_ingest_event and ingest_chunk were created by this revision versus merely adopted with added constraints. In downgrade(), drop only tables owned and created by this revision; for pre-existing tables, revert only the unique constraints added by this migration and preserve their rows and schema.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py`:
- Around line 38-43: Align downgrade ownership checks with the conditional
upgrades: in
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.py lines
38-43, update downgrade() to drop document_metadata from cre and node only when
each column exists, preserving pre-existing columns; in
migrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.py lines
30-33, apply the same existence check before dropping embedding_vec from
embeddings. Add downgrade coverage for absent columns and populated pre-existing
columns at both sites.
In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py`:
- Around line 57-61: Before the artifact_ingest_event batch rewrite and
create_unique_constraint operation, handle the ingest_chunk foreign key
explicitly: temporarily enable foreign-key enforcement, drop the child
constraint, and recreate it with its existing cascade behavior after the
rewrite. Add a regression test covering pre-existing artifact_ingest_event rows
and verifying the ingest_chunk relationship remains valid.
- Around line 57-61: Update the existing-table migration paths using
op.batch_alter_table for artifact_ingest_event and ingest_chunk so unnamed
UNIQUE constraints are preserved during SQLite table recreation. Copy or replace
each supported unnamed uniqueness rule before adding the named constraint,
reject unsupported legacy schemas, or use recreate="always" with explicit
table_args; ensure no existing uniqueness rule is silently dropped.
- Around line 22-26: Update constraint_exists to validate both the constraint
name and its column_names against the migration’s expected target columns,
returning true only for an exact definition match; if the name exists with
different columns, do not treat it as present so the migration can correct it.
Apply the same validation at the checks around the constraints named
uq_artifact_ingest_event_run_artifact and uq_ingest_chunk_artifact_chunk.
---
Outside diff comments:
In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py`:
- Around line 94-97: Update the migration’s upgrade/downgrade flow to track
whether artifact_ingest_event and ingest_chunk were created by this revision
versus merely adopted with added constraints. In downgrade(), drop only tables
owned and created by this revision; for pre-existing tables, revert only the
unique constraints added by this migration and preserve their rows and schema.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7d32bd9f-c55b-4fa9-8cb2-af49b08dfa50
📒 Files selected for processing (3)
migrations/versions/055dbd9f8bfe_add_document_metadata_to_cre_and_node.pymigrations/versions/967016ee10fa_add_embedding_vec_to_embeddings_for_.pymigrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py
| with op.batch_alter_table("artifact_ingest_event") as batch_op: | ||
| batch_op.create_unique_constraint( | ||
| "uq_artifact_ingest_event_run_artifact", | ||
| ["run_id", "artifact_id"] | ||
| ) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- SQLite foreign-key configuration ---'
rg -n -C 6 --glob '*.py' 'PRAGMA\s+foreign_keys|foreign_keys\s*=' . || true
printf '%s\n' '--- Existing-table migration tests ---'
rg -n -C 10 --glob '*test*' 'artifact_ingest_event|ingest_chunk|batch_alter_table' . || trueRepository: OWASP/OpenCRE
Length of output: 4041
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- Candidate migration file ---'
fd -a '9f1a2b3c4d5e_add_artifact_ingest_persistence.py|migrations/versions.*artifact_ingest' . || true
printf '%s\n' '--- Migration file with line numbers ---'
if [ -f migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py ]; then
cat -n migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py
fi
printf '%s\n' '--- Models and SQLite config references ---'
rg -n -C 5 --glob '*.py|*.cfg|*.ini|*.yaml|*.yml|*.toml|Makefile' 'sqlite|SQLALCHEMY_DATABASE_URL|Alembic|batch_alter_table|foreign key|foreign_keys' . || trueRepository: OWASP/OpenCRE
Length of output: 5168
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- All foreign_keys mentions ---'
rg -n -C 4 'foreign[ _-]?keys|PRAGMA foreign_keys|pragma.*foreign' . --glob '!docs/**' --glob '!venv/**' --glob '!node_modules/**' || true
printf '%s\n' '--- Alembic env configuration ---'
fd -a 'env.py|alembic.ini' migrations .
for f in $(fd 'env.py|alembic.ini' .); do
echo "### $f"
cat -n "$f"
done
printf '%s\n' '--- Alembic batch constraints in migrations only ---'
rg -n -C 3 --glob 'migrations/versions/*.py' 'batch_alter_table|op\.(drop|add)_foreign_key|drop_constraint|create_foreign_key|foreign_keys' migrations || trueRepository: OWASP/OpenCRE
Length of output: 50370
Handle ingest_chunk foreign keys before rewriting artifact_ingest_event.
In existing artifact_ingest_event rows without the unique constraint, op.batch_alter_table("artifact_ingest_event", ...):
with op.batch_alter_table("artifact_ingest_event") as batch_op:
batch_op.create_unique_constraint(
"uq_artifact_ingest_event_run_artifact",
["run_id", "artifact_id"]
)drops and recreates the table. ingest_chunk has ondelete="CASCADE" for that parent table, so enable foreign keys temporarily, handle or drop/re-add the child foreign key, and add a regression test for existing rows before the parent-table rewrite.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py` around
lines 57 - 61, Before the artifact_ingest_event batch rewrite and
create_unique_constraint operation, handle the ingest_chunk foreign key
explicitly: temporarily enable foreign-key enforcement, drop the child
constraint, and recreate it with its existing cascade behavior after the
rewrite. Add a regression test covering pre-existing artifact_ingest_event rows
and verifying the ingest_chunk relationship remains valid.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 8 --glob '*.py' 'UniqueConstraint|unique\s*=\s*True|create_index.*unique|artifact_ingest_event|ingest_chunk' . || trueRepository: OWASP/OpenCRE
Length of output: 48645
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- alembic constraints in migration ---\n'
cd migrations/versions
sed -n '1,115p' 9f1a2b3c4d5e_add_artifact_ingest_persistence.py
printf '\n--- all batch_alter_table references ---\n'
rg -n -C 3 'batch_alter_table|create_unique_constraint|get_unique_constraints|unique_unique=|sqlite_implicit.*unique|preserve_default' migrations/versions applications 2>/dev/null || true
printf '\n--- package versions mentioning alembic/sqlalchemy ---\n'
for f in requirements.txt pyproject.toml Pipfile Pipfile.lock poetry.lock setup.cfg setup.py; do
[ -f "$f" ] && { echo "--- $f"; rg -n 'Alembic|alembic|SQLAlchemy|sqlalchemy' "$f" || true; }
doneRepository: OWASP/OpenCRE
Length of output: 242
🌐 Web query:
Alembic op.batch_alter_table SQLite unnamed UNIQUE constraints documentation
💡 Result:
In Alembic, the primary challenge with SQLite and unnamed UNIQUE constraints is that SQLite allows constraints to exist without a name, whereas other database backends assign names to all constraints [1][2]. Because Alembic’s batch operations (op.batch_alter_table) rely on dropping and recreating tables to perform schema changes—and because dropping a constraint in this process requires a name to target the correct object—unnamed constraints often cannot be automatically handled by the standard migration workflow [1][3]. Key points regarding unnamed UNIQUE constraints in SQLite batch migrations include: Naming Conventions: You can resolve issues with unnamed constraints by using the naming_convention argument within op.batch_alter_table [1][4]. This allows you to apply a naming convention to reflected constraints during the migration process, ensuring that previously unnamed constraints are assigned names that Alembic can then manage [1][3]. Explicit Inclusion: If a constraint remains unnamed or is difficult to manage via reflection, you can manually include it in the recreated table by using the table_args parameter within the batch_alter_table block [1][2]. This forces the inclusion of the specified constraint during the table recreation process [1]. Workarounds for Removal: Because there is no name to reference, dropping an existing unnamed constraint is inherently difficult. In some scenarios, developers must perform manual, multi-step migrations: for example, creating a new table with the desired named constraint, migrating data, dropping the old table, and renaming the new one, or using specialized naming conventions to retroactively "identify" the constraint for removal [5]. For best practices, it is strongly recommended to use SQLAlchemy’s constraint naming conventions globally so that all UNIQUE constraints are named when they are created, preventing these issues from arising in future migrations [3][5].
Citations:
- 1: https://alembic.sqlalchemy.org/en/latest/batch.html
- 2: sqlalchemy/alembic@8b8fe7b
- 3: https://alembic.sqlalchemy.org/en/latest/ops.html?highlight=unique
- 4: https://alembic.sqlalchemy.org/en/latest/batch.html?highlight=autogen+batch
- 5: unnamed constraint concerns sqlalchemy/alembic#1715
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- alembic constraints in migration ---'
sed -n '1,115p' migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py
printf '%s\n' ''
printf '%s\n' '--- all batch_alter_table references ---'
rg -n -C 3 'batch_alter_table|create_unique_constraint|get_unique_constraints|unique_unique=|sqlite_implicit.*unique|preserve_default' migrations/versions application application/tests 2>/dev/null || true
printf '%s\n' ''
printf '%s\n' '--- package versions mentioning alembic/sqlalchemy ---'
for f in requirements.txt pyproject.toml Pipfile Pipfile.lock poetry.lock setup.cfg setup.py; do
[ -f "$f" ] && { echo "--- $f"; rg -n 'Alembic|alembic|SQLAlchemy|sqlalchemy' "$f" || true; }
doneRepository: OWASP/OpenCRE
Length of output: 23567
🌐 Web query:
Alembic batch alter table create_unique_constraint preserves unnamed SQLite unique constraints documentation
💡 Result:
Alembic's batch_alter_table operation does not reliably preserve unnamed unique constraints when migrating SQLite databases [1][2]. Because SQLite allows unique constraints to exist without an explicit name, these constraints can be difficult for Alembic to track or recreate during the table-copy process that batch mode requires [3][4][5]. Key points regarding this behavior include: 1. Limitation in Batch Mode: When batch_alter_table is used, Alembic often recreates the table to support operations (like dropping columns or constraints) that SQLite does not natively support [3][6]. If an existing unique constraint is unnamed, Alembic may fail to include it in the new table definition automatically, effectively dropping it during the migration [1][2]. 2. Naming Conventions as the Solution: To ensure unique constraints are preserved, it is strongly recommended to use SQLAlchemy and Alembic's naming convention features [3][7]. By configuring a naming convention (e.g., in your env.py via target_metadata), Alembic will automatically assign a name to all constraints [7]. This makes them explicitly identifiable, ensuring they are correctly included when the table is recreated [3][7]. 3. Workarounds: If you encounter issues with existing unnamed constraints, you may need to explicitly define them in your migration scripts with a name [4]. In some cases, it may be necessary to add a named constraint, drop the old unnamed one, or manually ensure the new table definition includes all required constraints [4]. For reliable schema management on SQLite, implementing global constraint naming conventions is the standard practice in Alembic [8][9][7].
Citations:
- 1: https://alembic.sqlalchemy.org/en/latest/ops.html?highlight=unique
- 2: https://alembic.sqlalchemy.org/en/latest/ops.html
- 3: https://alembic.sqlalchemy.org/en/latest/batch.html
- 4: unnamed constraint concerns sqlalchemy/alembic#1715
- 5: https://groups.google.com/g/sqlalchemy-alembic/c/jnMDRSDlxrk
- 6: SQLite batch migrations: an option for always using `recreate` style on SQLite sqlalchemy/alembic#882
- 7: https://alembic.sqlalchemy.org/en/latest/naming.html
- 8: Unique constraint missing name/duplicated create_unique/drop_constraint sqlalchemy/alembic#1532
- 9:
ValueError: Constraint must have a namein alembic 1.10.0 sqlalchemy/alembic#1195
🌐 Web query:
Alembic version 1.14 create_unique_constraint unnamed SQLite unique constraint preserved
💡 Result:
In SQLite, unique constraints do not require a name, which creates significant challenges for migration tools like Alembic because standard operations—specifically dropping or altering constraints—typically require a name to target the correct database object [1][2][3]. Alembic does not automatically "preserve" unnamed SQLite unique constraints in a way that allows them to be seamlessly managed or dropped in subsequent migrations [1][4]. If you have an existing unnamed unique constraint in SQLite, Alembic's batch operations cannot target it directly using drop_constraint because there is no name to reference [1][5]. To resolve this, the recommended approach is to ensure all constraints are explicitly named [3][6]. For managing this in migrations: 1. Naming Conventions: Use a naming_convention in your environment configuration (env.py) to ensure all constraints, including those that would otherwise be unnamed, are assigned a predictable name [1][6]. 2. Manual Correction: If you already have unnamed constraints in your SQLite database, you may need to recreate the affected table. This is because SQLite does not support ALTER TABLE...DROP CONSTRAINT; dropping a constraint in SQLite requires creating a new table with the desired schema (without the constraint) and copying the data over [1][4][5]. 3. Batch Mode: When working with SQLite, always use batch_alter_table. If you have defined a naming convention, you can use batch_op.f('your_constraint_name') to correctly reference the named constraint within your migration scripts [1][5][6]. Alembic 1.14 continues to rely on these established practices for SQLite, as the limitation is intrinsic to the SQLite database engine itself [1][7][2].
Citations:
- 1: https://alembic.sqlalchemy.org/en/latest/batch.html
- 2: https://alembic.sqlalchemy.org/en/latest/ops.html?highlight=unique
- 3: sqlite inspection with get_unique_constraints doesn't pick up (all) UNIQUE constraints sqlalchemy/sqlalchemy#3244
- 4: Rename Unique Constraint sqlalchemy/alembic#246
- 5: unnamed constraint concerns sqlalchemy/alembic#1715
- 6: https://alembic.sqlalchemy.org/en/latest/naming.html
- 7: https://pypi.org/project/alembic/1.14.1/
🌐 Web query:
Alembic batch_alter_table recreate constraint reflection unnamed unique constraints source
💡 Result:
When using Alembic's batch_alter_table, particularly with SQLite, unnamed unique constraints present a challenge because the table recreation process relies on constraint names to manage, drop, or copy existing schema elements [1][2][3]. Because unnamed constraints lack a referenceable name, they are often silently omitted during the table copy operation [1][2]. To handle this, you have the following options: 1. Use a Naming Convention (Recommended) The most robust solution is to define a naming convention in your environment's MetaData. This ensures that all constraints, including those that would otherwise be unnamed, are assigned consistent, predictable names [1][2][4]. When you provide a naming_convention argument to batch_alter_table, Alembic can use this to resolve names during the reflection and recreation process [1][2]. 2. Explicitly Re-add Constraints If a constraint is unnamed and not being preserved automatically, you can explicitly include it in the batch_alter_table block by passing it to the table_args parameter [1][3]. Example: with op.batch_alter_table("some_table", table_args=[UniqueConstraint('col1', 'col2', name='my_unique_constraint')]) as batch_op: batch_op.add_column(Column('new_col', Integer)) 3. Manual Workaround If you cannot change the schema naming at the model level, you can manually force the naming and recreation in your migration script by performing a sequence of operations that replaces the unnamed constraint with a named one. This typically involves adding a named constraint, then dropping and recreating the table/constraint structure within the migration [4]. Key Considerations: - Reflection: Batch mode relies on table reflection to understand the existing structure [2][3]. If the database is not in "online" mode, or if the underlying database driver (like SQLite) does not provide a name for the constraint during reflection, Alembic cannot track it [2][3]. - SQLite Specifics: Unlike other database backends that assign implicit names to all constraints, SQLite allows unique constraints to exist without names [1][3]. This is the primary reason why these issues are most frequently encountered with SQLite batch migrations [1][4]. - Upgrade/Downgrade: Since unnamed constraints cannot be targeted by drop_constraint, you must ensure your migration properly defines names for all constraints that need to be removed or modified in future migrations [4][5].
Citations:
- 1: https://alembic.sqlalchemy.org/en/latest/batch.html
- 2: https://alembic.sqlalchemy.org/en/latest/ops.html
- 3: sqlalchemy/alembic@8b8fe7b
- 4: unnamed constraint concerns sqlalchemy/alembic#1715
- 5: ValueError: Constraint must have a name sqlalchemy/alembic#1625
🌐 Web query:
Alembic batch_alter_table recreate_always table copy unnamed unique constraint SQLite source
💡 Result:
In Alembic, when using batch_alter_table on SQLite, the system often performs a table recreation ("move and copy") to handle structural changes [1][2][3]. A known limitation is that this process may fail to copy unnamed UNIQUE constraints [1]. Because these constraints lack a name, Alembic cannot reliably reference, drop, or recreate them during the migration process [4][5]. To resolve or avoid issues with unnamed UNIQUE constraints in SQLite batch migrations, consider the following approaches: 1. Use Naming Conventions: The most robust solution is to define a naming convention in your SQLAlchemy MetaData [2][4]. By assigning names to all constraints, you ensure that Alembic can properly track and recreate them during batch operations [2]. 2. Manual Recreation: If you have existing unnamed constraints, you may need to write a migration that manually drops and recreates the table or constraints to explicitly assign them names [4]. This process typically involves: - Creating a named version of the constraint. - Dropping the constraint (which, depending on the situation, might require careful sequencing). - Once named, the constraint will be correctly managed by future batch operations [4]. 3. Recreate Always: Setting recreate='always' in batch_alter_table forces the table to be recreated [1][2]. While this ensures the table structure is fully re-reflected, it does not inherently solve the problem of unnamed constraints if they were not correctly identified or included in the reflection process to begin with [1][6]. Alembic documentation explicitly advises that the table copy operation may not copy unnamed UNIQUE constraints and suggests using naming conventions as the primary workaround [1][2]. For detailed implementation of naming conventions, refer to the SQLAlchemy documentation on integrating naming conventions into autogenerate processes [2].
Citations:
- 1: https://alembic.sqlalchemy.org/en/latest/ops.html
- 2: https://alembic.sqlalchemy.org/en/latest/batch.html
- 3: https://github.com/sqlalchemy/alembic/blob/main/docs/build/batch.rst
- 4: unnamed constraint concerns sqlalchemy/alembic#1715
- 5: https://blog.miguelgrinberg.com/post/fixing-alter-table-errors-with-flask-migrate-and-sqlite
- 6: batch_alter_table forgets unchanged Enum CHECK constraint sqlalchemy/alembic#400
Preserve existing unnamed unique constraints during SQLite batch rewrite.
The existing-table paths use op.batch_alter_table before creating the named unique constraint. Alembic can drop and recreate SQLite tables in batch mode without copying unnamed UNIQUE constraints. If artifact_ingest_event or ingest_chunk has an unnamed uniqueness rule, copy/replace it first, reject unsupported legacy schemas, or use recreate="always" with explicit table_args to preserve it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@migrations/versions/9f1a2b3c4d5e_add_artifact_ingest_persistence.py` around
lines 57 - 61, Update the existing-table migration paths using
op.batch_alter_table for artifact_ingest_event and ingest_chunk so unnamed
UNIQUE constraints are preserved during SQLite table recreation. Copy or replace
each supported unnamed uniqueness rule before adding the named constraint,
reject unsupported legacy schemas, or use recreate="always" with explicit
table_args; ensure no existing uniqueness rule is silently dropped.
1 participant
Problem
Several recent migrations were not fully compatible with SQLite, causing errors when developers run
make migrate-upgradeor./scripts/update-cwe.sh:op.create_unique_constraint()afterCREATE TABLEfails withNo support for ALTER of constraints in SQLite dialect.Missing
document_metadatacolumn increandnodetables causesno such columnerrors.Missing
embedding_veccolumn inembeddingstable causesno such column: embeddings.embedding_vec.Some migrations are not idempotent, causing
table already existserrors on re-runs.Solution
9f1a2b3c4d5e– DefineUniqueConstraintinsideCREATE TABLE;addtable_existsguards.055dbd9f8bfe– Adddocument_metadatawith column existence checks (new migration).967016ee10fa– Addembedding_vecasTEXTfor SQLite with existence check (new migration).Testing
make migrate-upgrade→make upstream-sync→./scripts/update-cwe.shall succeed.✅ Existing database: re-runs are idempotent, skip already-created objects.
✅ No regressions for PostgreSQL.
Impact
Developers using SQLite can now run migrations and import data without manual workarounds.
Makes the project more contributor‑friendly for SQLite users.
Ready for review. Let me know if any adjustments are needed.