ENG-2881 Raise error on startup when migrations fail

[erosselli]

Ticket ENG-2881

Description Of Changes

Currently when migrations run on startup and fail for whatever reason, the server still starts up normally. This means application code is deployed and “healthy”, but it assumes the migrations ran successfully (e.g assumes a new table was created when it wasn’t) , leading to errors and a generally inconsistent state.

We should simply raise the error when migrations fail.

Steps to Confirm

1. Create a migration that just runs a raise Exception('error') code, or otherwise break your migrations (e.g removing the latest migration file)
2. Check the error is raised and server startup does not continue

output from my test:

fides  | 2026-03-04 16:06:43.011 | ERROR    | fides.api.app_setup:run_database_startup:229 - Error occurred during database configuration: Can't locate revision identified by 'bf12f05ef8eb'
fides  | Traceback (most recent call last):
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/base.py", line 233, in _catch_revision_errors
fides  |     yield
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/base.py", line 443, in _upgrade_revs
fides  |     for script in reversed(list(revs))
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 799, in iterate_revisions
fides  |     revisions, heads = fn(
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 1454, in _collect_upgrade_revisions
fides  |     current_revisions = self.get_revisions(lower)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 527, in get_revisions
fides  |     return sum([self.get_revisions(id_elem) for id_elem in id_], ())
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 552, in get_revisions
fides  |     return tuple(
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 553, in <genexpr>
fides  |     self._revision_for_ident(rev_id, branch_label)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/revision.py", line 624, in _revision_for_ident
fides  |     raise ResolutionError(
fides  | alembic.script.revision.ResolutionError: No such revision or branch 'bf12f05ef8eb'
fides  | 
fides  | The above exception was the direct cause of the following exception:
fides  | 
fides  | Traceback (most recent call last):
fides  |   File "<string>", line 1, in <module>
fides  |   File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 122, in spawn_main
fides  |     exitcode = _main(fd, parent_sentinel)
fides  |   File "/usr/local/lib/python3.13/multiprocessing/spawn.py", line 135, in _main
fides  |     return self._bootstrap(parent_sentinel)
fides  |   File "/usr/local/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
fides  |     self.run()
fides  |   File "/usr/local/lib/python3.13/multiprocessing/process.py", line 108, in run
fides  |     self._target(*self._args, **self._kwargs)
fides  |   File "/opt/fides/lib/python3.13/site-packages/uvicorn/_subprocess.py", line 80, in subprocess_started
fides  |     target(sockets=sockets)
fides  |   File "/opt/fides/lib/python3.13/site-packages/uvicorn/server.py", line 65, in run
fides  |     return asyncio.run(self.serve(sockets=sockets))
fides  |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 195, in run
fides  |     return runner.run(main)
fides  |   File "/usr/local/lib/python3.13/asyncio/runners.py", line 118, in run
fides  |     return self._loop.run_until_complete(task)
fides  |   File "/opt/fides/lib/python3.13/site-packages/uvicorn/lifespan/on.py", line 86, in main
fides  |     await app(scope, self.receive, self.send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
fides  |     return await self.app(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/applications.py", line 1134, in __call__
fides  |     await super().__call__(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/applications.py", line 107, in __call__
fides  |     await self.middleware_stack(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/middleware/errors.py", line 151, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/fides/src/fides/api/asgi_middleware.py", line 57, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/fides/src/fides/api/asgi_middleware.py", line 57, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/middleware/gzip.py", line 19, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/fides/src/fides/api/asgi_middleware.py", line 57, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 49, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
fides  |     await self.app(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
fides  |     await self.middleware_stack(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/routing.py", line 725, in app
fides  |     await self.lifespan(scope, receive, send)
fides  |   File "/opt/fides/lib/python3.13/site-packages/starlette/routing.py", line 694, in lifespan
fides  |     async with self.lifespan_context(app) as maybe_state:
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
fides  |     return await anext(self.gen)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/routing.py", line 211, in merged_lifespan
fides  |     async with original_context(app) as maybe_original_state:
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
fides  |     return await anext(self.gen)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/routing.py", line 211, in merged_lifespan
fides  |     async with original_context(app) as maybe_original_state:
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
fides  |     return await anext(self.gen)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/routing.py", line 211, in merged_lifespan
fides  |     async with original_context(app) as maybe_original_state:
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
fides  |     return await anext(self.gen)
fides  |   File "/opt/fides/lib/python3.13/site-packages/fastapi/routing.py", line 211, in merged_lifespan
fides  |     async with original_context(app) as maybe_original_state:
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
fides  |     return await anext(self.gen)
fides  |   File "/fides/src/fides/api/main.py", line 89, in lifespan
fides  |     await run_database_startup(wrapped_app)
fides  | > File "/fides/src/fides/api/app_setup.py", line 220, in run_database_startup
fides  |     configure_db(CONFIG.database.sync_database_uri)
fides  |   File "/fides/src/fides/api/db/database.py", line 163, in configure_db
fides  |     migrate_db(database_url, revision=revision)  # type: ignore[arg-type]
fides  |   File "/fides/src/fides/api/db/database.py", line 98, in migrate_db
fides  |     upgrade_db(alembic_config, revision)
fides  |   File "/fides/src/fides/api/db/database.py", line 66, in upgrade_db
fides  |     command.upgrade(alembic_config, revision)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/command.py", line 322, in upgrade
fides  |     script.run_env()
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/base.py", line 569, in run_env
fides  |     util.load_python_file(self.dir, "env.py")
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
fides  |     module = load_module_py(module_id, path)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
fides  |     spec.loader.exec_module(module)  # type: ignore
fides  |   File "<frozen importlib._bootstrap_external>", line 1023, in exec_module
fides  |   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
fides  |   File "/fides/src/fides/api/db/../alembic/migrations/env.py", line 88, in <module>
fides  |     run_migrations_online()
fides  |   File "/fides/src/fides/api/db/../alembic/migrations/env.py", line 82, in run_migrations_online
fides  |     context.run_migrations()
fides  |   File "<string>", line 8, in run_migrations
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
fides  |     self.get_context().run_migrations(**kw)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/runtime/migration.py", line 611, in run_migrations
fides  |     for step in self._migrations_fn(heads, self):
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/command.py", line 311, in upgrade
fides  |     return script._upgrade_revs(revision, rev)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/base.py", line 431, in _upgrade_revs
fides  |     with self._catch_revision_errors(
fides  |   File "/usr/local/lib/python3.13/contextlib.py", line 162, in __exit__
fides  |     self.gen.throw(value)
fides  |   File "/opt/fides/lib/python3.13/site-packages/alembic/script/base.py", line 265, in _catch_revision_errors
fides  |     raise util.CommandError(resolution) from re
fides  | alembic.util.exc.CommandError: Can't locate revision identified by 'bf12f05ef8eb'

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

Summary by CodeRabbit

• Bug Fixes
  • Server startup now fails with a clear error message if database migrations cannot be applied, preventing the server from running in a potentially inconsistent state.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ready	Preview, Comment	Mar 4, 2026 4:24pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
fides-privacy-center	Ignored		Mar 4, 2026 4:24pm

[coderabbitai]

📝 Walkthrough

Walkthrough

The pull request adds error handling to abort server startup when database migrations fail. An exception during migration is caught, logged in detail, and re-raised as a FidesError to prevent the server from starting in an inconsistent state. A changelog entry documents this behavior change.

Changes

Cohort / File(s)	Summary
Changelog Documentation changelog/7562-raise-error-on-startup-when-migrations-fail.yaml	Changelog entry documenting that startup now raises an error when database migrations fail.
Startup Error Handling src/fides/api/app_setup.py	Modified exception handler in run_database_startup to construct detailed error logs and re-raise migration failures as FidesError to abort server startup.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 When migrations stumble and fail in the night,
We catch the error and set it right—
No silent failures, no startup so sly,
Just a clean abort and a clear goodbye! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title clearly and specifically summarizes the main change: raising an error on startup when database migrations fail.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	PR description is comprehensive and well-structured, covering all major template sections with clear explanation of changes, confirmation steps, and a thorough pre-merge checklist.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch erosselli/ENG-2881

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Greptile Summary

This PR addresses a correctness issue where database migration failures during server startup were silently logged, allowing the server to continue running in an inconsistent state. The fix converts the silent logging approach into a hard error that aborts startup.

Key changes:

• src/fides/api/app_setup.py: In run_database_startup, the exception handler for the automigrate path now logs the error with logger.exception() and re-raises it as a FidesError (with proper exception chaining via from e), ensuring startup fails immediately on any migration error. This is consistent with how other startup errors in the same function are handled.
• changelog/7562-raise-error-on-startup-when-migrations-fail.yaml: Changelog entry documenting the behavior change.

What's good:

• The logic change is straightforward and correct: failing migrations should abort startup
• Proper exception chaining with from e preserves the original traceback
• Includes a clear comment explaining why the re-raise is intentional
• Follows the pattern of other startup error handlers in the same function

Confidence Score: 4/5

• Safe to merge — the change is small, well-scoped, and correctly solves a real correctness problem by aborting server startup on migration failure.
• The logic change is straightforward and correct. Converting silent error logging to a hard abort on migration failure is the right behavior. Exception chaining is properly implemented. The only minor note is the absence of new tests to verify the abort-on-failure behavior, which slightly reduces confidence for long-term maintainability, but the change itself is correct and safe to deploy immediately.
• No files require special attention.

_{Last reviewed commit: 0b8bd68}

[adamsachs]

looks good, thanks for taking this on! 🙏

the main risk i see here is just around expectation setting/understanding implications for deployment. i know we've communicated with our platform team, so that's a good start. i'm thinking that in addition to that, it's probably worth:

1. calling that out again to the platform team to make sure this is fully consistent + good with our deployment approach on our cloud
2. make sure we're over-communicating about it for customer-hosted deployments (e.g. it needs to be high up in release notes, probably worth some explicit heads-up on slack, etc.)

i don't anticipate any red flags, but this is a fairly consequential change in app behavior (a positive one!) that we should be sure to give heads-up on.