fix(backfill): Handle SQLite lock errors with atomic creation and graceful 503

[lohitkolluri]

Closes: #66726

Summary

Creating a backfill via the API on SQLite can fail under concurrent access (Scheduler, DAG Processor, and API Server sharing the same database).

Currently, this may result in:

• Partially-created backfills without DagRuns
• SQLite errors from unsupported row-lock operations
• HTTP 500 responses exposing internal database errors

This PR improves the experience by making backfill creation atomic, handling SQLite-specific locking behavior correctly, and returning a retryable error when the database is temporarily locked.

Changes

• Atomic Backfill Creation

  • Replace session.commit() with session.flush() during backfill creation
  • Allows the outer transaction to commit or roll back the entire operation atomically
  • Prevents partially-created backfills when DagRun creation fails

• SQLite Compatibility

  • Skip with_for_update() for SQLite in with_row_locks()
  • Avoids generating unsupported SELECT ... FOR UPDATE statements

• Better Error Handling

  • Catch SQLite "database is locked" OperationalErrors
  • Return HTTP 503 Service Unavailable with a clear retry message
  • Continue propagating other OperationalErrors unchanged

Result

• No partial backfill state on failure
• No SQLite row-lock compilation issues
• Clear, retryable API response instead of a generic 500
• No behavior changes for PostgreSQL or MySQL

Testing

• Added test_create_backfill_database_locked
• Verified 503 response and expected error message
• Existing backfill endpoint tests continue to pass
• Ruff and formatting checks pass

AI Assistance

• Yes

[23tae]

Thanks for working on this.

#66726 also reports that the backfill may still be created even though the request fails.

This PR seems to improve the error response from 500 to 503, but it is not clear whether it changes the partial-success behavior. Could we clarify this in the PR description?

[23tae]

Thanks for the update. I left a few comments.

[henry3260]

Thanks for the fix!