Task Summary
Found while testing the flag-gated per-user warehouse feature (#6870); no deployment is affected because the flag defaults to off.
LakekeeperClient.deleteWarehouseEmptyFirst (used by DELETE /warehouse/{whid}, introduced in #7473) empties the warehouse by dropping every table with purgeRequested=true, then immediately deletes the warehouse entity. Lakekeeper runs those purges as asynchronous tasks (queue tabular_purge), so the warehouse delete races the queue: while any purge task is still pending, Lakekeeper answers
HTTP 409 {"error":{"message":"Warehouse has unfinished tasks. Cannot delete warehouse until all tasks are finished.","type":"WarehouseHasUnfinishedTasks","code":409}}
and the client surfaces it as a failure. In practice the first delete of any warehouse that has ever stored execution results always fails; a second attempt after the queue drained succeeds.
To reproduce: create a warehouse, run one workflow execution into it (this creates three tables: result, stats, console), then delete the warehouse — the first attempt fails with the 409 above ("Unfinished tasks: 3 Tasks in queue 'tabular_purge'"), and a retry a few seconds later succeeds. Deterministic unit-level repro: a LakekeeperClientSpec case whose stubbed management endpoint answers the warehouse delete with 409 WarehouseHasUnfinishedTasks twice and then 204 — deleteWarehouseEmptyFirst should succeed, but currently throws on the first 409.
Expected: deleteWarehouseEmptyFirst treats 409 WarehouseHasUnfinishedTasks as transient and retries the final warehouse delete with a bounded backoff (the purge normally drains within seconds), failing only if the queue never drains within the bound. All other error responses still fail immediately.
Part of #6870.
Task Type
Task Summary
Found while testing the flag-gated per-user warehouse feature (#6870); no deployment is affected because the flag defaults to off.
LakekeeperClient.deleteWarehouseEmptyFirst(used byDELETE /warehouse/{whid}, introduced in #7473) empties the warehouse by dropping every table withpurgeRequested=true, then immediately deletes the warehouse entity. Lakekeeper runs those purges as asynchronous tasks (queuetabular_purge), so the warehouse delete races the queue: while any purge task is still pending, Lakekeeper answersand the client surfaces it as a failure. In practice the first delete of any warehouse that has ever stored execution results always fails; a second attempt after the queue drained succeeds.
To reproduce: create a warehouse, run one workflow execution into it (this creates three tables: result, stats, console), then delete the warehouse — the first attempt fails with the 409 above (
"Unfinished tasks: 3 Tasks in queue 'tabular_purge'"), and a retry a few seconds later succeeds. Deterministic unit-level repro: aLakekeeperClientSpeccase whose stubbed management endpoint answers the warehouse delete with 409WarehouseHasUnfinishedTaskstwice and then 204 —deleteWarehouseEmptyFirstshould succeed, but currently throws on the first 409.Expected:
deleteWarehouseEmptyFirsttreats 409WarehouseHasUnfinishedTasksas transient and retries the final warehouse delete with a bounded backoff (the purge normally drains within seconds), failing only if the queue never drains within the bound. All other error responses still fail immediately.Part of #6870.
Task Type