Fix delta deploy failure when cached and changed files share directories#689
Fix delta deploy failure when cached and changed files share directories#689
Conversation
TarFS used os.Mkdir to create directories from tar entries, which fails if the directory already exists. During a delta deploy, PrepareUpload stages cached files into the session directory (creating parent dirs with os.MkdirAll), then BuildFromPrepared extracts the changed-files tar into that same directory. If the tar contains directory entries that were already created during staging, the deploy blows up with "error extracting changed files: mkdir .../db: file exists". Tolerate pre-existing directories by checking os.IsExist on the error, matching how extractFilesFromLayer already handles this.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe Comment |
2 participants
We hit this in prod deploying cloud to a cluster running v0.6.0 — the deploy failed with
error extracting changed files: mkdir /tmp/buildkit-3500590234/db: file exists. The workaround was to SSH in and nuke the source code cache, which forced a full upload instead of a delta.The issue is a collision between the two phases of a delta deploy.
PrepareUploadstages cached files into a temp directory, creating parent directories as needed withos.MkdirAll. ThenBuildFromPreparedextracts the changed-files tar into that same directory usingTarFS, which calls bareos.Mkdir— no tolerance for directories that already exist. If any directory appears in both the cached and changed file sets (likedb/in this case), boom.One-line fix: add an
os.IsExistcheck soTarFSskips directory entries that are already present, just likeextractFilesFromLayeralready does. Added a test that reproduces the exact scenario.