feat: COW workspace caching for local runs#26
Merged
Conversation
Add build_host_config helper and start_long_lived_with_mounts method to DockerClient so containers can mount host workspace directories via HostConfig bind mounts. This enables the COW workspace caching feature to bind-mount overlay filesystems into build containers.
Introduces WorkspaceManager that manages per-step workspace directories for a single pipeline run. Auto-selects between clone strategy (macOS APFS / Linux reflink / fallback cp) and overlay strategy (fuse-overlayfs) based on platform detection from hm-util::cow. The rest of the system only sees workspace_path() — strategy is transparent.
- Add duplicate step_key guard in WorkspaceManager - Add tests for create_workspace_from_cache and duplicate key - Deduplicate start_long_lived via delegation to start_long_lived_with_mounts
When RunContext.workspace is Some, DockerRunner dispatches to a new run_step_cow path that bind-mounts the host workspace directory into the container instead of extracting a tar archive and committing a docker snapshot. This eliminates the tar extraction and docker commit overhead for the COW execution mode.
Persist completed step workspaces to ~/.harmont/cache/workspaces/ and restore them on cache hits, replacing Docker image-based caching when running in COW mode. Stale cache directories from previous keys are evicted after a successful build.
manual_let_else, missing_errors_doc, missing_panics_doc, expect_used, redundant_closure_for_method_calls, unnecessary_unwrap, collapsible_if.
…tance COW mode now commits container images after each step, so child steps inherit system-level changes (apt-get installs, rustup, etc.) via Docker image lineage while workspace files propagate via COW bind mounts.
Remove --cow flag, non-COW Docker tar-extract path, Docker-image-only cache backend, and associated dead code. COW bind mounts + Docker image commits for system state is now the default and only mode. -465 lines removed.
FUSE mounts are only visible to the mounting user by default. Docker daemon runs as root so it can't bind-mount from fuse-overlayfs paths. Adding allow_other fixes this. The probe now also tests a real mount with allow_other so it falls back gracefully when user_allow_other is not enabled in /etc/fuse.conf. CI workflows updated to uncomment user_allow_other in /etc/fuse.conf.
diagnose_strategies() probes each COW backend and reports availability with reasons. WorkspaceManager warns at startup when falling back to FullCopy and logs which strategies were tried and why they failed.
Docker containers run as root but fuse-overlayfs runs as the host user. Without UID squashing, the FUSE daemon can't create files owned by UID 0 in the upper dir, causing EPERM on mkdir inside bind-mounted overlays.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.