Kubernetes Dockerless build drops .devsy-internal during StreamMount and fails before build starts

[joshyorko]

Summary

On Devsy 1.15.0 with the Kubernetes provider, a Dockerfile-backed devcontainer can fail before the Dockerless build starts because the generated .devsy-internal build artifacts are excluded from the streamed workspace mount.

Observed error:

dockerless build: rename dir: open /workspaces/.dockerless/.devsy-internal: no such file or directory

I reproduced this with https://github.com/joshyorko/actions against a Kubernetes context where the equivalent Skevetter DevPod 0.26.1 flow gets through the Dockerless build successfully.

Reproducer

Workspace:

https://github.com/joshyorko/actions

The repo contains a Dockerfile-backed .devcontainer/devcontainer.json, so the Kubernetes provider correctly falls back to Dockerless.

Run:

devsy workspace up https://github.com/joshyorko/actions

Environment from the failing run:

• Devsy Desktop / CLI: 1.15.0
• provider: Kubernetes
• Kubernetes context: ror
• namespace: devsy
• Dockerless image: ghcr.io/devsy-org/dockerless:0.2.0
• architecture: amd64

Actual behavior

Devsy successfully:

1. clones the repository;
2. resolves .devcontainer/Dockerfile;
3. resolves the Python base image;
4. calculates the prebuild hash;
5. creates the PVC and Kubernetes pod;
6. injects the Devsy agent;
7. streams the workspace into /workspaces/actions.

The pod is configured with:

DOCKERLESS_CONTEXT=/workspaces/actions
DOCKERLESS_DOCKERFILE=/workspaces/actions/.devsy-internal/Dockerfile-without-features

The workspace upload reaches ~102 MB, then setup fails immediately:

Copy .../actions into DevContainer /workspaces/actions
Uploaded 102.69 MB
dockerless build: rename dir: open /workspaces/.dockerless/.devsy-internal: no such file or directory

Dockerless never reaches its actual build command.

Expected behavior

The generated Dockerless build information should survive the host -> Kubernetes workspace handoff so that prepareBuildDirectory can find either:

/workspaces/actions/.devsy-internal

or the fallback:

/workspaces/.dockerless/.devsy-internal

and then start the Dockerless build.

Likely regression / code-path evidence

The Dockerless fallback creates generated build artifacts under the context path:

// pkg/devcontainer/build.go

devsyInternalFolder := filepath.Join(contextPath, config.DevsyContextFeatureFolder)
err := os.MkdirAll(devsyInternalFolder, 0o755)
...
devsyDockerfile = filepath.Join(devsyInternalFolder, "Dockerfile-without-features")

DevsyContextFeatureFolder is the generated .devsy-internal directory.

However, pkg/agent/tunnelserver/tunnelserver.go currently applies build-artifact exclusions when streaming a mount:

excludes := append(t.workspaceIgnoreExcludes(), config.BuildArtifactExcludes()...)
...
extract.WriteTarExclude(buf, mount.Source, false, excludes)

and pkg/devcontainer/config/artifacts.go defines:

var buildArtifactNames = []string{
    DevsyContextFeatureFolder,
}

So .devsy-internal is generated locally and then explicitly excluded from the mount stream that carries the workspace into the Kubernetes pod.

Inside the pod, prepareBuildDirectory then cannot find the build directory at the workspace path and falls back to /workspaces/.dockerless/.devsy-internal, which is also absent, producing the observed rename dir failure.

Working control: Skevetter DevPod

I ran the same joshyorko/actions workspace through Skevetter's DevPod 0.26.1 using the Kubernetes provider and the same Kubernetes context.

DevPod's analogous StreamMount only applies its workspace / .devpodignore exclusions; it does not add an equivalent build-artifact exclusion for .devpod-internal.

That run successfully reaches:

Copy .../actions/content into DevContainer /workspaces/actions
Uploaded 98.90 MB
starting dockerless build: /.dockerless/dockerless build ...
...
dockerless build completed

Suggested fix direction

StreamMount should not blanket-exclude .devsy-internal when that stream is also responsible for transporting the generated Dockerless build artifacts required by the Kubernetes fallback.

Possible approaches:

• do not apply BuildArtifactExcludes() to this Dockerless workspace stream; or
• explicitly seed/persist .devsy-internal into /workspaces/.dockerless before excluding it from the normal workspace payload.

The key invariant is that the generated Dockerless build directory must exist in one of the locations expected by prepareBuildDirectory before DockerlessBuild runs.

Acceptance criteria

• A fresh Kubernetes-provider workspace up for a Dockerfile-backed devcontainer reaches the Dockerless build instead of failing in prepareBuildDirectory.
• .devsy-internal is available at the workspace build context or the documented Dockerless fallback location inside the pod.
• Existing cleanup behavior still removes generated build artifacts after setup.
• Add a regression test covering streamed mounts + Dockerless generated build artifacts so BuildArtifactExcludes() cannot remove the only copy needed by the in-pod builder.

[coderabbitai]

🔗 Related PRs

#398 - fix: use actual container mount path after workspace rename [merged]
#486 - fix(workspace): delete container on rename to fix stale bind mounts [merged]
#519 - fix(docker): stale bind-mount inode to Docker Desktop on up/delete/up [merged]
#603 - feat(ci): add devsy ci command for devcontainer CI [merged]
#615 - fix(delivery): cleanup build files for mounted volume [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[joshyorko]

Additional independent reproducer: joshyorko/room-of-requirement.

This is a useful control because the same repo is known to build successfully on Kubernetes with Skevetter's DevPod provider path, while Devsy 1.15.0 fails at the same .devsy-internal handoff before Dockerless starts.

Devsy run details:

• workspace: https://github.com/joshyorko/room-of-requirement
• provider: Kubernetes
• Kubernetes context: ror
• namespace: devsy
• Dockerless image: ghcr.io/devsy-org/dockerless:0.2.0
• architecture: amd64

Devsy successfully resolves the Dockerfile and the docker-in-docker devcontainer feature, calculates the prebuild hash, creates the PVC/pod, injects the agent, and begins streaming the workspace.

The pod is configured with:

DOCKERLESS_CONTEXT=/workspaces/room-of-requirement
DOCKERLESS_DOCKERFILE=/workspaces/room-of-requirement/.devsy-internal/Dockerfile-with-features
DOCKERLESS_TARGET=dev_containers_target_stage

Then immediately after the workspace copy begins, setup fails with:

Copy .../room-of-requirement into DevContainer /workspaces/room-of-requirement
dockerless build: rename dir: open /workspaces/.dockerless/.devsy-internal: no such file or directory

This reproduces the same failure with a second Dockerfile-backed devcontainer, including a generated Dockerfile-with-features path rather than the Dockerfile-without-features path in the original actions reproducer.

That makes the failure appear independent of the specific Dockerfile contents and independent of whether Devcontainer Features are involved; the common failure point is the generated .devsy-internal directory not surviving the streamed Kubernetes workspace handoff.

[coderabbitai]

Coding Plan

Summary

• Remove the config.BuildArtifactExcludes() exclusion from StreamMount so the streamed workspace mount carries the generated .devsy-internal Dockerless build directory into the Kubernetes pod.
• Keep .devsyignore handling and all other BuildArtifactExcludes() callers unchanged, because .devsy-internal is generated only in the Dockerless flow that needs it.
• Leave host-side and pod-side cleanup unchanged, so generated artifacts are still removed after setup.
• Add a StreamMount regression test that asserts the .devsy-internal build directory is present in the streamed tar.

Design Choices

Design Choice 1: How should the generated `.devsy-internal` build directory reach the pod?

Options Considered:

1. Stop applying config.BuildArtifactExcludes() in StreamMount, so the streamed mount carries .devsy-internal into the pod. This matches the working DevPod control behavior, where the mount stream does not exclude the equivalent internal folder.
2. Keep the exclude in StreamMount, but add a separate step that seeds/persists .devsy-internal into the /workspaces/.dockerless volume before excluding it from the normal payload. This requires a new transport path, because /workspaces/.dockerless is a Type: "volume" mount that is never part of config.GetMounts() and is never streamed.

Chosen Option: 1

Rationale: This option matches the proven DevPod control flow. This option is the minimal change. .devsy-internal is generated only for the Dockerless (Kubernetes/non-Docker) flow, which is exactly the flow that needs the folder in the pod.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Deliver the Dockerless build directory to the pod through the mount stream

This phase makes the streamed workspace mount carry the generated .devsy-internal build directory to the Kubernetes pod. The goal is that prepareBuildDirectory finds the build directory at the workspace context path before DockerlessBuild runs.

Task 1: Stop excluding build artifacts in StreamMount

Remove the build-artifact exclusion from the mount stream so the generated Dockerless build directory reaches the pod.

• Update StreamMount in pkg/agent/tunnelserver/tunnelserver.go. Change its exclude list so it no longer appends config.BuildArtifactExcludes().
• Keep workspaceIgnoreExcludes() in place. The .devsyignore behavior must not change.
• Rationale: .devsy-internal is generated only in the Dockerless (non-Docker driver) build path. The Docker driver never creates this folder. So removing the exclude affects only the Dockerless flow, where the pod requires the folder.
• Add a short code comment that explains why StreamMount must not exclude .devsy-internal. State that the Dockerless build directory travels to the pod through this mount stream.
• Scope note: Do not change StreamWorkspace or StreamSnapshotVolumes in this ticket. The reported failure uses the StreamMount path for the primary workspace mount. Leave the other RPCs and pkg/agent/delivery/workspace_seed.go and pkg/devcontainer/config/prebuild.go unchanged.

Task 2: Confirm host cleanup and pod fallback remain intact

Verify that existing cleanup behavior still runs and that no additional change is required for the fallback path.

• Confirm cleanupBuildInformation (deferred in pkg/devcontainer/run.go Up()) still removes the host .devsy-internal folder after setup. Do not change this behavior.
• Confirm the pod-side cleanupBuildDirectory in pkg/agent/dockerless.go still moves .devsy-internal into /workspaces/.dockerless after the build. Do not change this behavior.
• Confirm config.BuildArtifactExcludes() and config.RemoveBuildArtifacts() in pkg/devcontainer/config/artifacts.go remain unchanged, because other callers (prebuild hash, volume seeding) still need them.

🤖 Prompt for AI agents

Implement the fix that lets the generated Dockerless build directory reach the
Kubernetes pod through the streamed workspace mount. The goal is that
`prepareBuildDirectory` finds the build directory before `DockerlessBuild` runs.

- In `pkg/agent/tunnelserver/tunnelserver.go`, modify `StreamMount`. Remove
`config.BuildArtifactExcludes()` from the excludes list, so `.devsy-internal` is
no longer stripped from the streamed tar.
- Keep `workspaceIgnoreExcludes()` unchanged in `StreamMount`. Do not change
`.devsyignore` behavior.
- Add a short code comment in `StreamMount` that explains why the exclude was
removed: the Dockerless build directory must travel to the pod through this
mount stream.
- Do not modify `StreamWorkspace` or `StreamSnapshotVolumes`. Do not modify
`pkg/agent/delivery/workspace_seed.go` or `pkg/devcontainer/config/prebuild.go`.
- Verify, without code changes, that `cleanupBuildInformation` (deferred in
`pkg/devcontainer/run.go` `Up()`) still removes the host `.devsy-internal`
folder after setup.
- Verify, without code changes, that `cleanupBuildDirectory` in
`pkg/agent/dockerless.go` still moves `.devsy-internal` into
`/workspaces/.dockerless` after the build.
- Verify, without code changes, that `config.BuildArtifactExcludes()` and
`config.RemoveBuildArtifacts()` in `pkg/devcontainer/config/artifacts.go` remain
unchanged, since other callers (prebuild hash, volume seeding) depend on them.

Phase 2: Add regression coverage

This phase adds a test that fails if a future change re-excludes the Dockerless build directory from the mount stream.

Add a StreamMount regression test

Add a test that proves StreamMount includes .devsy-internal in the streamed tar.

• Add the test to pkg/agent/tunnelserver/tunnelserver_test.go. Follow the existing TestStreamSnapshotVolumes_TarsMountTargets pattern.
• Define a fakeStreamMountServer that embeds tunnel.Tunnel_StreamMountServer and collects sent chunks, with Send and Context methods.
• In the test, create a temp dir with t.TempDir(). Add a normal file. Add a config.DevsyContextFeatureFolder subdirectory (.devsy-internal) that contains a file such as Dockerfile-without-features.
• Build a tunnelServer struct literal with mounts set to a single *config.Mount whose Source is the temp dir.
• Call srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()}, fake). Reassemble the tar from the collected chunks. Walk the tar with archive/tar.
• Assert that the .devsy-internal/Dockerfile-without-features entry is present in the tar. This is the core regression assertion.
• Assert that the normal file is also present, to confirm the stream is otherwise complete.
• Use github.com/stretchr/testify/require, consistent with the existing test file.

🤖 Prompt for AI agents

Add a regression test that fails if a future change re-excludes the Dockerless
build directory from the mount stream.

- Add the test to `pkg/agent/tunnelserver/tunnelserver_test.go`, following the
existing `TestStreamSnapshotVolumes_TarsMountTargets` pattern.
- Define `fakeStreamMountServer`, embedding `tunnel.Tunnel_StreamMountServer`,
and collect sent chunks. Implement `Send` and `Context` methods on it.
- In the test, create a temp directory with `t.TempDir()`. Add a normal file to
it. Add a `config.DevsyContextFeatureFolder` subdirectory (`.devsy-internal`)
that contains a file such as `Dockerfile-without-features`.
- Build a `tunnelServer` struct literal. Set `mounts` to a single
`*config.Mount` whose `Source` is the temp directory.
- Call `srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()},
fake)`. Reassemble the tar from the collected chunks. Walk the tar with
`archive/tar`.
- Assert that the `.devsy-internal/Dockerfile-without-features` entry is present
in the tar. This assertion is the core regression check.
- Assert that the normal file is also present in the tar, to confirm the stream
is otherwise complete.
- Use `github.com/stretchr/testify/require`, consistent with the existing test
file.
- Use `config.DevsyContextFeatureFolder` for the folder name in the test instead
of a string literal, after confirming it equals `.devsy-internal`.
- Do not alter `BuildArtifactExcludes()` or its other callers.
- Run `go test ./pkg/agent/tunnelserver/...` and `go test
./pkg/devcontainer/config/...` to confirm no regressions.

Research

The host CLI process resolves the devcontainer config and builds the image. For non-Docker drivers (Kubernetes), dockerlessFallback in pkg/devcontainer/build.go writes .devsy-internal/Dockerfile-without-features under the build context path. This path sits inside the local workspace folder.

The pod agent runs cmd/internal/agentcontainer/setup.go. This process first calls syncMounts, which streams each mount from the host through the StreamMount RPC in pkg/agent/tunnelserver/tunnelserver.go. The process then runs agent.DockerlessBuild (pkg/agent/dockerless.go).

prepareBuildDirectory expects .devsy-internal at <DOCKERLESS_CONTEXT>/.devsy-internal or at the fallback /workspaces/.dockerless/.devsy-internal.

StreamMount currently appends config.BuildArtifactExcludes() (which returns .devsy-internal) to its exclude list. As a result, the only host copy of the build directory is stripped from the tar before it reaches the pod.

Host-side cleanup runs later, at the end of Up(), through a deferred RemoveBuildArtifacts call.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the fix that lets the generated Dockerless build directory reach the
Kubernetes pod through the streamed workspace mount. The goal is that
`prepareBuildDirectory` finds the build directory before `DockerlessBuild` runs.

- In `pkg/agent/tunnelserver/tunnelserver.go`, modify `StreamMount`. Remove
`config.BuildArtifactExcludes()` from the excludes list, so `.devsy-internal` is
no longer stripped from the streamed tar.
- Keep `workspaceIgnoreExcludes()` unchanged in `StreamMount`. Do not change
`.devsyignore` behavior.
- Add a short code comment in `StreamMount` that explains why the exclude was
removed: the Dockerless build directory must travel to the pod through this
mount stream.
- Do not modify `StreamWorkspace` or `StreamSnapshotVolumes`. Do not modify
`pkg/agent/delivery/workspace_seed.go` or `pkg/devcontainer/config/prebuild.go`.
- Verify, without code changes, that `cleanupBuildInformation` (deferred in
`pkg/devcontainer/run.go` `Up()`) still removes the host `.devsy-internal`
folder after setup.
- Verify, without code changes, that `cleanupBuildDirectory` in
`pkg/agent/dockerless.go` still moves `.devsy-internal` into
`/workspaces/.dockerless` after the build.
- Verify, without code changes, that `config.BuildArtifactExcludes()` and
`config.RemoveBuildArtifacts()` in `pkg/devcontainer/config/artifacts.go` remain
unchanged, since other callers (prebuild hash, volume seeding) depend on them.
===============================================================================

Task: 2

Add a regression test that fails if a future change re-excludes the Dockerless
build directory from the mount stream.

- Add the test to `pkg/agent/tunnelserver/tunnelserver_test.go`, following the
existing `TestStreamSnapshotVolumes_TarsMountTargets` pattern.
- Define `fakeStreamMountServer`, embedding `tunnel.Tunnel_StreamMountServer`,
and collect sent chunks. Implement `Send` and `Context` methods on it.
- In the test, create a temp directory with `t.TempDir()`. Add a normal file to
it. Add a `config.DevsyContextFeatureFolder` subdirectory (`.devsy-internal`)
that contains a file such as `Dockerfile-without-features`.
- Build a `tunnelServer` struct literal. Set `mounts` to a single
`*config.Mount` whose `Source` is the temp directory.
- Call `srv.StreamMount(&tunnel.StreamMountRequest{Mount: mount.String()},
fake)`. Reassemble the tar from the collected chunks. Walk the tar with
`archive/tar`.
- Assert that the `.devsy-internal/Dockerfile-without-features` entry is present
in the tar. This assertion is the core regression check.
- Assert that the normal file is also present in the tar, to confirm the stream
is otherwise complete.
- Use `github.com/stretchr/testify/require`, consistent with the existing test
file.
- Use `config.DevsyContextFeatureFolder` for the folder name in the test instead
of a string literal, after confirming it equals `.devsy-internal`.
- Do not alter `BuildArtifactExcludes()` or its other callers.
- Run `go test ./pkg/agent/tunnelserver/...` and `go test
./pkg/devcontainer/config/...` to confirm no regressions.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!