Cannot write (or read?) from temp mounts

[sagikazarmark]

What is the issue?

After writing to a temp mount in a container, the mounted directory shows up as empty, but there is no error indicating that the command writing to the mount fails.

When opening up a terminal in the same container, it works.

Dagger version

dagger v0.11.2 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

 return dag.Container().From("alpine").
  WithUser("root").
  WithMountedTemp("/root/foo").
  WithExec([]string{"sh", "-c", "echo baz > /root/foo/bar"}).
  WithExec([]string{"sh", "-c", "cat /root/foo/bar"})

Log output

cat: can't open '/root/foo/bar': No such file or directory

[sipsma]

This is the expected behavior of temp mounts, they are tmpfs mounts created during each exec. So any writes are ephemeral to just a single exec.

On mobile so need to check the docs later, but I'm guessing they don't make this very clear and should be updated.

What was the use case you had in mind @sagikazarmark for using this instead of a normal mount? We added this API way back in the day mostly because it was something buildkit allowed, but we have a lot more flexibility now and can change it's behavior to better support whatever you had in mind if needed.

[sagikazarmark]

@sipsma it came up here (suggested by @marcosnils) as a way to prevent secrets written to the filesystem making their way to the layer cache.

Tools like Helm (and basically every other tool interacting with OCI, including docker) save credentials to the filesystem (or use a credential helper, but not all tools support that AFAIK).

[sagikazarmark]

More context: I mimic Helm's CLI API in my helm module, so you can do dag.Helm().Login(username, password) which writes the registry config to the filesystem.

I'll probably create the registry config and mount it in the container for now, to make sure it stays secret.

Anyway, I think there are two use cases two consider here:

• Tools writing secrets to the filesystem
• Tools interacting with OCI

Better (ie. first class) support for the latter (auth) would be nice. But I imagine the first one is easier to tackle.

[sipsma]

I see, yeah I get the thinking behind the idea of using tmpfs mounts here but the implementation we're currently inheriting from buildkit is strictly only for mounts that are ephemeral to each exec.

• I do see how we could just implement this ourselves instead, some recent internal changes make that a lot more tractable, but I think there's hopefully better ways of supporting this use case.

More context: I mimic Helm's CLI API in my helm module, so you can do dag.Helm().Login(username, password) which writes the registry config to the filesystem.
I'll probably create the registry config and mount it in the container for now, to make sure it stays secret.

@sagikazarmark Is the registry config just a single file? If so, there is the .WithMountedSecret API that lets you mount secrets as files in such a way that they won't be persisted to disk/cache but also have the "stickiness" across WithExecs that you wanted here.

• You'd need to use SetSecret with the value being the contents of the file and then mount that via .WithMountedSecret

Unfortunately, we don' t yet support mounting secret directories, just single secret files, so if the registry config is a lot of files (or contains files with paths you can't predict) then we'll need to add support for that. But if it's just one or a few predictable files then it sounds like this may work for you already? Let me know.

[sagikazarmark]

I see, yeah I get the thinking behind the idea of using tmpfs mounts here but the implementation we're currently inheriting from buildkit is strictly only for mounts that are ephemeral to each exec.

In the meantime, it would probably make sense to at least document this behavior.

@sagikazarmark Is the registry config just a single file? If so, there is the .WithMountedSecret API that lets you mount secrets as files in such a way that they won't be persisted to disk/cache but also have the "stickiness" across WithExecs that you wanted here.

It is, but I was trying to avoid manually creating it (using Helm's API would have been preferred).

I doubt Helm is going to be the only affected tool. Basically, anything that works with OCI will have the same problem.

[sipsma]

It is, but I was trying to avoid manually creating it (using Helm's API would have been preferred).
I doubt Helm is going to be the only affected tool. Basically, anything that works with OCI will have the same problem.

Oh for sure, I am just trying to figure out what's possible at this moment.

If the creation of that file is available as a library (i.e. from helm, or if it's just the standard docker .config file then probably available somewhere, I can poke around if helpful), then you could use that direct in the module code.

For longer term, we could think about adding some sort of core API functionality here (i.e. give me a valid registry config secret file based on the credentials of the caller session).

Or maybe better yet this just calls for a module in daggerverse (return a secret file for registry config based on the provided username/password secret).

In the meantime, it would probably make sense to at least document this behavior.

Totally agree: #7281

[sagikazarmark]

@sipsma I created this: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/registry-config@5b7f915455413ec91b3463c6b631888c52f283f5

It should be enough for the short term.

For longer term, we could think about adding some sort of core API functionality here (i.e. give me a valid registry config secret file based on the credentials of the caller session).

Yeah, probably....keep an eye on OCI WG Auth, though.