Container KRM Mounts are not mounting via function parameters

[CandiedCode]

Describe the bug
When specifying mounts in the config.kubernetes.io/function container functions, the volumes are not being mounted.

Files that can reproduce the issue

I have a sample repo that I have mount info to the example transformer yaml.

apiVersion: transformers.example.co/v1
kind: ValueAnnotator
metadata:
  name: notImportantHere
  annotations:
    config.kubernetes.io/function: |
      container:
        image: example.docker.com/my-functions/valueannotator:1.0.0   
        mounts:
        - type: bind
          src: /Users/candiedcode/Code/
          dst: /test/
value: 'testing-volume-mounts'

I expect when this runs, if I run a docker inspect on this running container, I will be able to see volume mounts, but I do not.
Running kustomize build --enable-alpha-plugins . and inspecting the container

Expected output
I expected to see mounted volumes similar to when i run this kustomize build --enable-alpha-plugins --mount type=bind,src=${PWD},dst=/candiedcode/ .

 docker inspect 8ea0f93585aa
[
    {
        "Id": "8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e",
        "Created": "2021-11-12T16:47:30.1595402Z",
        "Path": "function",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 223543,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-11-12T16:47:30.5462121Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:fa31f02d46e560dfbf476e28899eba0e607a91070c1d9a0f2a902899fbbb51d2",
        "ResolvConfPath": "/var/lib/docker/containers/8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e/hostname",
        "HostsPath": "/var/lib/docker/containers/8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e/hosts",
        "LogPath": "/var/lib/docker/containers/8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e/8ea0f93585aa3423a57f21c9a09fe1151658eca956270f0fabf875fb2707da8e-json.log",
        "Name": "/crazy_hugle",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "none",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "no-new-privileges"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/Users/candiedcode/Code/kustomize-container-plugin",
                    "Target": "/candiedcode/",
                    "ReadOnly": true
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/aff7b536a3d363608e001440b0b8aeb72d7eb74d72edca9072029882b53cf517-init/diff:/var/lib/docker/overlay2/xtf5wem4k5ct8887jhwsdpc1p/diff:/var/lib/docker/overlay2/0a2512b2ee7ff7d0f0aa247cc73912b626f96666bb356068f2c6aa85eb2d7e6b/diff",
                "MergedDir": "/var/lib/docker/overlay2/aff7b536a3d363608e001440b0b8aeb72d7eb74d72edca9072029882b53cf517/merged",
                "UpperDir": "/var/lib/docker/overlay2/aff7b536a3d363608e001440b0b8aeb72d7eb74d72edca9072029882b53cf517/diff",
                "WorkDir": "/var/lib/docker/overlay2/aff7b536a3d363608e001440b0b8aeb72d7eb74d72edca9072029882b53cf517/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/host_mnt/Users/candiedcode/Code/kustomize-container-plugin",
                "Destination": "/candiedcode",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "8ea0f93585aa",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "LOG_TO_STDERR=true",
                "STRUCTURED_RESULTS=true",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "example.docker.com/my-functions/valueannotator:1.0.0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "function"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "fa6695cc80d4eb2a79367b9f5e250aab6c764025e5f7e174b5d2747e76547a03",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/fa6695cc80d4",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "none": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "a88ac1a2225d9bab786e419555986c3bd6c3d20d69b922bd3c5fe98a3f3a2534",
                    "EndpointID": "b8bb028cf63c3fe75d38a451898bbb02f0841ff86b3efef476150baa5816de96",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Actual output

docker inspect 7695ac4aadc6
[
    {
        "Id": "7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7",
        "Created": "2021-11-12T16:39:04.2170879Z",
        "Path": "function",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 218437,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-11-12T16:39:04.5755526Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:fa31f02d46e560dfbf476e28899eba0e607a91070c1d9a0f2a902899fbbb51d2",
        "ResolvConfPath": "/var/lib/docker/containers/7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7/hostname",
        "HostsPath": "/var/lib/docker/containers/7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7/hosts",
        "LogPath": "/var/lib/docker/containers/7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7/7695ac4aadc612063b4e8998f3ca6a2402850925b6b788e0f39413fa33fbf0c7-json.log",
        "Name": "/beautiful_jackson",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "none",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": true,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "no-new-privileges"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/cb7766dab54ed208f1cff1bd999ad675988965fe1ba2eede65be94d018aa6a7e-init/diff:/var/lib/docker/overlay2/xtf5wem4k5ct8887jhwsdpc1p/diff:/var/lib/docker/overlay2/0a2512b2ee7ff7d0f0aa247cc73912b626f96666bb356068f2c6aa85eb2d7e6b/diff",
                "MergedDir": "/var/lib/docker/overlay2/cb7766dab54ed208f1cff1bd999ad675988965fe1ba2eede65be94d018aa6a7e/merged",
                "UpperDir": "/var/lib/docker/overlay2/cb7766dab54ed208f1cff1bd999ad675988965fe1ba2eede65be94d018aa6a7e/diff",
                "WorkDir": "/var/lib/docker/overlay2/cb7766dab54ed208f1cff1bd999ad675988965fe1ba2eede65be94d018aa6a7e/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "7695ac4aadc6",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": true,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": true,
            "StdinOnce": true,
            "Env": [
                "STRUCTURED_RESULTS=true",
                "LOG_TO_STDERR=true",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "example.docker.com/my-functions/valueannotator:1.0.0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "function"
            ],
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "8f3ebf4cca12764569e77a30ee2ace9ab0c47d77a00d6a5ddb2be5df03723890",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/8f3ebf4cca12",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "none": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "a88ac1a2225d9bab786e419555986c3bd6c3d20d69b922bd3c5fe98a3f3a2534",
                    "EndpointID": "3a1191b35f6e257d6e54548a94061f4dfbd4c5c02ace10afcf8b24960075de60",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

Kustomize version
{Version:kustomize/v4.4.1 GitCommit:b2d65ddc98e09187a8e38adc27c30bab078c1dbf BuildDate:2021-11-11T23:27:14Z GoOs:darwin GoArch:amd64}

Platform
Mac

Additional context

[natasha41575]

If you change the dst to /tmp/test or tmp/test does it show up? I think I remember having a similar problem in the past, and it was because the function runs as the user nobody, who does not have permission to write to any folder except tmp.

[CandiedCode]

Thanks @natasha41575 I'll check. However, I did find this in the roadmap today as well

Deprecate KRM plugin configuration options that promote violations of Kustomize’s policy that everything required for a build should be committed (no side-effects from env, cli flags, etc). All plugin config should be in the KRM config object for the plugin.
Starlark plugins: Deprecate generic URL download for starlark plugins, replacing it with git-specific functionality in line with Kustomization’s own git URL support. Subject the relative path to loader restrictions.
Container plugins: Deprecate network access, storage mount and env options.
Exec plugins: Subject the exec path to loader restrictions.

Which looks like this functionality will be removed in the future or already has?

I tried both /tmp/test and tmp/test in

apiVersion: transformers.example.co/v1
kind: ValueAnnotator
metadata:
  name: notImportantHere
  annotations:
    config.kubernetes.io/function: |
      container:
        image: example.docker.com/my-functions/valueannotator:1.0.0   
        mounts:
        - type: bind
          src: /Users/candiedcode/Code/
          dst: tmp/test/
value: 'testing-volume-mounts'

mounting dst via /candiedcode/ works fine via kustomize build --enable-alpha-plugins --mount type=bind,src=${PWD},dst=/candiedcode/

I did use

kustomize/kyaml/fn/runtime/runtimeutil/runtimeutil_test.go

Lines 1167 to 1223 in 71b978d

	apiVersion: v1beta1
	kind: Example
	metadata:
	annotations:
	config.kubernetes.io/function: |-
	container:
	image: foo:v1.0.0
	mounts: [ {type: bind, src: /mount/path, dst: /local/}, {src: myvol, dst: /local/, type: volume}, {dst: /local/, type: tmpfs} ]
	`,
	expectedFn: `
	container:
	image: foo:v1.0.0
	mounts:
	- type: bind
	src: /mount/path
	dst: /local/
	- type: volume
	src: myvol
	dst: /local/
	- type: tmpfs
	dst: /local/
	`,
	},
	{
	name: "storage mounts yaml style",
	resource: `
	apiVersion: v1beta1
	kind: Example
	metadata:
	annotations:
	config.kubernetes.io/function: |-
	container:
	image: foo:v1.0.0
	mounts:
	- src: /mount/path
	type: bind
	dst: /local/
	- dst: /local/
	src: myvol
	type: volume
	- type: tmpfs
	dst: /local/
	`,
	expectedFn: `
	container:
	image: foo:v1.0.0
	mounts:
	- type: bind
	src: /mount/path
	dst: /local/
	- type: volume
	src: myvol
	dst: /local/
	- type: tmpfs
	dst: /local/
	`,

as a guide and created a new unit test for my scenario and it passed as expected.

[natasha41575]

I will talk to the other maintainers when we refresh the roadmap for 2022. The functionality has not been removed yet, and I don't have the context for why there was a plan to remove network access and storage mounts. I can leave a comment with updates after discussion.

[CandiedCode]

Thanks @natasha41575 I updated my previous comments with testing your suggestion as well.

[aodinokov]

I will talk to the other maintainers when we refresh the roadmap for 2022. The functionality has not been removed yet, and I don't have the context for why there was a plan to remove network access and storage mounts. I can leave a comment with updates after discussion.

@natasha41575 , here is some idea why it was dropped:
kptdev/kpt#1151
In particular - this line disables that :) :
https://github.com/kubernetes-sigs/kustomize/blob/master/kyaml/fn/runtime/runtimeutil/functiontypes.go#L213
To me the possibility to mount global path declaratively seems dangerous.
Since some time ago I liked the idea of declarative mount, at some point I proposed this #3336 that allowed only relative paths (if I remember correctly I also wanted to check that that path is inside the original path provided to build cmd), but decided to close that since hasn't got any response.

UPD:
yes, the PR 3336 checks if relative path is in scope: https://github.com/kubernetes-sigs/kustomize/pull/3336/files#diff-76e859f6c4057a1daae30f139e37c8ee1ba6ab515f7dc6a84eae4cf0175ab98fR217

PS: it's a separate story how to make it work with Windows. 3336 only makes sure that UT work for Windows, but perhaps it's necessary to state that relative path will always be using 'linux-style' separators and in case of different platform convert path to the current style. BTW I think different separators was something mentioned as 'lesser extend portability concerns'...

[natasha41575]

/triage accepted

[natasha41575]

I think #4489 will fix the issue of files not being correctly mounted into the container.

And as far as support goes, https://github.com/kubernetes/enhancements/tree/master/keps/sig-cli/2953-kustomize-plugin-graduation has been updated to state the following:

The network and storage mount options will be retained, as they are needed to support some common generator plugin use cases. Guidance on their appropriate use will be added to the plugin developer documentation.

To me the possibility to mount global path declaratively seems dangerous.

I agree, IMO we should (a) deprecate the --mounts flag and (b) limit mounts to be relative paths. This idea isn't really fleshed out and probably deserves its own issue. I can file one when I have some more time to think about it.