Linux containers on FreeBSD #7000

akhramov · 2022-05-29T13:17:42Z

This allows running Linux containers on FreeBSD and modifies the
mounts so that they represent the linux emulated filesystems, as per:
https://wiki.freebsd.org/LinuxJails

Supersedes #5480 per convo with @gizahNL in OCI Slack chat

k8s-ci-robot · 2022-05-29T13:17:51Z

Hi @akhramov. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

samuelkarp · 2022-05-29T23:39:52Z

oci/spec_opts.go

+		//FreeBSD has support for running Linux binaries via syscall redirection, while that would be enough
+		//for some binaries others depend on emulated linux file systems (procfs & sysfs).
+		//So when we are running FreeBSD and our image OS is Linux we modify our mounts to mount the emulated
+		//filesystems.
+		//We do it here since here we have access to both the OCI spec & the image OS.
+		if runtime.GOOS == "freebsd" && ociimage.OS == "linux" {
+			freebsdLinuxEmulationMounts(s)
+		}


Carrying over the comment I left on the previous PR:

I'd like to see this extracted out into a function like appendOSMounts, which is then separately defined in each of spec_opts_freebsd.go/spec_opts_linux.go/spec_opts_windows.go with whatever content is appropriate there. The Linux one would likely be empty, but I imagine there could be some use for this on Windows with the LCOW/WCOW work.

In addition to providing a possibly-reusable abstraction for FreeBSD and Windows to achieve similar goals, an advantage is keeping FreeBSD-specific filesystems like "linprocfs" and "linsysfs" out of the common file that is compiled for all operating systems.

I'd also like to see a WithLinuxEmulationMounts functional option that clients of containerd can use without having to use WithImageConfig/WithImageConfigArgs or even having an image with a declared OS of "linux".

If you're not interested in doing that work, I can pick up and carry this PR.

I'd like to see this extracted out into a function like appendOSMounts

That makes a lot of sense, implemented. As for WithLinuxEmulationMounts I'm not that familiar with the codebase yet, so I would defer it to you

That works for me, I can pick up WithLinuxEmulationMounts in a follow-up PR.

samuelkarp · 2022-05-29T23:41:55Z

platforms/defaults_freebsd.go

+		// The Variant field will be empty if arch != ARM.
+		Variant: cpuVariant(),
+	})
+}


nit: can you add a newline at the end of this file?

Nice catch, thanks!

samuelkarp · 2022-05-31T07:47:13Z

oci/spec_opts.go

@@ -377,6 +377,7 @@ func WithImageConfigArgs(image Image, args []string) SpecOpts {
 			return fmt.Errorf("unknown image config media type %s", ic.MediaType)
 		}

+		appendOSMounts(s, ociimage.OS)


Looks like the linter is failing here because it's also attempting to lint & compile on MacOS (darwin), where this new function is not defined. My apologies for not realizing that in my original suggestion, but I think you can fix this by either (a) adding a new spec_opts_notfreebsd.go and moving the definition for everything not-FreeBSD there (including Windows and Linux) or (b) additionally adding a spec_opts_darwin.go for MacOS.

samuelkarp

Thanks for continuing to work with me on this! I've got a few more questions and suggestions after looking at the code a bit more closely.

oci/spec_opts_freebsd.go

samuelkarp · 2022-05-31T17:31:25Z

oci/spec_opts_freebsd.go

+			Destination: "/proc",
+			Type:        "linprocfs",
+			Source:      "linprocfs",
+			Options:     []string{},


In the in-place case above, we're adding "nosuid" and "noexec" as options, but in this case we're not. Is there any particular reason for the difference?

No reason, we should have nosuid & noexec in both ases.

samuelkarp · 2022-05-31T17:32:12Z

oci/spec_opts_freebsd.go

+	mounts = append(mounts, specs.Mount{
+		Destination: "/sys",


Should we handle /sys in-place like we're doing for /proc and /dev/fd too?

I agree, but that is less relevant with the new approach I have taken. I will implement this if the approach does not work.

samuelkarp · 2022-05-31T17:32:45Z

oci/spec_opts_freebsd.go

+
+	if !haveDevFd {
+		mounts = append(mounts, specs.Mount{
+			Destination: "/dev/fd",


Should we also be adding devfs for /dev?

I suspect we already have both from the default mounts:

containerd/oci/mounts_freebsd.go

Lines 23 to 38 in 78cd9d3

func defaultMounts() []specs.Mount {

return []specs.Mount{

{

Destination: "/dev",

Type: "devfs",

Source: "devfs",

Options: []string{"ruleset=4"},

},

{

Destination: "/dev/fd",

Type: "fdescfs",

Source: "fdescfs",

Options: []string{},

},

}

}

akhramov · 2022-05-31T22:51:09Z

@samuelkarp I drastically simplified the code. Let me know if I'm missing something. The reasoning is below.

We should already have /dev & /dev/fd as default mounts:

containerd/oci/mounts_freebsd.go

Lines 23 to 38 in 78cd9d3

    
           func defaultMounts() []specs.Mount { 
        
           	return []specs.Mount{ 
        
           		{ 
        
           			Destination: "/dev", 
        
           			Type:        "devfs", 
        
           			Source:      "devfs", 
        
           			Options:     []string{"ruleset=4"}, 
        
           		}, 
        
           		{ 
        
           			Destination: "/dev/fd", 
        
           			Type:        "fdescfs", 
        
           			Source:      "fdescfs", 
        
           			Options:     []string{}, 
        
           		}, 
        
           	} 
        
           }

We need to provide /sys, /proc, and /dev/shm per https://wiki.freebsd.org/LinuxJails and https://github.com/opencontainers/runtime-spec/blob/1c3f411f041711bbeecf35ff7e93461ea6789220/config-linux.md#default-filesystems

/sys and /proc are backed by their corresponding file-systems, which I include in appendOSMounts

/dev/shm is backed by the linuxulator kernel module (albeit the device is not whitelisted by devfs ruleset 4, but that's a separate issue).

This allows running Linux containers on FreeBSD and modifies the mounts so that they represent the linux emulated filesystems, as per: https://wiki.freebsd.org/LinuxJails Co-authored-by: Gijs Peskens <gijs@peskens.net>, Samuel Karp <samuelkarp@users.noreply.github.com> Signed-off-by: Artem Khramov <akhramov@pm.me>

kzys · 2022-06-01T18:33:03Z

platforms/defaults_freebsd.go

+// Default returns the default matcher for the platform.
+func Default() MatchComparer {
+	return Ordered(DefaultSpec(), specs.Platform{
+		OS:           "linux",


Because most containers image are for Linux today?

This is ordered with default (FreeBSD) first and Linux as a fallback.

akhramov · 2022-06-04T16:57:31Z

Alright, CI seems green 🎉 It seems like we are good to go.

@samuelkarp is there any items I missed?

samuelkarp · 2022-06-07T01:41:57Z

@akhramov The previous approach modified the /proc and /dev/fd mounts in-place if they existed rather than appending them. Did you change to append unconditionally because you'd expect the mounts at the end of the spec to hide any previous mount to those mountpoint and the hidden mount is irrelevant?

samuelkarp · 2022-06-07T04:25:45Z

Aside from the question about mounts, I've tested this and it works!

$ sudo ctr run --rm --runtime wtf.sbk.runj.v1 --tty --snapshotter zfs docker.io/library/alpine:latest test sh -c 'cat /etc/os-release && uname -a'
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.0
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
Linux 3.17.0 FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC x86_64 Linux

akhramov · 2022-06-07T04:37:27Z

@samuelkarp

Did you change to append unconditionally because you'd expect the mounts at the end of the spec to hide any previous mount to those mountpoint and the hidden mount is irrelevant?

Not really, but it is up to the runtime (us) to create mountpoints at these locations. Since we don't have these as default mountpoints, I don't expect them to exist at all. Uh, I hope I'm not missing anything.

samuelkarp · 2022-06-07T04:45:59Z

Since we don't have these as default mountpoints, I don't expect them to exist at all.

Mounts can be arbitrarily defined in the runtime configuration which is an argument to the functional option returned by WithImageConfigArgs (see the signature func(ctx context.Context, client Client, c *containers.Container, s *Spec) error; the s *Spec is an arbitrary configuration which may already have mounts defined). (It's worth keeping in mind that containerd's primary interface/first-class interface is the thick client library of which this is a part, which then interacts with the daemon over a gRPC API. Clients like ctr and firecracker-containerd and so forth then use this library.)

akhramov · 2022-06-07T04:49:59Z

@samuelkarp got it, thanks. So what should happen if a client overrides a default mountpoint?

samuelkarp · 2022-06-07T04:54:43Z

A client can choose which functional options to apply (including ones it supplies itself) and can also directly manipulate the object; ultimately the *Spec at the end of those functional options being applied is what gets sent to the underlying runtime (typically runc on Linux and runj on FreeBSD). The question here is: what behavior is appropriate for this functional option? I think it's probably fine for it to append-only and cause any prior mounts at that mountpoint to be shadowed (and on Linux I'd be reasonably confident of the behavior), but I'm not sure if there are footguns I don't know about on FreeBSD (and I think you know more than me). Specific areas I'd be concerned about include performance and what the behavior might be if a shadowed mount was a network filesystem that went away during the lifetime of the container.

akhramov · 2022-06-08T15:42:54Z

@samuelkarp I think the shadowing behavior is pretty much the same on FreeBSD as on Linux.

@emaste, can you kindly confirm?

samuelkarp

Thanks @akhramov! If the shadowing behavior is roughly the same, I think we can go ahead and merge this and adjust to the more-complicated replace-in-place logic if we discover it being a problem.

LGTM!

emaste · 2022-06-08T22:04:08Z

@samuelkarp I think the shadowing behavior is pretty much the same on FreeBSD as on Linux.

@emaste, can you kindly confirm?

That is my expectation. I asked some folks who would know for certain, but haven't heard back.

samuelkarp · 2022-06-09T01:07:23Z

@emaste Thanks!

@akhramov and @gizahNL Thank you so much for working on this and pushing through! Now that containerd/project#90 is approved I'll go ahead and merge this myself. 😄

Obtained from: containerd/containerd#7000

k8s-ci-robot added the needs-ok-to-test label May 29, 2022

akhramov mentioned this pull request May 29, 2022

Freebsd linux containers #5480

Closed

akhramov force-pushed the freebsd_linux_containers branch from 0e57bf3 to 6726a95 Compare May 29, 2022 14:17

samuelkarp reviewed May 29, 2022

View reviewed changes

akhramov force-pushed the freebsd_linux_containers branch from 6726a95 to 6487912 Compare May 30, 2022 21:53

samuelkarp reviewed May 31, 2022

View reviewed changes

akhramov force-pushed the freebsd_linux_containers branch 2 times, most recently from 92f180e to e72aeaf Compare May 31, 2022 07:58

samuelkarp reviewed May 31, 2022

View reviewed changes

akhramov force-pushed the freebsd_linux_containers branch from a5163d0 to b9fcb18 Compare May 31, 2022 22:41

akhramov force-pushed the freebsd_linux_containers branch from b9fcb18 to ae22854 Compare May 31, 2022 22:56

kzys approved these changes Jun 1, 2022

View reviewed changes

dmcgowan added this to Ready For Review in Code Review Jun 1, 2022

samuelkarp approved these changes Jun 8, 2022

View reviewed changes

samuelkarp added ok-to-test and removed needs-ok-to-test labels Jun 8, 2022

samuelkarp merged commit 2b4b0cf into containerd:main Jun 9, 2022

Code Review automation moved this from Ready For Review to Done Jun 9, 2022

samuelkarp added impact/changelog kind/feature platform/freebsd FreeBSD labels Jun 9, 2022

freebsd-git pushed a commit to freebsd/freebsd-ports that referenced this pull request Jun 11, 2022

sysutils/containerd: Add patch to support Linux containers on FreeBSD

fa5e3e5

Obtained from: containerd/containerd#7000

emaste mentioned this pull request Jul 29, 2022

FreeBSD, linux container, mount lin* filesystems instead of native BSD filesystems #5432

Closed

kbruner mentioned this pull request Sep 7, 2022

Networking samuelkarp/runj#20

Closed

dmcgowan added this to the 1.7 milestone Oct 20, 2022

dmcgowan mentioned this pull request Dec 10, 2022

Prepare release notes for v1.7.0-beta.1 #7793

Merged

thaJeztah mentioned this pull request Dec 27, 2022

oci: appendOSMounts(): remove unused error, and move #7874

Merged

dmcgowan mentioned this pull request Jan 3, 2023

Prepare release notes for v1.7.0-beta.2 #7903

Merged

dmcgowan mentioned this pull request Jan 28, 2023

Prepare release notes for v1.7.0-beta.3 #8014

Merged

dmcgowan mentioned this pull request Feb 15, 2023

Prepare release notes for v1.7.0 beta.4 #8110

Merged

This was referenced Feb 25, 2023

Prepare release notes for v1.7.0-rc.0 #8162

Merged

Prepare release notes for v1.7.0-rc.1 #8192

Merged

This was referenced Mar 7, 2023

Add release notes for v1.7.0-rc.2 #8216

Merged

Prepare release notes for v1.7.0-rc.3 #8233

Merged

Prepare release notes for v1.7.0 #8242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux containers on FreeBSD #7000

Linux containers on FreeBSD #7000

akhramov commented May 29, 2022

k8s-ci-robot commented May 29, 2022

samuelkarp May 29, 2022

akhramov May 30, 2022

samuelkarp May 31, 2022

samuelkarp May 29, 2022

akhramov May 30, 2022

samuelkarp May 31, 2022

samuelkarp left a comment

samuelkarp May 31, 2022

akhramov May 31, 2022

samuelkarp May 31, 2022

akhramov May 31, 2022

samuelkarp May 31, 2022

akhramov May 31, 2022

akhramov commented May 31, 2022

kzys Jun 1, 2022

samuelkarp Jun 1, 2022

akhramov commented Jun 4, 2022

samuelkarp commented Jun 7, 2022

samuelkarp commented Jun 7, 2022

akhramov commented Jun 7, 2022

samuelkarp commented Jun 7, 2022 •

edited

akhramov commented Jun 7, 2022

samuelkarp commented Jun 7, 2022

akhramov commented Jun 8, 2022

samuelkarp left a comment

emaste commented Jun 8, 2022

samuelkarp commented Jun 9, 2022

	func defaultMounts() []specs.Mount {
	return []specs.Mount{
	{
	Destination: "/dev",
	Type: "devfs",
	Source: "devfs",
	Options: []string{"ruleset=4"},
	},
	{
	Destination: "/dev/fd",
	Type: "fdescfs",
	Source: "fdescfs",
	Options: []string{},
	},
	}
	}

Linux containers on FreeBSD #7000

Linux containers on FreeBSD #7000

Conversation

akhramov commented May 29, 2022

k8s-ci-robot commented May 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelkarp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhramov commented May 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhramov commented Jun 4, 2022

samuelkarp commented Jun 7, 2022

samuelkarp commented Jun 7, 2022

akhramov commented Jun 7, 2022

samuelkarp commented Jun 7, 2022 • edited

akhramov commented Jun 7, 2022

samuelkarp commented Jun 7, 2022

akhramov commented Jun 8, 2022

samuelkarp left a comment

Choose a reason for hiding this comment

emaste commented Jun 8, 2022

samuelkarp commented Jun 9, 2022

samuelkarp commented Jun 7, 2022 •

edited