many: add support for /home on NFS #3958

Merged
merged 62 commits into from Oct 23, 2017

Conversation

Projects
None yet
6 participants
Contributor

zyga commented Sep 25, 2017

This branch adds support for systems running on hosts with NFS-shared /home (or
fragment of /home).

The main problem is that NFS is not transparent to apparmor. When a process
attempts to access NFS-mounted files or directories it will be subject to
network mediation. In the future the kernel may understand what is going on
better and no longer require any workarounds but for now those are necessary.

With this branch, on startup only, snapd will interrogate mounted filesystems
and if NFS is found it will insert additional apparmor snippet into every
apparmor profile. This includes snap application profiles as well as the
profile of snap-confine itself.

The additional profile rules allow "network inet" and "network inet6" access.
For now we are unable to detect NFS filesystems mounted "in flight" at runtime
but we do analyze /etc/fstab and if statically declared NFS filesystem is found
there we assume it may be mounted in the future, even if it is not necessarily
mounted at the given time.

Fixes: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552
Signed-off-by: Zygmunt Krynicki zygmunt.krynicki@canonical.com

zyga added some commits Sep 20, 2017

interfaces: allow backends to initialize
This patch adds a dummy Initialize method to all security backends.
In the future certain backends will perform one-off initialization to
ensure they can be used correctly. Currently that is done in a hackish
way by abusing Setup. This is slightly incorrect as it relies on us
actually doing something that causes snap-specific security setup to be
required. We want to decouple setting up snap security and setting up
security systems so that each responsibility is clear.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
overlord/ifacestate: initialize security backends on startup
This patch makes the interface manager initialize security backends
before they are used. This will be used to allow the apparmor backend to
initialize and perform some operations on startup.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
dirs: add directory representing local apparmor policy for snap-confine
The apparmor profile for snap-confine now includes an include directive
that causes apparmor parser to recursively load an compile any other
policy files that are present in a specified directory.

We want to refer to that directory from snapd hence a new variable.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add apparmor snippet for systems using NFS
The snippet is needed when NFS is in use on the system. Unfortunately
kernel is not hiding this fact and confined processes need access to
generic network operations.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: generate local policy for snap-confine if using NFS
This patch ties the sequence together. On startup snapd will inspect
the system to see if there are any NFS mount points. If so apparmor
profile for snap-confine

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: log presence of extra NFS permissions
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: inject NFS snippet if anything uses NFS
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

What about reexec?

There is also 'nfs'. I wonder if we should include 'cifs' and 'smbfs'....

zyga added some commits Sep 22, 2017

interfaces/apparmor: treat nfs and nfs4 the same way
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: constrain NSF support to /home
This is an artificial restriction but it matches our desired support
plan where /home may be NFS mounted and that is expected to work.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add a TODO
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: don't load s-c profile if reexecing
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: make AnythingUsesNfs testable and test it
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: mock away potential NFS influence during testing
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: rename anythingUsesNfs to isHomeUsingNFS
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: make setupSnapConfineGeneratedPolicy testable
This patch adds the ability to mock access to /proc/self/exe

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: create local policy directory if missing
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add tests for SetupSnapConfineGeneratedPolicy
This patch adds tests for several combinations of NFS and re-execution.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: create policy directory earlier
This ensures that in case NFS interrogation fails we still end up with
the local directory that may not be present in packaging.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add more unit tests
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add tests for Backend.Intialize
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: skip one test running as root
That test relies on DAC checks and those don't work for root :)

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: detect potentially mounted NFS via fstab
This patch extends the currently implemented checks that look at what
is actually mounted to include what may be mounted in the future. This
should ensure good coverage for real world systems in absence of active
notification to mount events.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: re-factor NFS tests for brevity
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Contributor

zyga commented Sep 25, 2017

@jdstrand I've updated this based on your earlier review on IRC. This includes changes to:

  • detect re-exec and skip distribution profile reload
  • detect possible NFS mounts by looking at /etc/fstab (in addition to actual mounts)
  • constrain the search to /home/ or subdirectories
  • detect "nfs" in addition to "nfsv4"

In addition to this I added full unit test coverage and I'll re-create the spread test I lost with my snapd checkout last week.

codecov-io commented Sep 25, 2017

Codecov Report

Merging #3958 into master will increase coverage by <.01%.
The diff coverage is 75.86%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #3958      +/-   ##
=========================================
+ Coverage    75.8%   75.8%   +<.01%     
=========================================
  Files         433     433              
  Lines       37174   37287     +113     
=========================================
+ Hits        28180   28266      +86     
- Misses       7022    7046      +24     
- Partials     1972    1975       +3
Impacted Files Coverage Δ
interfaces/mount/backend.go 58.33% <0%> (-2.54%) ⬇️
overlord/ifacestate/helpers.go 60.26% <0%> (-1.23%) ⬇️
interfaces/kmod/backend.go 74.07% <0%> (-2.85%) ⬇️
interfaces/systemd/backend.go 53.84% <0%> (-1.22%) ⬇️
interfaces/ifacetest/backend.go 0% <0%> (ø) ⬆️
interfaces/seccomp/backend.go 75% <0%> (-1.6%) ⬇️
interfaces/dbus/backend.go 60% <0%> (-1.65%) ⬇️
interfaces/udev/backend.go 75.36% <0%> (-2.25%) ⬇️
dirs/dirs.go 98.13% <100%> (ø) ⬆️
interfaces/apparmor/backend.go 78.59% <91.57%> (+7.86%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66efd05...8107d45. Read the comment docs.

@zyga zyga added this to the 2.29 milestone Sep 25, 2017

Contributor

zyga commented Sep 25, 2017

Ohhh, it's green on first push :-) Must be my lucky day

zyga added some commits Sep 28, 2017

tests: add spread test for NFS mounted home
This patch ensures that we can really run snaps on top of NFS-mounted
home. The test checks both "really mounted" and "may to be mounted"
configurations that represent real-world scenarios.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: correct NFS-on-/home check
The test didn't capture all of /home being exported, it now does.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
tests: use rot13 to hide program from spellchecker
Our spell checker dislikes "exportfs" so ... use rot13 to hide the real
name. /o\

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

Thanks for working on this! Some early first-pass comments.

interfaces/apparmor/backend.go
+ policy := make(map[string]*osutil.FileState)
+
+ // Create the local policy directory if it is not there.
+ if err := os.MkdirAll(dir, 0755); err != nil {
@stolowski

stolowski Oct 3, 2017

Contributor

I'd use dirs.SnapConfineAppArmorDir directly, no need for an intermediate temp variable in a simple case like this.

@zyga

zyga Oct 3, 2017

Contributor

Ack, done.

+ // Check if NFS is mounted at or under $HOME. Because NFS is not
+ // transparent to apparmor we must alter our profile to counter that and
+ // allow snap-confine to work.
+ if nfs, err := isHomeUsingNFS(); err != nil {
@stolowski

stolowski Oct 3, 2017

Contributor

I wonder if an inability to determine if we are using NFS here should be fatal (it's fatal in this code if I'm not mistaken), unless we've full faith in our fstab/mountinfo parsing (which we probably need to have given that it's used in some other places afair). Anyway, just mentioning the concern since NFS is not a typical use case - perhaps a warning would be enough. For your consideration.

@zyga

zyga Oct 3, 2017

Contributor

Interesting question. While we don't expect any errors I agree we should stop the world about it. It is just a workaround for a missing kernel feature (NFS transparency in LSMs). I'll make this non fatal.

@zyga

zyga Oct 3, 2017

Contributor

Done now.

@stolowski

stolowski Oct 5, 2017

Contributor

Thank you!

interfaces/apparmor/backend.go
+ return false, fmt.Errorf("cannot parse /proc/self/mountinfo, %s", err)
+ }
+ for _, entry := range mountinfo {
+ if (entry.FsType == "nfs4" || entry.FsType == "nfs") && (strings.HasPrefix(entry.MountDir, "/home/") || entry.MountDir == "/home") {
@stolowski

stolowski Oct 3, 2017

Contributor

I know we've a deeper and historical problem with hardcoded /home, but I'm not super happy about adding another hardcoded case here. Actually, if we ever fix the problem with hardcoded /home in dirs.go, this fix will break completely and this makes me a sad panda...
Would it be possible to make it more generic and future-proof (e.g. check if any of the filesystems that the snap may potentially be interested in is NFS)? Don't we get the same problem if - say - /usr or /var is NFS-mounted?

@zyga

zyga Oct 3, 2017

Contributor

This was added by a request from @jdstrand - the idea is that we want to slowly open this up to more use cases. For now we just want to support /home on NFS. We plan to expand this to CIFS as well but not in arbitrary places yet.

Note that /home is already explicitly described in apparmor profiles and there's no easy way around it today.

@stolowski

stolowski Oct 5, 2017

Contributor

Fair enough, thanks.

@jdstrand

jdstrand Oct 5, 2017

Contributor

To check for home or not is a difficult question. If we consider that we are lessening the security of the system if NFS is detected, then I think pursuing only doing it when needed is worthwhile. For example, if there is an NFS mount in /srv for something, snaps don't have access to that anyway, so adding network rules to everything snappy for something no snaps will access is wrong. Determining exactly when to apply NFS rules might take some effort, but I think if we are thoughtful and open to iterating, we can do it without too much complexity.

Note that /home is not actually hard-coded in AppArmor policy as included in snappy for app snaps, it is defined via an apparmor tunable which the user can adjust and the apparmor policy for app snaps will be ok.

We do hardcode /home in snap-confine code and apparmor policy though. Until this is resolved, there is no requirement to adjust this PR. If you wanted to make this PR dynamic today, you could do the equivalent of getent passwd, looking at every uid >= 1000, collecting the dirname's of the found homes and then iterate through those to check if on NFS. Of course, this wouldn't catch someone who has a symlink from /home/foo to /var/exports/homes/foo, so could readlink() first. Alternatively, perhaps in the future the core snap can gain 'snap set' config for adding additional locations for HOME (which would update the apparmor tunables). If so, this code could be updated to check /home and any additional directories that were set with snap set.

@@ -0,0 +1,96 @@
+summary: Test that snaps still work when /home is a NFS mount
@stolowski

stolowski Oct 3, 2017

Contributor

❤️

@zyga

zyga Oct 3, 2017

Contributor

As things go, this test obviously caught a bug in the initial code that worked fine for /home sub-directories but not the full thing :)

zyga added some commits Oct 3, 2017

interfaces/apparmor: use dirs.SnapConfineAppArmorDir directly
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: don't fail if we cannot determine NFS status
It is safer to just assume it's off and not block snapd startup.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

@zyga zyga requested review from jdstrand and chipaca Oct 5, 2017

Looks good to me. Thanks for making the changes and for extensive tests!

+ // Check if NFS is mounted at or under $HOME. Because NFS is not
+ // transparent to apparmor we must alter our profile to counter that and
+ // allow snap-confine to work.
+ if nfs, err := isHomeUsingNFS(); err != nil {
@stolowski

stolowski Oct 3, 2017

Contributor

I wonder if an inability to determine if we are using NFS here should be fatal (it's fatal in this code if I'm not mistaken), unless we've full faith in our fstab/mountinfo parsing (which we probably need to have given that it's used in some other places afair). Anyway, just mentioning the concern since NFS is not a typical use case - perhaps a warning would be enough. For your consideration.

@zyga

zyga Oct 3, 2017

Contributor

Interesting question. While we don't expect any errors I agree we should stop the world about it. It is just a workaround for a missing kernel feature (NFS transparency in LSMs). I'll make this non fatal.

@zyga

zyga Oct 3, 2017

Contributor

Done now.

@stolowski

stolowski Oct 5, 2017

Contributor

Thank you!

interfaces/apparmor/backend.go
+ return false, fmt.Errorf("cannot parse /proc/self/mountinfo, %s", err)
+ }
+ for _, entry := range mountinfo {
+ if (entry.FsType == "nfs4" || entry.FsType == "nfs") && (strings.HasPrefix(entry.MountDir, "/home/") || entry.MountDir == "/home") {
@stolowski

stolowski Oct 3, 2017

Contributor

I know we've a deeper and historical problem with hardcoded /home, but I'm not super happy about adding another hardcoded case here. Actually, if we ever fix the problem with hardcoded /home in dirs.go, this fix will break completely and this makes me a sad panda...
Would it be possible to make it more generic and future-proof (e.g. check if any of the filesystems that the snap may potentially be interested in is NFS)? Don't we get the same problem if - say - /usr or /var is NFS-mounted?

@zyga

zyga Oct 3, 2017

Contributor

This was added by a request from @jdstrand - the idea is that we want to slowly open this up to more use cases. For now we just want to support /home on NFS. We plan to expand this to CIFS as well but not in arbitrary places yet.

Note that /home is already explicitly described in apparmor profiles and there's no easy way around it today.

@stolowski

stolowski Oct 5, 2017

Contributor

Fair enough, thanks.

@jdstrand

jdstrand Oct 5, 2017

Contributor

To check for home or not is a difficult question. If we consider that we are lessening the security of the system if NFS is detected, then I think pursuing only doing it when needed is worthwhile. For example, if there is an NFS mount in /srv for something, snaps don't have access to that anyway, so adding network rules to everything snappy for something no snaps will access is wrong. Determining exactly when to apply NFS rules might take some effort, but I think if we are thoughtful and open to iterating, we can do it without too much complexity.

Note that /home is not actually hard-coded in AppArmor policy as included in snappy for app snaps, it is defined via an apparmor tunable which the user can adjust and the apparmor policy for app snaps will be ok.

We do hardcode /home in snap-confine code and apparmor policy though. Until this is resolved, there is no requirement to adjust this PR. If you wanted to make this PR dynamic today, you could do the equivalent of getent passwd, looking at every uid >= 1000, collecting the dirname's of the found homes and then iterate through those to check if on NFS. Of course, this wouldn't catch someone who has a symlink from /home/foo to /var/exports/homes/foo, so could readlink() first. Alternatively, perhaps in the future the core snap can gain 'snap set' config for adding additional locations for HOME (which would update the apparmor tunables). If so, this code could be updated to check /home and any additional directories that were set with snap set.

chipaca approved these changes Oct 5, 2017

Thank you for this.

Could you add a test using proto=udp?

Contributor

zyga commented Oct 5, 2017

@chipaca oh, great idea. I'll do that.

This is looking really good. A few questions, comments, nit-picks inline.

interfaces/apparmor/backend.go
+// Initialize prepares customized apparmor policy for snap-confine.
+func (b *Backend) Initialize() error {
+ // TODO: also generate policy for snap-confine from core.
+ return setupSnapConfineGeneratedPolicy()
@jdstrand

jdstrand Oct 5, 2017

Contributor

Based on a comment further down below, I think this comment can be removed?

@zyga

zyga Oct 5, 2017

Contributor

I was about to say "but I really want to do that eventually" and then I realized it needs to be the way it is for now. I'll change this comment to explain why.

interfaces/apparmor/backend.go
+
+ // If snapd is executing from the core snap the it means it has
+ // re-executed. In that case we are no longer using the copy of
+ // snap-confined from the host distribution but our own copy. We don't have
@jdstrand

jdstrand Oct 5, 2017

Contributor

s/snap-confined/snap-confine/

@zyga

zyga Oct 5, 2017

Contributor

Fixed

interfaces/apparmor/backend.go
+ // setupSnapConfineReexec below.
+ exe, err := os.Readlink(procSelfExe)
+ if err != nil {
+ return fmt.Errorf("cannot read /proc/self/exe, %s", err)
@jdstrand

jdstrand Oct 5, 2017

Contributor

You are using procSelfExe in Readlink() but hardcoding /proc/self/exe in Errorf.

@zyga

zyga Oct 5, 2017

Contributor

Done

interfaces/apparmor/backend.go
+ // load any of the files placed in /var/lib/snapd/apparmor/snap-confine.d/.
+ // We are not using apparmor.LoadProfile() because it uses other cache.
+ if output, err := exec.Command("apparmor_parser", "--replace",
+ // TODO: the name varies across distros, fix that :/
@jdstrand

jdstrand Oct 5, 2017

Contributor

I think this TODO needs to be resolved before merging, no?

@zyga

zyga Oct 5, 2017

Contributor

Yes, I'll address this next.

@zyga

zyga Oct 5, 2017

Contributor

Done.

interfaces/apparmor/backend.go
+ if output, err := exec.Command("apparmor_parser", "--replace",
+ // TODO: the name varies across distros, fix that :/
+ "--write-cache", filepath.Join(dirs.SystemApparmorDir, "usr.lib.snapd.snap-confine.real"),
+ "--cache-loc", dirs.SystemApparmorCacheDir).CombinedOutput(); err != nil {
@jdstrand

jdstrand Oct 5, 2017

Contributor

While not as important here since we shouldn't be reloading this policy all the time, I think you want '"-O", "no-expr-simplify"'. See LoadProfile() in interfaces/apparmor/apparmor.go for details.

@zyga

zyga Oct 5, 2017

Contributor

Done

interfaces/apparmor/backend.go
+ return false, fmt.Errorf("cannot parse /proc/self/mountinfo, %s", err)
+ }
+ for _, entry := range mountinfo {
+ if (entry.FsType == "nfs4" || entry.FsType == "nfs") && (strings.HasPrefix(entry.MountDir, "/home/") || entry.MountDir == "/home") {
@stolowski

stolowski Oct 3, 2017

Contributor

I know we've a deeper and historical problem with hardcoded /home, but I'm not super happy about adding another hardcoded case here. Actually, if we ever fix the problem with hardcoded /home in dirs.go, this fix will break completely and this makes me a sad panda...
Would it be possible to make it more generic and future-proof (e.g. check if any of the filesystems that the snap may potentially be interested in is NFS)? Don't we get the same problem if - say - /usr or /var is NFS-mounted?

@zyga

zyga Oct 3, 2017

Contributor

This was added by a request from @jdstrand - the idea is that we want to slowly open this up to more use cases. For now we just want to support /home on NFS. We plan to expand this to CIFS as well but not in arbitrary places yet.

Note that /home is already explicitly described in apparmor profiles and there's no easy way around it today.

@stolowski

stolowski Oct 5, 2017

Contributor

Fair enough, thanks.

@jdstrand

jdstrand Oct 5, 2017

Contributor

To check for home or not is a difficult question. If we consider that we are lessening the security of the system if NFS is detected, then I think pursuing only doing it when needed is worthwhile. For example, if there is an NFS mount in /srv for something, snaps don't have access to that anyway, so adding network rules to everything snappy for something no snaps will access is wrong. Determining exactly when to apply NFS rules might take some effort, but I think if we are thoughtful and open to iterating, we can do it without too much complexity.

Note that /home is not actually hard-coded in AppArmor policy as included in snappy for app snaps, it is defined via an apparmor tunable which the user can adjust and the apparmor policy for app snaps will be ok.

We do hardcode /home in snap-confine code and apparmor policy though. Until this is resolved, there is no requirement to adjust this PR. If you wanted to make this PR dynamic today, you could do the equivalent of getent passwd, looking at every uid >= 1000, collecting the dirname's of the found homes and then iterate through those to check if on NFS. Of course, this wouldn't catch someone who has a symlink from /home/foo to /var/exports/homes/foo, so could readlink() first. Alternatively, perhaps in the future the core snap can gain 'snap set' config for adding additional locations for HOME (which would update the apparmor tunables). If so, this code could be updated to check /home and any additional directories that were set with snap set.

interfaces/apparmor/backend.go
+ }
+ fstab, err := mount.LoadProfile(etcFstab)
+ if err != nil {
+ return false, fmt.Errorf("cannot parse /etc/fstab, %s", err)
@jdstrand

jdstrand Oct 5, 2017

Contributor

You are specifying etcFstab to mount.LoadProfile() but hardcoding /etc/fstab in Errorf

@zyga

zyga Oct 5, 2017

Contributor

Done

interfaces/apparmor/backend.go
@@ -254,6 +361,9 @@ func addContent(securityTag string, snapInfo *snap.Info, opts interfaces.Confine
// the super-broad template we are starting with.
} else {
tagSnippets = snippetForTag
+ if nfs, _ := isHomeUsingNFS(); nfs {
@jdstrand

jdstrand Oct 5, 2017

Contributor

Can you add a comment here like you did above:

// Check if NFS is mounted at or under $HOME. Because NFS is not
// transparent to apparmor we must alter the profile to counter that and
// allow access to SNAP_USER_* files.
@zyga

zyga Oct 5, 2017

Contributor

Done

interfaces/apparmor/backend_test.go
+ {
+ fstab: "localhost:/srv/nfs /home/zyga/nfs nfs defaults 0 0",
+ nfs: true,
+ },
@jdstrand

jdstrand Oct 5, 2017

Contributor

fstab entry has 'nfs', but comment is NFSv4 here and the test below. I like these tests, we probably want tests for nfs4 and (at least spot checks for) nfs.

@zyga

zyga Oct 5, 2017

Contributor

This is how it works actually. You get NFSv4 by default but you can still specify NFSv3 by using nfsvers=3. I'll expand tests to include NFSv3 as well.

interfaces/apparmor/backend_test.go
+ // Make it appear as if NFS was mounted under /home.
+ restore := apparmor.MockMountInfo("680 27 0:59 / /home/zyga/nfs rw,relatime shared:478 - nfs4 localhost:/srv/nfs rw,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1")
+ defer restore()
+ restore = apparmor.MockEtcFstab("")
@jdstrand

jdstrand Oct 5, 2017

Contributor

I noticed that we have a lot of tests using MockMountInfo("...") when using MockEtcFstab(""), but not the other way around or when both are set (outside of TestIsHomeUsingNFS()). Was this intentional?

@zyga

zyga Oct 5, 2017

Contributor

I wrote mountinfo support code first and only then added the trigger code that looked for fstab. Since other tests are checking that they have equivalent power I should perhaps just say "mock stuff so that NFS workaround is enabled" without going into the details of how that is done.

@zyga

zyga Oct 5, 2017

Contributor

Also done now.

interfaces/apparmor/backend_test.go
+ // We created the policy file.
+ files, err := ioutil.ReadDir(dirs.SnapConfineAppArmorDir)
+ c.Assert(err, IsNil)
+ c.Assert(files, HasLen, 1)
@jdstrand

jdstrand Oct 5, 2017

Contributor

If we can't read /proc/self/exe, why would we create the policy file? We want to create the directory but I see no reason why we would fallback to creating the file, even if we don't reload, since something outside of snapd might reload the policy and then have the NFS policy included.

@zyga

zyga Oct 5, 2017

Contributor

Because the policy is equally applicable to the re-exec case. We generate the local policy file but don't reload the distro profile. When Setup is called it will choose (or not) to reload the core profile. If it chooses to reload it at that time the generated include file will be present and we will benefit from it.

In all cases I think that if we cannot examine /proc/self/exe we will have other trouble elsewhere so I think this is good as is.

@jdstrand

jdstrand Oct 5, 2017

Contributor

I understand with the way you wrote setupSnapConfineGeneratedPolicyImpl() why you wrote the test this way, but my point is to fail closed rather than open (and reorder the read on procSelfExe so we don't generate the policy). The idea is if there is a crazy kernel bug on reading /proc/self/exe, we fail early and loud rather than try to plod along.

I also understand your point that being able to read /proc/self/exe or not doesn't really have anything to do with isHomeUsingNFS(), which is why you wrote out the policy. I'll leave the decision up to you (ie, not a blocker).

@zyga

zyga Oct 16, 2017

Contributor

I'm changing this now.

interfaces/apparmor/backend_test.go
+
+ // Setup generated policy for snap-confine.
+ err = apparmor.SetupSnapConfineGeneratedPolicy()
+ c.Assert(err, ErrorMatches, "cannot reload snap-confine apparmor profile: testing")
@jdstrand

jdstrand Oct 5, 2017

Contributor

If the policy load fails, it seems to make sense to clean up the generated profile?

@zyga

zyga Oct 5, 2017

Contributor

Hmmm, I think the re-exec-from-core case may still want that but if we cannot load it ourselves then yeah, lets remove it. I'll make the adjustment.

@zyga

zyga Oct 5, 2017

Contributor

Done.

interfaces/apparmor/template.go
+// to operate when NFS is used. This is an imperfect solution as this grants
+// some network access to all the snaps on the system.
+var nfsSnippet = `
+ # Workaround for systems using NFS, for details see:
@jdstrand

jdstrand Oct 5, 2017

Contributor

Can you: s/# Workaround for/# snapd autogenerated workaround for/

@zyga

zyga Oct 5, 2017

Contributor

Done

+ Snapd now contains a feature where NFS-mounted /home (or any sub-directory)
+ initializes a workaround mode where all snaps gain minimal amount of network
+ permissions sufficient for NFS to operate.
+systems: [ubuntu-16.04-64] # TODO: expand this list
@jdstrand

jdstrand Oct 5, 2017

Contributor

Do you plan to address this TODO in this PR?

@zyga

zyga Oct 5, 2017

Contributor

No, it's a lot of hairy stuff already. I'll address this with separate PRs.

tests/main/nfs-support/task.yaml
+execute: |
+ run_export_nfs() {
+ # NOTE: This is so silly because spellchecker complains about the real
+ # command name so w have to go and obfuscate it /o\
@jdstrand

jdstrand Oct 5, 2017

Contributor

s/so w/so we/

@zyga

zyga Oct 5, 2017

Contributor

Fixed

zyga added some commits Oct 5, 2017

tests: test various combinations of NFS: NFS{v3-tcp,v3-udp,v4}
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: include real filename in error message
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: include real filename in error message
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: revise TODO comment into a NOTE
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: fix typo
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: use -O no-expr-simplify
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: include real filename in error message
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add comment about why NFS workaround is needed
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: expand and document NFS tests
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: use more explicit mock helpers
This patch changes two chained mock calls into a call to one helper that
has explicit purpose (enable or disable NFS workaround conditions)
without clouding particular tests about how that is done.

Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: remove generated policy if loading fails
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
tests: fix typo "we"
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Contributor

zyga commented Oct 5, 2017

@jdstrand I've addressed or commented on all the items you have highlighted except for the .real filename. I'll look into fixing that one properly now.

interfaces/apparmor: handle both .real and plain profiles
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Contributor

zyga commented Oct 5, 2017

@jdstrand I believe everything is changed or explained (in one case)

Contributor

jdstrand commented Oct 5, 2017

Approved (though see my last comment (not a blocker)).

Contributor

zyga commented Oct 16, 2017

@jdstrand after thinking about it I'll rework the code slightly.

zyga added some commits Oct 16, 2017

interfaces/apparmor: don't generate policy if /proc/self/exe is broken
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

Nice, thanks for working on this.

Some simple, and probably last, notes.

dirs/dirs.go
@@ -176,9 +176,9 @@ func SetRootDir(rootdir string) {
SnapDataDir = filepath.Join(rootdir, "/var/snap")
SnapDataHomeGlob = filepath.Join(rootdir, "/home/*/snap/")
SnapAppArmorDir = filepath.Join(rootdir, snappyDir, "apparmor", "profiles")
+ SnapConfineAppArmorDir = filepath.Join(rootdir, snappyDir, "apparmor", "snap-confine.d")
@niemeyer

niemeyer Oct 19, 2017

Contributor

Can we please fix the name for this directory to "snap-confine" while we have time (no .d)? That convention doesn't make much sense here.

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/backend.go
+ return setupSnapConfineGeneratedPolicy()
+}
+
+// setupSnapConfineGeneratedPolicyImpl inspects the system and sets up local
@niemeyer

niemeyer Oct 19, 2017

Contributor

That's three levels of indirection, and looking through the code it doesn't look like we need any of it. Can we please have the logic below under the real Initialize, and use the real initialize from tests?

@zyga

zyga Oct 20, 2017

Contributor

Sure, done now. This also removed some extra boiler plate code :)

interfaces/apparmor/backend.go
+func setupSnapConfineGeneratedPolicyImpl() error {
+ // Create the local policy directory if it is not there.
+ if err := os.MkdirAll(dirs.SnapConfineAppArmorDir, 0755); err != nil {
+ return fmt.Errorf("cannot create snap-confine policy directory, %s", err)
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/,/:/. Every error and log message in this PR seems to use this convention, and this is not our usual convention. Can you please review all error messages and make sure they follow suit?

As a more general note, let's please be careful to preserve consistency in the code base. I've seen that sort of minor discrepancy often, and if it goes unnoticed the code gradually becomes messy. We can change conventions, but we cannot change them in isolation and without notice.

@zyga

zyga Oct 20, 2017

Contributor

I was wondering why I got this wrong. I must have got the impression that the cannot quux: %s is the wrong way and cannot quux, %s is the right way of formatting errors somehow (as I tried to be consistent, just going the wrong way).

I've reviewed and corrected all the error messages around this code.

interfaces/apparmor/backend.go
+ }
+
+ // Check the /proc/self/exe symlink, this is needed below but we want to
+ // fail early if this fails for whatever reason.
@niemeyer

niemeyer Oct 19, 2017

Contributor

Thanks for making that clear. That sort of comment helps much when someone else goes to refactor the logic.

interfaces/apparmor/backend.go
+}
+
+var (
+ procSelfMountInfo = mount.ProcSelfMountInfo
@niemeyer

niemeyer Oct 19, 2017

Contributor

Let's please move this up to the common block above.

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/backend.go
+ procSelfMountInfo = mount.ProcSelfMountInfo
+)
+
+// isHomeUsingNFS returns true if /home contains or might contain NFS mounts.
@niemeyer

niemeyer Oct 19, 2017

Contributor

The "might contain" is a truism and would be nice to clarify, in the sense that /etc/fstab might contain NFS entries at all times. Clearly there's something more specific implied there since it mentions both "contains" and "might contain", though.

@zyga

zyga Oct 20, 2017

Contributor

Re-worded to

// isHomeUsingNFS returns true if NFS mounts are defined or mounted under /home.

@@ -121,7 +260,7 @@ func setupSnapConfineReexec(snapInfo *snap.Info) error {
// create for policy extensions for snap-confine. This is required for the
// profiles to compile but distribution package may not yet contain this
// directory.
- if err := os.MkdirAll(dirs.SnapAppArmorConfineDir, 0755); err != nil {
+ if err := os.MkdirAll(dirs.SnapConfineAppArmorDir, 0755); err != nil {
@niemeyer

niemeyer Oct 19, 2017

Contributor

That name makes more sense indeed, thanks.

interfaces/apparmor/backend_test.go
+ fstab: "localhost:/srv/nfs /home nfs4 defaults 0 0",
+ nfs: true,
+ },
+ {
@niemeyer

niemeyer Oct 19, 2017

Contributor

We tend to unify those lines elsewhere, as in }, {. The extra spacing doesn't buy us much. The comments can go inside the brackets instead of outside, which even improves the readability as it's closer to the data.

@zyga

zyga Oct 20, 2017

Contributor

Done, the whole section has significantly de-indented itself now.

interfaces/apparmor/export_test.go
+ old := procSelfMountInfo
+ f, err := ioutil.TempFile("", "mountinfo")
+ if err != nil {
+ panic(fmt.Errorf("cannot open temporary file %s", err))
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/file/file:/

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/export_test.go
+ panic(fmt.Errorf("cannot open temporary file %s", err))
+ }
+ if err := ioutil.WriteFile(f.Name(), []byte(text), 0644); err != nil {
+ panic(fmt.Errorf("cannot write mock mountinfo file %s", err))
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/file/file:/

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/export_test.go
+
+// MockEtcFstab mocks content of /etc/fstab read by isHomeUsingNFS
+func MockEtcFstab(text string) (restore func()) {
+ old := procSelfMountInfo
@niemeyer

niemeyer Oct 19, 2017

Contributor

Bad copy & paste here.

@zyga

zyga Oct 20, 2017

Contributor

Ouch, thank you for noticing that. Corrected.

interfaces/apparmor/export_test.go
+ old := procSelfMountInfo
+ f, err := ioutil.TempFile("", "fstab")
+ if err != nil {
+ panic(fmt.Errorf("cannot open temporary file %s", err))
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/file/file:/

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/export_test.go
+ panic(fmt.Errorf("cannot open temporary file %s", err))
+ }
+ if err := ioutil.WriteFile(f.Name(), []byte(text), 0644); err != nil {
+ panic(fmt.Errorf("cannot write mock fstab file %s", err))
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/file/file:/

@zyga

zyga Oct 20, 2017

Contributor

Done

interfaces/apparmor/export_test.go
+ }
+ etcFstab = f.Name()
+ return func() {
+ os.Remove(etcFstab)
@niemeyer

niemeyer Oct 19, 2017

Contributor

We need a check here that panics if this is /etc/fstab. We don't want to kill the system by mistake.

@zyga

zyga Oct 20, 2017

Contributor

Oh, good idea. Done

+
+// nfsSnippet contains extra permissions necessary for snaps and snap-confine
+// to operate when NFS is used. This is an imperfect solution as this grants
+// some network access to all the snaps on the system.
@niemeyer

niemeyer Oct 19, 2017

Contributor

I'm honestly quite surprised that the application needs network access to access the filesytem. I won't question that as we'd not have gotten that far if that wasn't true, but I'd like to hear more about how this ended up like this. It seems to make no sense to me at least.

@zyga

zyga Oct 19, 2017

Contributor

I'll find the relevant apparmor bug and link it here. EDIT. After some discussion with jj there's a new bug report: https://bugs.launchpad.net/apparmor/+bug/1724903

@zyga

zyga Oct 20, 2017

Contributor

I also added the reference to the bug report into the source code.

overlord/ifacestate/helpers.go
@@ -74,13 +74,19 @@ func (m *InterfaceManager) addInterfaces(extra []interfaces.Interface) error {
return nil
}
-func (m *InterfaceManager) addBackends(extra []interfaces.SecurityBackend) error {
+func (m *InterfaceManager) addAndInitBackends(extra []interfaces.SecurityBackend) error {
@niemeyer

niemeyer Oct 19, 2017

Contributor

It's actually init and then add, not the opposite, but instead of fixing that let's revert and preserve the original function name here as it's symmetrical with other functions around it. It will do whatever it needs to do for adding the backends. We don't need to call out every operation in its name.

@zyga

zyga Oct 20, 2017

Contributor

Done

tests/main/nfs-support/task.yaml
+
+ # Install NFS server and a simple shell snap.
+ distro_install_package nfs-kernel-server
+ distro_install_package bsdgames
@niemeyer

niemeyer Oct 19, 2017

Contributor

The package installations should happen at the global location next to every other package installation, and it doesn't have to be purged at restore time. Please coordinate with Sergio so that these packages also go into the image maintenance spread runs so that once we finally manage to cook packages into the images this will be there too.

@zyga

zyga Oct 20, 2017

Contributor

I'm coordinating this with Sergio now.

tests/main/nfs-support/task.yaml
+execute: |
+ run_export_nfs() {
+ # NOTE: This is so silly because spellchecker complains about the real
+ # command name so we have to go and obfuscate it /o\
@niemeyer

niemeyer Oct 19, 2017

Contributor

Ouch. If we end up with more of those, we may as well drop spellchecker.

@zyga

zyga Oct 20, 2017

Contributor

It would be good to be able to teach the spellchecker about new words. I'll see if I can do that instead of this hack.

@zyga

zyga Oct 20, 2017

Contributor

Ha, that was actually easier than I thought. Done now.

tests/main/nfs-support/task.yaml
+ ensure_normal_perms
+
+ # Back up the /etc/fstab file and define a NFS mount mount there.
+ cp /etc/fstab fstab.bak
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/bak/orig/

@zyga

zyga Oct 20, 2017

Contributor

Done

tests/main/nfs-support/task.yaml
+ # Unmount NFS mount over /home if one exists.
+ umount /home || true
+
+ # Restore the fstab backup file if one exits.
@niemeyer

niemeyer Oct 19, 2017

Contributor

exists

@zyga

zyga Oct 20, 2017

Contributor

Done

tests/main/nfs-support/task.yaml
+
+ # Restore the fstab backup file if one exits.
+ if [ -e fstab.bak ]; then
+ mv fstab.bak /etc/fstab
@niemeyer

niemeyer Oct 19, 2017

Contributor

s/bak/orig/

@zyga

zyga Oct 20, 2017

Contributor

Done

zyga added some commits Oct 20, 2017

tests: use .orig instead of .bak
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
tests: fix typo exits -> exists
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
overlord/ifacestate: restore original method name
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: refer to NFS non-transparency bug
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: add safety check to MockEtcFstab
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: separate errors with a colon
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
Contributor

zyga commented Oct 20, 2017

With the exception of unified location of package installation this is all done now. I've also renamed the generated file to nfs-support and changed glob to *. Let's see if I missed anything or if tests will work.

zyga added some commits Oct 20, 2017

interfaces/apparmor: fix incorrect copy/pasted code
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: tweak spacing of test code
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: tweak comment for clarity
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: move definition to common var block
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: separate errors with a colon
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor: remove indirection and test Initialize directly
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
tests: add "exportfs" to known words and remove the spell hack
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
many: drop .d from /var/lib/snapd/apparmor/snap-confine.d
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
interfaces/apparmor,tests: rename generated-nfs to nfs-support
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@canonical.com>

@niemeyer niemeyer merged commit a4eb1bb into snapcore:master Oct 23, 2017

5 of 7 checks passed

xenial-amd64 autopkgtest finished (failure)
Details
zesty-amd64 autopkgtest finished (failure)
Details
artful-amd64 autopkgtest finished (success)
Details
artful-i386 autopkgtest finished (success)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
xenial-i386 autopkgtest finished (success)
Details
xenial-ppc64el autopkgtest finished (success)
Details

@zyga zyga deleted the zyga:feature/nfs-support branch Oct 23, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment