Fix memory_deny_write_execute on x86 and s390 with libseccomp 2.4.2

[cpaelzer]

Hi,
for now this is meant to start a discussion and make people aware.
Discussions with libseccomp upstream are still ongoing and also I'm not sure on your preferences in regard to falling back to non _exact seccomp calls.

Background

systemd commit 67fb5f already tried to fix expectations for test_memory_deny_write_execute_shmat.

But it turns out on s390x/s390 and 32bit x86 the same nature of being multiplexed (or not) makes seccomp_rule_add_exact fail on some combinations of kernel/libseccomp - in particular since libseccomp 2.4.2.

This was discussed with libseccomp in an issue on their tracker.
And it seems that is intended behavior and for rules on shmat we can't use _exact at all anymore.

Issue to systemd

That leads to a bad rc returned seccomp_rule_add_exact -> add_seccomp_syscall_filter -> seccomp_memory_deny_write_execute which will then in turn skip the loading of the rules as the continue due to the bad rc will skip&loop over seccomp_load(seccomp).

As a suggestion from libseccomp upstream we could use the non _exact versions of those calls for items we know to be affected. This makes it "work again" on x86 and "work" on s390x (which it did before on some combinations of glibc/libseccomp but not on others).

It is possible (see libseccomp discussion) be that with libseccomp 2.4.3 s390x will not need this anymore, but it seems x86 and s390 will continue to do so.

With the changes applied I retested s390x and x86 in Ubuntu 20.04 (where I had the initial issues).

• s390x-2.4.1: pkey_mprotect throws Bad address on s390 (not a new issue), but test succeeds (not broken by the new changes), blocked bad access as intended
• s390x-2.4.2: no errrors anymore, blocked bad access as intended
• x86-2.4.1: no errrors anymore, blocked bad access as intended
• x86-2.4.2: no errrors anymore, blocked bad access as intended

I haven't touched older kernels / glibcs in my tests yet, these tests ran on Ubuntu 20.04 with 5.3.0-18-generic and 2.30-0ubuntu2 respectively.

Extra fixes around seccomp handling in this MR

The first commit in the series tries to fix the issue I hit when moving to libseccomp 2.4.2, the other two in the MR try to fix issues I spotted along the way - the commit messages explain in detail what they are about.

Based on #14162

P.S. I first revoked commit a81f7aa from rbalint on my Ubuntu tests as the move to libseccomp 2.4.2 broke s390, but with my fixes it works again therefore this series is for now intentionally based on-top of his MR.

The whole situation is convoluted in between kernel/glibc/libseccomp versions, so please let me know what you think. Also please feel free to chime in on the libseccomp discussion. I'll ask there to do the same here.

[keszybz]

We prefer to return 1if something happened, and 0 if nothing did. Also 8 space indentation. Maybe just do:

if (loaded == 0)
            log_debug_errno(r, "Failed to install any seccomp rules for MemoryDenyWriteExecute=");
return loaded;

This return value is not used by anything? I'm not sure what you are trying to do here.

[cpaelzer]

Thanks for taking a look @keszybz
While src/core/execute.c doesn't care about the retval at least the tests in src/test/test-seccomp.c would and that way could spot issues like the main issue the first commit tries to address.

The already existing semantic of the retval is following seccomp_rule_* which is to "return zero on success, negative errno values on failure". Your suggestion would still into that as a positive value (the rules loaded) would match well that pattern. So I'm adopting your suggestion.

Also I fixed the 8 space indent, thanks for the hint.

This will be in a coming push to the branch, but I see that @poettering just added review feedback as well. I'll go through that first before pushing anything.

[poettering]

we stopped using the non-exact versions a long to ago since you never knew what you#d get, when used interchangably in whitelists and blacklists... i'd prefer if we can just deal with this issue in our own code.

indentation is off btw, please stick to 8ch, see https://systemd.io/CODING_STYLE

[poettering]

(also, we use bool for booleans, except in public, exposed APIs, which this is not)

[cpaelzer]

Hi @poettering,
indention already fixed a minute ago after @keszybz pointing it out.

I'll carry the preference to stick to _excat calls to the discussion in libseccomp project.
If there is a way to stick to _exact that works I'm happy to use that and will refresh this pull request.

But ahead of time if there isn't a way what would we want to do about s390 and x86 then instead. We could obviously just ignore the retval in those cases, but that means less protection. Do you have any alternative ideas in mind already?

[cpaelzer]

I asked a similar question before and the answer I just got was

I would suggest using seccomp_rule_add(...) as it will handle all the architecture oddities for you; in fact for 99% of the cases out there seccomp_rule_add(...) is preferred over seccomp_rule_add_exact(...).

To me that seems that like only way to go unless we switch to more drastic changes (after all TBH who cares about 31/32bit - lets maybe just remove their shmat call, /me runs away ...)

I asked @pcmoore to come over and participate in the discussion here directly.

[cpaelzer]

(also, we use bool for booleans, except in public, exposed APIs, which this is not)

Fixed as well, thanks

[cpaelzer]

After speaking out loud my heretic thought of "TBH who cares about 31/32bit" I gave this a new check from this POV. We already have this on ppc:

Note that shmat() isn't available, and the call is multiplexed through ipc().
We ignore that here, which means there's still a way to get writable/executable
memory, if an IPC key is mapped like this. That's a pity, but no total loss.

So we might just make x86, s390 and s390x fall into the same category as ppc* of not defining a rule for shmat at all?
I that would be generally preferred to non-exact seccomp calls I'm happy to re-implement, retest and update this branch. Let me know what you think about that please.

[poettering]

BTW, we used to use the non-_exact() calls, until I ripped it all out: 469830d. It was not maintainable since we never knew what libseccomp would actually do if a rule could not be expressed on some arch: i.e. apply the rule incompletely or not at all, and the difference matters in some cases, i.e. when you do whitelisting rather than blacklisting. the strategy used by the non-_exact() stuff was a blackbox for us and too often not the strategy we actually want. I am not convinced we should start with that now.

[poettering]

(we also started to write out rules per-arch ourselves in the same rework, instead of relying on libseccomp doing that, for similar reasons)

(and we actually need the info what happens on which arch anyway, so that we can test for it. having libseccomp hide that in a black box and changing strategy behind our backs can't work if we test for our sandboxing to work correctly)

[poettering]

anyway, i don't grok the actual problem here in the first place?

is this just some multiplexing issue exactly like on i386? if so, then handle it the same way.

[cpaelzer]

anyway, i don't grok the actual problem here in the first place?
is this just some multiplexing issue exactly like on i386?

libseccomp 2.4.2 got stricter on those multiplexed cases which breaks the behavior of systems secomp code for shmat on i386+s390+s390x.
It is stricter but not broken, as it is the expected behavior, so they won't fix/change it there.
I'm trying to come up with a fix for it in systemd that makes seccomp handling work again on those architectures.

if so, then handle it the same way.

Could I have a pointer to which same way you are referring - as i386 is broken at the moment with the new libseccomp.

[cpaelzer]

Rework and Motivation

I reworked the PR according to the feedback of @poettering. That means I avoided the non-_exact seccomp calls and instead handled it like we already do on other architectures (e.g. ppc already had the same case that now affects i386, s390 and s390x as well)

And since there formerly were questions about "why/what we'd need to fix here" - maybe it is too present in my head - let me add a few references to help with that:

• Ubuntu bug report - more about the symptom
• libseccomp bug report - more about the reasons
• This PR here is the usual three makes a charm and is about getting it fixed

I have built this on top of Ubuntus systemd (243) and re-ran the self tests.
The results are promising and show the tests as good now on all architectures.
Example tests logs from the Ubuntu package CI runs:

• x86-64
• i386
• s390x

I also ran the test-seccomp tests that spotted the issue initially individually in VMs and they now look like this:

No-Fix + new libseccomp (2.4.2)

/* test_memory_deny_write_execute_mmap */
Operating on architecture: s390
Failed to add shmat() rule for architecture s390, skipping: Invalid argument
Operating on architecture: s390x
Failed to add shmat() rule for architecture s390x, skipping: Invalid argument
Assertion 'p == MAP_FAILED' failed at src/test/test-seccomp.c:493, function test_memory_deny_write_execute_mmap(). Aborting.
memoryseccomp-mmap terminated by signal ABRT.
Assertion 'wait_for_terminate_and_check("memoryseccomp-mmap", pid, WAIT_LOG) == EXIT_SUCCESS' failed at src/test/test-seccomp.c:507, function test_memory_deny_write_execute_mmap(). Aborting.
Aborted

/* test_memory_deny_write_execute_mmap */
Operating on architecture: x86
Failed to add shmat() rule for architecture x86, skipping: Invalid argument
Assertion 'p == MAP_FAILED' failed at src/test/test-seccomp.c:493, function test_memory_deny_write_execute_mmap(). Aborting.
memoryseccomp-mmap terminated by signal ABRT.
Assertion 'wait_for_terminate_and_check("memoryseccomp-mmap", pid, WAIT_LOG) == EXIT_SUCCESS' failed at src/test/test-seccomp.c:507, function test_memory_deny_write_execute_mmap(). Aborting.
FAIL: test-seccomp (code: 134)
Aborted (core dumped)

Note: the invalid argument in these examples is the effect of seccomp_rule_add_exact on multiplexed calls

Fix + old libseccomp (2.4.1)

/* test_memory_deny_write_execute_mmap */
Operating on architecture: s390
Failed to add n/a() rule for architecture s390, skipping: Bad address
Operating on architecture: s390x
memoryseccomp-mmap succeeded.
/* test_memory_deny_write_execute_shmat */
arch s390: SCMP_SYS(mmap) = 90
arch s390: SCMP_SYS(mmap2) = -10029
arch s390: SCMP_SYS(shmget) = 395
arch s390: SCMP_SYS(shmat) = 397
arch s390: SCMP_SYS(shmdt) = 398
arch s390x: SCMP_SYS(mmap) = 90
arch s390x: SCMP_SYS(mmap2) = -10029
arch s390x: SCMP_SYS(shmget) = 395
arch s390x: SCMP_SYS(shmat) = 397
arch s390x: SCMP_SYS(shmdt) = 398
Operating on architecture: s390
Failed to add n/a() rule for architecture s390, skipping: Bad address
Operating on architecture: s390x
shmat(SHM_EXEC): Success
shmat(0): Success
memoryseccomp-shmat succeeded.

/* test_memory_deny_write_execute_mmap */
Operating on architecture: x86
memoryseccomp-mmap succeeded.
/* test_memory_deny_write_execute_shmat */
arch x86: SCMP_SYS(mmap) = 90
arch x86: SCMP_SYS(mmap2) = 192
arch x86: SCMP_SYS(shmget) = 395
arch x86: SCMP_SYS(shmat) = 397
arch x86: SCMP_SYS(shmdt) = 398
Operating on architecture: x86
shmat(SHM_EXEC): Success
shmat(0): Success
memoryseccomp-shmat succeeded.

Note: working with older libseccomp, our test already had a relaxed check: "Depending on kernel, libseccomp, and glibc versions, other architectures might fail or not. Let's not assert success."
Note II: the "Bad adress" is pkey_mprotect and unrelated to my changes

Fix + new libseccomp (2.4.2)

/* test_memory_deny_write_execute_mmap */
Operating on architecture: s390
Operating on architecture: s390x
memoryseccomp-mmap succeeded.
/* test_memory_deny_write_execute_shmat */
arch s390: SCMP_SYS(mmap) = 90
arch s390: SCMP_SYS(mmap2) = -10029
arch s390: SCMP_SYS(shmget) = 395
arch s390: SCMP_SYS(shmat) = 397
arch s390: SCMP_SYS(shmdt) = 398
arch s390x: SCMP_SYS(mmap) = 90
arch s390x: SCMP_SYS(mmap2) = -10029
arch s390x: SCMP_SYS(shmget) = 395
arch s390x: SCMP_SYS(shmat) = 397
arch s390x: SCMP_SYS(shmdt) = 398
Operating on architecture: s390
Operating on architecture: s390x
shmat(SHM_EXEC): Success
shmat(0): Success
memoryseccomp-shmat succeeded.

/* test_memory_deny_write_execute_mmap */
Operating on architecture: x86
memoryseccomp-mmap succeeded.
/* test_memory_deny_write_execute_shmat */
arch x86: SCMP_SYS(mmap) = 90
arch x86: SCMP_SYS(mmap2) = 192
arch x86: SCMP_SYS(shmget) = 395
arch x86: SCMP_SYS(shmat) = 397
arch x86: SCMP_SYS(shmdt) = 398
Operating on architecture: x86
shmat(SHM_EXEC): Success
shmat(0): Success
memoryseccomp-shmat succeeded.

Now force-pushing the new fix onto this PR for re-review.

[poettering]

PR looks good to me, but CI doesn't like it:

/* test_memory_deny_write_execute_mmap */
Operating on architecture: s390
Failed to add pkey_mprotect() rule for architecture s390, skipping: Numerical argument out of domain
Operating on architecture: s390x
Failed to add pkey_mprotect() rule for architecture s390x, skipping: Numerical argument out of domain
Failed to install any seccomp rules for MemoryDenyWriteExecute=
Assertion 'p == MAP_FAILED' failed at src/test/test-seccomp.c:536, function test_memory_deny_write_execute_mmap(). Aborting.

[cpaelzer]

I'm finally coming back to this, sorry for the delay.

So lets see how to make CI happy...
I first thought the reported issue is due to "88981fcc: seccomp: ensure rules are loaded in seccomp_memory_deny_write_execute" actually identifying a problem that already existed before. But isn't the root cause of the CI issues.

I found that it actually is because I based my branch on #14162 as I'm testing and working on Ubuntu 20.04. The CI seems to run into exactly the issue discussed on that PR as well.

As I outlined there, commit 67fb5f3 changed the shmat test as it is unable to predict if it will work/fail. The mmap test might need the same treatment to e.g. work well on Ubuntu 18.04 and 20.04 (and others) at the same time.

I rebased to latest master and pushed the branch with the mentioned test cleanups (replacing the one in #14162 so I'm no more basing/depending on that branch/PR).
This works on Ubuntu 20.04 in my tests on i386 as well as s390x.
Let us check if CI on 18.04 is happy as well now.

[poettering]

comments needs minor updating

[cpaelzer]

Absolutely, good catch - dropping s390x from this.

I only wanted to wait with the push to see the result of the s390x CI run on Bionic to be able to rework it right away if there are still issues. But that Test now completed and worked.

Pushing with the comment cleanup now ...

[poettering]

Hmpf... So for some reason the ubuntu CI disappeared from this PR? I'd really like to see its results though, since it compiles on s390 and other more exotic archs which none of the other CIs do.

We can't really restart the CI though, hence can you rebase on current master and repush? That should make the CIs pick this PR up anew again?

[cpaelzer]

Odd that "some" are missing. Maybe because I was waiting for the former CI runs to complete and I force-pushed when it was done on Bionic-s390x but still waiting on others?

Anyway - rebased and repushed for another run.

[poettering]

The CI infra is a bit flakey it seems, there's no system behind it when CI jobs disappear. At least none I recognize... But this time they got picked up properly, all 12 CIs are chugging away on this PR.

[keszybz]

LGTM.

[cpaelzer]

I'm glad that bionix-s390x completed as a good example this time, since when I look for 49219b5 on the link that Details leads to the tests look to me as if they'd hang after being fished with some build task (all of them, not just mine).
I'm almost assuming that amd64/arm64/i386/ppc64el CI runs will vanish like last time once the timeouts are reached.

It doesn't seem to be triggered by my changes and I can't really fix what those CI runs do :-/
How do we go on from here - is the s390x run enough?

[cpaelzer]

Ok, we got the CI please \o/

The remaining "requested changes" are from @keszybz but he later gave his LGTM, so we can mark that resolved as well.

With that this should be good to go right?
If anything else is missing let me know.

[keszybz]

Hmm, we merged this, but the commit message in 903659e is bogus. The commit changes the return value, but nothing looks are the return value to check if it is ==0 or >0. @cpaelzer what was the goal of that commit once again?

[keszybz]

#14265 makes the code "correct", but it still is not doing anything useful except a debug message.

[cpaelzer]

@keszybz - on that particular commit the intention was two fold.

• First we wanted a message if none of the rules were loaded, that was added and worked
• Since we already tracked the number for the above we could return that instead of the blunt zero which fitted in the usage of function and allowed to check it later

The initial thought was to always ensure rules are loaded, but that got changed later. please remember that the interim version we have discussed here made the rules work on affected architectures. And from there the idea was to eventually ensure that it always had rules loaded. But in the discussion we changed that - as we didn't wan't to use the non '_exact' libseccomp calls. With that we'd have diverged from that idea leaving the not-wrong but admittedly currently unused retval there.

Currently a check of it could easily be added at:

• systemd/src/core/execute.c: to check being >0 (breaking things on on affected architectures if MemoryDenyWriteExecute= isn't fully protected), I don't think we'd want this
• src/test/test-seccomp.c: this would break the tests on the affected architectures (but not the execution), I don't think we want that either.

Thanks for the fixup on the errno btw!

[keszybz]

OK. So essentially the commit message wasn't updated, but the code is fine as is. Thanks for the explanation.