BUG: inconsistent handling of ipc syscalls on s390 and s390x

[cpaelzer]

Hi,
I was trying to push 2.4.2 to Ubuntu and after the few cleanups we had in 2.4.1->2.4.2 that looked pretty good at first. But When the extended checks kicked in I reliably found the systemd self-tests breaking on i386 and s390x.
For what it is worth - they worked reliable on x86-64, armhf, arm64 and ppc64el.

I know that there were related systemd fixups but those are not strictly required since our fix triggered by my chrony tests.

After further debugging of the testcase I found that of the many that systemd runs all kinds of activity around shmat seems to have changed.

The number resolves to 397 with old and new seccomp, but fails with version 2.4.2

arch x86: SCMP_SYS(shmat) = 397
...
Failed to add shmat() rule for architecture x86, skipping: Invalid argument

I was going back and forth and isolated the test more and more to now have reached a minimal test program that on i386 as well as s390x reliable works/fails with 2.4.1/2.4.2.

$ cat > test-seccomp-shmat.c << EOF
#include <linux/seccomp.h>

#include <errno.h>
#include <seccomp.h>
#include <stdio.h>

#include <sys/shm.h>

/*
 * Test issues with libseccomp 2.4.1 -> 2.4.2
 * Derived from systemd testcase test_memory_deny_write_execute_shmat
 * which fails to install shmat rules with 2.4.2 on i386 and s390x
 */

int main()
{
   int shmat_syscall = -1;
   int rc = -1;
   scmp_filter_ctx ctx;

   ctx = seccomp_init(SCMP_ACT_ALLOW);
   if (ctx == NULL)
       return -1;

   shmat_syscall = SCMP_SYS(shmat);
   printf("SCMP_SYS(shmat) = %d\n", shmat_syscall);

   rc = seccomp_rule_add_exact(ctx, SCMP_ACT_ERRNO(EPERM), shmat_syscall, 1, SCMP_A2(SCMP_CMP_MASKED_EQ, SHM_EXEC, SHM_EXEC));
   printf("Rule installed RC = %d\n", rc);

   return 0;
}
EOF

Then build with:

$ gcc -Wall test-seccomp-shmat.c -o test-seccomp-shmat -lseccomp

Tests on i386 with the test I pasted above:

2.4.1:
./test-seccomp-shmat
SCMP_SYS(shmat) = 397
Rule installed RC = 0

2.4.2
./test-seccomp-shmat
SCMP_SYS(shmat) = 397
Rule installed RC = -22

s390x looks identical to the i386 output

Note: rebuilding on new libseccomp2 2.4.2 does not change this behavior

Now I'm wondering, is that a yet undiscovered issue in libseccomp 2.4.2 and if so what could it be?
Or if you can clearly state that this is right and should never have worked could you outline why it affects just i386 and s390x and how systemd (see the related systemd test / code here) would need to adapt so that I can start a discussion there?

P.S. kernels used are Ubuntus 5.3.0-18-generic in all tests

[pcmoore]

Hi @cpaelzer, thanks for the detailed problem report, but I'm afraid to report that this is the expected behavior, let me try to explain ...

At the heart of the problem is the use of the seccomp_rule_add_exact(...) call. If you check the manpage you'll notice the following in the first paragraph of the description:

The seccomp_rule_add_exact() and seccomp_rule_add_exact_array() functions will attempt
to add the rule exactly as specified so it may behave differently on different architectures.
While it does not guarantee a exact filter ruleset, seccomp_rule_add() and
seccomp_rule_add_array() do guarantee the same behavior regardless of the architecture.

In other words, seccomp_rule_add_exact(...) will attempt to create a filter exactly as specified, and it will fail if it can't do that because of architecture specific limitations. As an alternative, the seccomp_rule_add(...) function will make a best effort to add the rule to the filter, in some cases it will even transform the rule slightly for each individual architecture. In general most applications are better off using seccomp_rule_add(...), but we offer seccomp_rule_add_exact(...) for those applications that must have the filters exactly as described.

Why is this important here? It is important because you are attempting to add a rule for the shmat(2) syscall with an argument filter on x86. Historically, on 32-bit x86 Linux implemented shmat(2) as a multiplexed syscall, running on top of the ipc(2) syscall; because of the way this was defined in the x86 ABI, it was not possible to reference the syscall arguments in these multiplexed syscalls. While modern Linux kernels (v4.3+) have support for calling these syscalls directly, libseccomp has no idea at build time if it is going to run on an old or new kernel or if the caller is going to use the multiplexed or direct-wired shmat(2) syscall, so it creates a filter with both. Because it is creating a filter for both the multiplexed and direct-wired shmat(2) calls, libseccomp can't guarantee that it will be able to implement the filter with syscall argument filter which means the seccomp_rule_add_exact(...) in your example will fail. However, a similar call to seccomp_rule_add(...) will succeed.

Look at the revised example:

# cat ./test-seccomp-shmat.c
#include <linux/seccomp.h>

#include <errno.h>
#include <seccomp.h>
#include <stdio.h>

#include <sys/shm.h>

/*
 * Test issues with libseccomp 2.4.1 -> 2.4.2
 * Derived from systemd testcase test_memory_deny_write_execute_shmat
 * which fails to install shmat rules with 2.4.2 on i386 and s390x
 */

int main()
{
   int shmat_syscall = -1;
   int rc = -1;
   scmp_filter_ctx ctx;

   ctx = seccomp_init(SCMP_ACT_ALLOW);
   if (ctx == NULL)
       return -1;

   rc = seccomp_arch_remove(ctx, SCMP_ARCH_NATIVE);
   if (rc != 0)
       return rc;
   rc = seccomp_arch_add(ctx, SCMP_ARCH_X86);
   if (rc != 0)
       return rc;

   shmat_syscall = SCMP_SYS(shmat);
   printf("SCMP_SYS(shmat) = %d\n", shmat_syscall);

   rc = seccomp_rule_add_exact(ctx, SCMP_ACT_ERRNO(EPERM), shmat_syscall, 1,
                               SCMP_A2(SCMP_CMP_MASKED_EQ, SHM_EXEC, SHM_EXEC));
   printf("Rule installed (exact) RC = %d\n", rc);

   rc = seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), shmat_syscall, 1,
                         SCMP_A2(SCMP_CMP_MASKED_EQ, SHM_EXEC, SHM_EXEC));
   printf("Rule installed (standard) RC = %d\n", rc);

   return 0;
}

# gcc -o test-seccomp-shmat ./test-seccomp-shmat.c ./libseccomp.a
# ./test-seccomp-shmat
SCMP_SYS(shmat) = 30
Rule installed (exact) RC = -22
Rule installed (standard) RC = 0

Does this make sense?

However, that said, it does appear that we do have some inconsistency with s390 and/or s390x. For both ABIs the XXX_syscall_resolve_XXX(...) functions only have code to handle the socket related syscalls, but the XXX_syscall_mux(...), XXX_syscall_demux(...), and XXX_rule_add(...) functions have code for both the socket and ipc related functions. We need to sort that out for the next release.

@drakenclimber see the last paragraph above. This combined with the other recent fixes makes me think we need to do a v2.4.3 release.

[cpaelzer]

Thanks a lot @pcmoore for explaining!

You said

a similar call to seccomp_rule_add_exact(...) will succeed

But the example given shows only seccomp_rule_add succeeding.
I understood your explanation (in simple words) as: "since it needs to add both rules it will never work with ..._exact which is asking only for one rule causing a mismatch".

As I said this was triggered by systemd tests and it is of the backend implementing MemoryDenyWriteExecute so we'd want to fix not only the test but the code to work as intended.
I'd want to go to to systemd and suggest a fix there, therefore it is important if this would have to be be more like:

    seccomp_rule_add_exact # original call
#endif # x86_64 arm aarch64
#if defined(__i386__) || defined(__s390x__) and/or __s390__?
     seccomp_rule_add_exact  # different form of _exact
#endif

OR if it has to be non-exact version as there is no way out with_exact on x86/s390x

    seccomp_rule_add_exact # original call
#endif # x86_64 arm aarch64
#if defined(__i386__) || defined(__s390x__) and/or __s390__?
     seccomp_rule_add  # use non-exact on those
#endif

[cpaelzer]

Oh, and a hint to a commit or so why this changed between 2.4.1 and 2.4.2 would be great as well for when I try to reference things in a systemd discussion.

[cpaelzer]

Further clarification about s390x in this scope - you said there is some inconsistency. With 2.4.3 and a fix for that in place what should be expected? Will it be _"_exact for shmat (will) work on s390x, but on s390 it will be multiplexed and needs the non exact" or will it be different than that?

Currently I need the non-exact version of the call for x86, s390 and s390x.
Am I right to assume that with 2.4.3 I'll only need it for x86 and s390?

[cpaelzer]

And - I hope finally - for clarification is there anything exposed by seccomp to keep that domain-knowledge what is multiplexed in libseccomp and make use of that.
For example could I do something like:

 seccomp_syscall_is_multiplexed(SCMP_SYS(shmat))
 # if true, one would know that _exact on this one is in vain leading to the discussed issues

Maybe with an optional ARCH argument (=local-arch if not given).

[cpaelzer]

FYI: I opened a - still experimental - MR on systemd about this.

I'd be happy if you can here answer my questions I had since the initial answer.
But also please feel free to chime in their with your subject matter expertise.

[pcmoore]

You said

a similar call to seccomp_rule_add_exact(...) will succeed

But the example given shows only seccomp_rule_add succeeding.
I understood your explanation (in simple words) as: "since it needs to add both rules it will never work with ..._exact which is asking only for one rule causing a mismatch".

Ooops, sorry about that! That was a typo on my part, it should have read: "a similar call to seccomp_rule_add(...) will succeed".

I updated my original comment to fix this.

As I said this was triggered by systemd tests and it is of the backend implementing MemoryDenyWriteExecute so we'd want to fix not only the test but the code to work as intended.
I'd want to go to to systemd and suggest a fix there ...

I would suggest using seccomp_rule_add(...) as it will handle all the architecture oddities for you; in fact for 99% of the cases out there seccomp_rule_add(...) is preferred over seccomp_rule_add_exact(...).

[pcmoore]

Oh, and a hint to a commit or so why this changed between 2.4.1 and 2.4.2 would be great as well for when I try to reference things in a systemd discussion.

I would need to dig into the code to see why v2.4.1 behaved as it did, but the v2.4.2 behavior (with seccomp_rule_add_exact(...) failing in your test case) is the expected behavior.

[pcmoore]

Further clarification about s390x in this scope - you said there is some inconsistency. With 2.4.3 and a fix for that in place what should be expected? Will it be _"_exact for shmat (will) work on s390x, but on s390 it will be multiplexed and needs the non exact" or will it be different than that?

We would need to look into what the Linux kernel supports for s390 and s390x to make sure that libseccomp is doing the right thing. I haven't done that yet, but if you want to get involved in libseccomp we would love the help!

[pcmoore]

And - I hope finally - for clarification is there anything exposed by seccomp to keep that domain-knowledge what is multiplexed in libseccomp and make use of that.

No, and I don't think we will ever support something like that in libseccomp. One of the main goals in libseccomp is to abstract away all the ABI specifics. We want you to just be able to call seccomp_rule_add(shmat, ...) and let libseccomp take care of sorting out the details.

[cpaelzer]

Thanks for all the answers and clarifications @pcmoore those helped me a lot!
As you have seen I opened a systemd pull request for a discussion there.

Almost as I feared there now is a clash of you

I would suggest using seccomp_rule_add(...) as it will handle all the architecture oddities for you; in fact for 99% of the cases out there seccomp_rule_add(...) is preferred over seccomp_rule_add_exact(...).

vs @poettering in that discussion

we stopped using the non-exact versions a long to ago since you never knew what you#d get, when used interchangably in whitelists and blacklists... i'd prefer if we can just deal with this issue in our own code.

@pcmoore - could I "ask you over" to participate in the discussion there so that we can jointly identify a solution that works, but also is acceptable to systemd?

[pcmoore]

Yes, Lennart does what Lennart wants to do, and I suspect my involvement will have little benefit other than intruding on the holidays here in the US (Thanksgiving). I looked quickly at the systemd thread it appears that there are some possible paths towards a solution - good luck!

[pcmoore]

Hi @cpaelzer, any chance you could test the fix @drakenclimber made in #206?

[pcmoore]

This should be fixed in the release-2.4 branch by 266e1b4 and in master by 0a4c030.

[cpaelzer]

@pcmoore sorry - I was on PTO last week and I missed the ping 27 days ago by @drakenclimber .

I revived my test of the initial report (see above) and was using a local build of current libsecomp master as of today, but the result is still the same:

./test-seccomp-shmat 
SCMP_SYS(shmat) = 397
Rule installed RC = -22

So it didn't get worse, but the rule add with exact still didn't work - should it have worked?

[pcmoore]

No problem @cpaelzer, thanks for checking back and giving it a test. Unfortunately it would appear that more work is needed so I'm going to reopen this issue.

@drakenclimber and you okay to keep working on this with @cpaelzer?

[drakenclimber]

Sure. We can figure this out :)

[cpaelzer]

@drakenclimber AFAIK at the price of some personal data you can get a s390x VM here

[drakenclimber]

@drakenclimber AFAIK at the price of some personal data you can get a s390x VM here

Thanks! I'll give it a shot with my Oracle email. We'll see what IBM thinks of that ;)

[drakenclimber]

I was able to reserve an s390x machine. Thanks, @cpaelzer.

$ uname -r
3.10.0-957.21.3.el7.s390x

Currently I am unable to reproduce the behavior you have reported. Here is what I see when running libseccomp v2.4.1. (I added some extra printfs in the library itself to verify I was running the correct library.)

$ gcc -Wall test-seccomp-shmat.c -o test-seccomp-shmat -lseccomp
$ ./test-seccomp-shmat 
SCMP_SYS(shmat) = -221
Running v2.4.1 libseccomp
Built on Feb 24 2020 at 16:03:52
Rule installed RC = -22

And for completeness - I am seeing the exact same thing on the head of master.

$ ./test-seccomp-shmat 
SCMP_SYS(shmat) = -221
Running master libseccomp
Built on Feb 24 2020 at 16:06:14
Rule installed RC = -22

[drakenclimber]

Also, this is the line that is causing the failure. I would like to hear @pcmoore's thoughts on this section of code. It appears that s390 and x86 are designed to always reject multiplexed-socket syscall strict rules that have an argument. I'm not yet sure what changed from v2.4.1 to v2.4.2 for @cpaelzer to see different behavior.

src/arch-s390x.c#L370

	} else if ((sys <= -200 && sys >= -224) || (sys >= 392 && sys <= 402)) {
		/* (-200 to -224) : multiplexed ipc syscalls
		   (392 to 402) : direct ipc syscalls */

		/* strict check for the multiplexed socket syscalls */
		for (iter = 0; iter < ARG_COUNT_MAX; iter++) {
			if ((rule->args[iter].valid != 0) && (rule->strict)) {
				rc = -EINVAL;
				goto add_return;
			}
		}

[pcmoore]

I would like to hear @pcmoore's thoughts on this section of code. It appears that s390 and x86 are designed to always reject multiplexed-socket syscall strict rules that have an argument.

When adding a "strict" rule (in other words, using the *_exact(...) rule APIs) if the libseccomp library can not add a rule exactly as supplied by the caller it will fail. The non-strict/non-exact rule APIs allow the libseccomp library to do some minor "munging" to enable the operation to succeed where "munging" typically means arch/ABI related adjustments (e.g. multiplexed syscalls).

The reason we fail multiplexed socket syscalls when an argument check is supplied is that we can not perform the argument check due to the nature of how the syscall is multiplexed. In these cases the individual syscall is passed to the multiplexed syscall as arg0 and the individual syscall's arguments are combined into a structure which is passed, via a pointer, to the multiplexed syscall. It is because of this argument structure passing that we are unable to inspect the argument and do the argument filtering.

... and before anyone asks, this is a kernel limitation, not something we can fix in libseccomp. The solution is simply to use the non-exact rule APIs and let libseccomp handle it for you. It will do what you want[*], you just have to give it the license to do so ;)

[*] Including generating both the argument comparison filter for the direct wired syscall as well as the slightly munged filter for the multiplexed syscall so that if you run on a newer kernel/libc the argument comparison will be made, but it will still do the basic syscall checking on an older kernel/libc. I'm not aware of any other seccomp library/mechanism that does this for you.

[drakenclimber]

When adding a "strict" rule (in other words, using the *_exact(...) rule APIs) if the libseccomp library can not add a rule exactly as supplied by the caller it will fail. The non-strict/non-exact rule APIs allow the libseccomp library to do some minor "munging" to enable the operation to succeed where "munging" typically means arch/ABI related adjustments (e.g. multiplexed syscalls).

Thanks! That's how I was reading the situation, but it was nice to hear the official explanation. Much appreciated.

I guess I don't understand how this worked for @cpaelzer in v2.4.1 and then broke in v2.4.2. This code didn't change between v.2.4.1 and v2.4.2. (There is one commit that went into v2.4.0 that moved strict into struct db_api_rule_list, but the actual handling of strict didn't change.)

[cpaelzer]

@drakenclimber IIRC this was dependent on kernel and libseccomp. At the time I think I had a 5.3 kernel and on that the switch of 2.4.1->2.4.2 broke it. I got the same helpful explanation about exact by @pcmoore and eventually ended up changing the systemd behavior that was broken by the change.

We never tracked down the root cause changing it in 2.4.2 (see here ) but since the fail was the expected behavior we decided to fix it in systemd.

[drakenclimber]

Thanks for the help, @cpaelzer. I think we've done what we can - both in systemd and here in libseccomp.

I'll close this one out for now. Feel free to reach out if you have any further issues.