Add basic support for s390x #1297

uweigand · 2021-05-17T12:15:43Z

This adds basic support for Linux on s390x, using the fallback implementation of all algorithms and no assembler routines. This addresses #986 and makes "cargo test" fully pass on s390x.

There were two main changes I had to make to get this working:

Provide a bn_mul_mont fallback implementation. Note that while there is a fallback implementation on the Rust side for limbs_mont_mul, some elliptic curve code directly calls bn_mul_mont on the C side, where there is currently no fallback. However, given that we have a bn_from_montgomery_in_place fallback, it is straightforward to add bn_mul_mont (along the lines of the limbs_mont_mul fallback).
Support big-endian platforms. This consists of adding a number of endian access primitives in internal.h, and using them where required, currently in aes_nohw.c, poly1305.c, and the p256_scalar_bytes_from_limbs routine (for some reason, the p384 version of that routine is already endian-agnostic). I have chosen to use a plain C implementation of the primitives rather than bswap intrinsics, because that is easier to maintain, doesn't require platform #ifdef's, and still compiles to good code with any recent compiler.

@briansmith, please let me know if this approach makes sense to you or if this should be done differently.

Once this basic (unoptimized) support is in, we can then build on it by adding assembler optimizations (they're already present in OpenSSL, but would need to be copied over and adapted).

uweigand · 2021-06-09T11:24:20Z

Ping? Any comments on this approach?

briansmith · 2021-06-23T17:00:12Z

crypto/internal.h

-#if defined(__GNUC__) && __GNUC__ >= 2
-static inline uint32_t CRYPTO_bswap4(uint32_t x) {
-  return __builtin_bswap32(x);
+static inline uint32_t CRYPTO_read_le32(const uint8_t *p) {


I will do a merge from the BoringSSL sources to include these functions.

Note that the BoringSSL versions (CRYPTO_load_u32_le etc.) only work correctly on little-endian hosts. (I guess this goes back to the BoringSSL team's decision to not support big-endian hosts at all.)

The versions I've provided in this PR work on any host, little-endian or big-endian.

Yes, I know if I merge the BoringSSL versions then they'll need to be modified to support big endian. I also am very fond of the endian-neutral implementation style you chose. However, I remember that one compiler I tested (I think MSVC) didn't recognize those idioms as of a couple years ago. Also, it's easier to make use of the BoringSSL team's performance/benchmarking work and the AWS/Galois team's verification work if we keep the code the same for little-endian platforms. That's why I'm thinking to merge in the BoringSSL teams's functions and then we can modify them to support big-endian.

Ok, I see. That should work as well, thanks.

briansmith · 2021-06-23T17:01:17Z

crypto/fipsmodule/ec/p256_shared.h

@@ -50,7 +50,14 @@ typedef unsigned char P256_SCALAR_BYTES[33];

 static inline void p256_scalar_bytes_from_limbs(
    P256_SCALAR_BYTES bytes_out, const BN_ULONG limbs[P256_LIMBS]) {
-  OPENSSL_memcpy(bytes_out, limbs, 32);


I need to go through all the remaining uses of OPENSSL_memcpy and make sure they're OK.

crypto/fipsmodule/bn/montgomery.c

briansmith · 2021-06-23T17:07:05Z

crypto/fipsmodule/bn/montgomery.c

+    !defined(OPENSSL_ARM) && !defined(OPENSSL_AARCH64)
+void bn_mul_mont(BN_ULONG *rp, const BN_ULONG *ap, const BN_ULONG *bp,
+                 const BN_ULONG *np, const BN_ULONG *n0, size_t num) {
+  Limb tmp[2 * num];


I agree this makes sense. This would be the first place we've used variable-length arrays in ring though, which makes me hesitate. I will circle back to this.

I guess we could use a global maximum limit along the lines of MODULUS_MAX_LIMBS in src/arithmetic/bigint.rs. I'd be happy to implement whatever change you prefer -- just let me know.

vimalk78 · 2021-09-29T07:37:04Z

hi @uweigand @briansmith, i am also interested in getting ring supported for s390x. Let me know if i can contribute in some way.
From this PR, I was able to build ring using cross build --release --target s390x-unknown-linux-gnu
btw, how do we get the above tests to run on the PR?

vimalk78 · 2021-10-14T15:30:22Z

ping @briansmith @uweigand

uweigand · 2022-02-21T14:07:14Z

Hi @briansmith, let me try to make another attempt to get this going.

I've just updated the patch to use a different style of endian conversion, as you suggested. This keeps the CRYPTO_bswap4 implementation from boringssl and adds the corresponding CRYPTO_bswap8 implementation from there as well. The other endian-specific accessors are now all defined in term of CRYPTO_bswap4 / CRYPTO_bswap8 plus OPENSSL_memcpy (to handle any alignment issues if necessary).

Does this address your concerns?

If there's anything else I can do to help get this reviewed and merged, please let me know.

uweigand · 2022-05-04T15:49:39Z

Fixed merge conflict against current mainline. Any comments?

Xynnn007 · 2022-11-04T08:43:07Z

Hi @briansmith , can this pr be merged now?

uweigand · 2022-11-04T09:01:08Z

Rebased and fixed merge conflicts against current mainline.

uweigand · 2022-11-10T13:29:55Z

Added support for cross-building and testing as requested in #1555.
Split into three logically distinct commits.

uweigand · 2022-11-14T21:39:16Z

PR updated after #1558 was merged.

uweigand · 2023-09-27T06:32:48Z

Please rebase on top of main and resubmit.

Done.

Why did the earlier version of this have changes to p256_shared.h but the newest one doesn't?

Huh, I thought I added a comment but it seems to have disappeared. I've removed that change as re-testing showed it is no longer necessary in the current code base.

The change was to fix an endian issue in p256_scalar_bytes_from_limbs, but that function is now only called from the crypto/fipsmodule/ec/p256-nistz.c file, which in turn is only even built at all on the (little-endian) AARCH64 and X86_64 targets. So there is no need for this file to support big-endian platforms, and any code added there would be untested anyway.

briansmith · 2023-09-29T20:19:48Z

Good news:
The refactoring to fix the --no-default-features build was merged into main.

Less good news:
There's a merge conflict in crypto/internal.h from the latest BoringSSL merge. Also, PR #1663 moved the stuff in base.h to target.h. I will merge PR #1663 ASAP so you can rebase this on top of main for the release. I am hoping to finish the BoringSSL merge today so there shouldn't be any new conflicts.

uweigand · 2023-09-30T00:15:33Z

@briansmith: Rebased against current mainline to fix both merge conflicts.

uweigand · 2023-09-30T13:37:04Z

@briansmith - this is unfortunate. The recently added test_constant_time test now fails on s390x. However, the test passes when run on a native machine, and it turns out this is actually a bug in qemu - and one that was already fixed here: https://lists.gnu.org/archive/html/qemu-devel/2023-05/msg06965.html

For some reason, clang chooses to use this particular z13 instruction (LOCFHR - which is generally quite rare) when implementing the bssl_constant_time_test_main routine.

I'm wondering what the best way to proceed here would be. I can imagine a number of options:

Just disable this test on s390x (either generally, or only when running on qemu)
Use a qemu that has the above fix - but that would mean building qemu yourself, or switching to Ubuntu 23.04 (if github actions already support that?), as older distro versions aren't recent enough
Tell clang to not emit this instruction, e.g. by selecting an older target architecture (adding -march=zEC12 to CFLAGS_s390x_unknown_linux_gnu works for me)

Any preferences?

briansmith · 2023-09-30T15:07:03Z

Just disable this test on s390x (either generally, or only when running on qemu)

Let's pick this choice. Note that this test isn't new. It is one of the oldest ones, which is why it is written in C!

uweigand · 2023-09-30T16:01:30Z

Just disable this test on s390x (either generally, or only when running on qemu)

Let's pick this choice. Note that this test isn't new. It is one of the oldest ones, which is why it is written in C!

Huh, interesting. Not sure what exactly triggered this to show up just now - the test seems to have passed before the recent boringssl merges. Anyway, with things like that, some random changes in surrounding code may trigger different instruction selection choices.

I've updated the patch now to exclude this one test on s390x.

briansmith · 2023-09-30T16:27:49Z

Thanks. Maybe IBM can send me one of these 390x boxes? Then my daughter could do her homework on it when I'm not using it for testing. :)

uweigand · 2023-09-30T17:25:07Z

Thanks. Maybe IBM can send me one of these 390x boxes? Then my daughter could do her homework on it when I'm not using it for testing. :)

Well, we're not usually sending out boxes, but if you're actually interested, you could in fact get free access to s390x machines for the purpose of open-source development and testing :-)

See here: https://www.ibm.com/community/z/open-source/virtual-machines-request/

uweigand · 2023-09-30T19:00:56Z

Rebased once again after the riscv64 merge.

briansmith

Your PR is the next one to merge after the requested changes are made. Sorry about the many conflicts and delays.

briansmith · 2023-10-01T05:33:58Z

src/constant_time.rs

@@ -40,6 +40,7 @@ mod tests {
    use crate::{bssl, error};

    #[test]
+    #[cfg(not(target_arch = "s390x"))] // Triggers a qemu bug before 8.0.3


OK, this is not what I was expecting. I thought we'd have an #ifdef around a line or two within constant_time_test.cc.

Maybe instead we should change ci.yml so that we have something like:

- target: s390x-unknown-linux-gnu host_os: ubuntu-22.04 # XXX: `constant_time_test` fails when run under QEMU 8.0.2 and earlier # (https://link that you shared with me in your earlier comment). cargo_options: -- --skip constant_time_test

This way people would not ever run into the issue on bare metal testing and I could also play with it locally. WDYT?

I see, this is indeed nicer. I've changed the PR accordingly.

briansmith · 2023-10-01T05:35:13Z

.github/workflows/ci.yml

@@ -266,6 +267,9 @@ jobs:
          - target: x86_64-unknown-linux-gnu
            host_os: ubuntu-22.04

+          - target: s390x-unknown-linux-gnu
+            host_os: ubuntu-22.04


In the analogous PPC64LE PR I just merged, the author noted that they kept the alphabetical order intact. Could you please do the same throughout this PR? It's an arbitrary rule but it avoids people needing to wonder where to put things.

Used alphabetic order everywhere now.

briansmith · 2023-10-01T05:39:01Z

It seems like codecov doesn't understand s390x profiler output? Too bad, but we'll deal.

uweigand · 2023-10-01T07:10:59Z

It seems like codecov doesn't understand s390x profiler output? Too bad, but we'll deal.

Why do you say it doesn't understand the output? I saw changes reported for most of the new lines added (except for those in unused inline functions), and also for the now-skipped test_constant_time test ...

uweigand · 2023-10-01T07:34:33Z

Ah, the --skip test_constant_time unfortunately seems to break the full run:

     Running `qemu-s390x -L /usr/s390x-linux-gnu /home/uweigand/ring/target/s390x-unknown-linux-gnu/release/deps/aead-0c8df580230e6a2e --skip test_constant_time`
error: Found argument '--skip' which wasn't expected, or isn't valid in this context

It is passed to both tests and benchmarks, and benchmarks apparently do not expect the --skip argument.

I've now changed the patch to use -march=zEC12 instead. That seems the straightforward workaround (directly addresses the root cause that QEMU fails to correctly emulate a z13 instruction), only needs a change in one place, and also is used only when doing the QEMU-based cross testing.

What do you think? If you have any other suggestion, please let me know.

briansmith · 2023-10-01T21:33:22Z

Thanks!

uweigand · 2023-10-01T22:14:29Z

Thanks for your support in getting this merged!

briansmith reviewed Jun 23, 2021

View reviewed changes

raulcabello mentioned this pull request Jan 27, 2022

IBM z/Architecture (s390x) support kubewarden/kwctl#139

Closed

This was referenced Feb 4, 2022

C (or Rust) fallback for all functions that work for all little-endian targets #1455

Closed

Add support for ESP32 and RISC-V #1459

Closed

uweigand force-pushed the s390x branch from 352843c to 5cbdabc Compare February 21, 2022 14:03

uweigand force-pushed the s390x branch from 5cbdabc to 700fcca Compare May 4, 2022 15:48

uweigand mentioned this pull request Jun 10, 2022

[feature request]Do you have plan to support "s390x-linux" target? #1494

Closed

MabezDev mentioned this pull request Jul 27, 2022

Support Xtensa, RISC-V and ESP-IDF #1506

Closed

decathorpe mentioned this pull request Aug 1, 2022

support alternative TLS backends quinn-rs/quinn#1389

Closed

ecnelises mentioned this pull request Aug 25, 2022

No s390 support ? #1511

Closed

AidanConnelly mentioned this pull request Sep 19, 2022

Is this project actively maintained at the moment? #1516

Closed

This was referenced Oct 17, 2022

kata-ctl: Disable network check on s390x kata-containers/kata-containers#5439

Merged

kata-ctl: Re-enable network check on s390x kata-containers/kata-containers#5438

Closed

BbolroC mentioned this pull request Nov 3, 2022

CC | dependency error on kata-agent for s390x kata-containers/kata-containers#5582

Open

uweigand force-pushed the s390x branch from 700fcca to 2f50bb4 Compare November 4, 2022 09:00

uweigand force-pushed the s390x branch from 2f50bb4 to 8c0ead1 Compare November 10, 2022 13:28

uweigand mentioned this pull request Nov 10, 2022

Add support for big-endian targets #1555

Closed

briansmith mentioned this pull request Nov 12, 2022

Add MIPS support #1277

Closed

joseluisq mentioned this pull request Nov 13, 2022

Support for Linux PowerPC (PPC64LE) and S390x targets static-web-server/static-web-server#159

Merged

uweigand force-pushed the s390x branch from 8c0ead1 to 35d79db Compare November 14, 2022 21:38

uweigand force-pushed the s390x branch from 35d79db to e3e3ba1 Compare November 23, 2022 15:49

briansmith mentioned this pull request Sep 27, 2023

Add support and CI workflow for riscv64 #1627

Merged

uweigand force-pushed the s390x branch from 843b35c to 1164d0b Compare September 27, 2023 06:25

uweigand force-pushed the s390x branch from 1164d0b to 361eb6d Compare September 30, 2023 00:14

uweigand force-pushed the s390x branch from 361eb6d to 9ff7d9c Compare September 30, 2023 15:58

uweigand force-pushed the s390x branch from 9ff7d9c to bb4eb01 Compare September 30, 2023 19:00

briansmith mentioned this pull request Sep 30, 2023

Support 64-bit big endian PowerPC target #1594

Closed

briansmith requested changes Oct 1, 2023

View reviewed changes

Support big-endian platforms

20e1493

uweigand force-pushed the s390x branch from bb4eb01 to 8688eb3 Compare October 1, 2023 07:08

Basic support for s390x target

3062477

uweigand force-pushed the s390x branch from 8688eb3 to 3062477 Compare October 1, 2023 07:32

briansmith merged commit baa823b into briansmith:main Oct 1, 2023
166 of 168 checks passed

uweigand deleted the s390x branch October 1, 2023 22:13

stevenhorsman mentioned this pull request Oct 28, 2023

CCv0: Update image-rs, AA and td-shim kata-containers/kata-containers#7852

Merged

This was referenced Dec 19, 2023

Bump image-rs rev to include image-layer-ordering fix kata-containers/kata-containers#8670

Merged

image-rs re-introduce ring version that is incompatible with s390x? confidential-containers/guest-components#422

Closed

Add basic support for s390x #1297

Add basic support for s390x #1297

Conversation

uweigand commented May 17, 2021

uweigand commented Jun 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vimalk78 commented Sep 29, 2021

vimalk78 commented Oct 14, 2021

uweigand commented Feb 21, 2022

uweigand commented May 4, 2022

Xynnn007 commented Nov 4, 2022

uweigand commented Nov 4, 2022

uweigand commented Nov 10, 2022

uweigand commented Nov 14, 2022

uweigand commented Sep 27, 2023

briansmith commented Sep 29, 2023

uweigand commented Sep 30, 2023

uweigand commented Sep 30, 2023

briansmith commented Sep 30, 2023

uweigand commented Sep 30, 2023

briansmith commented Sep 30, 2023 • edited

uweigand commented Sep 30, 2023

uweigand commented Sep 30, 2023

briansmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

briansmith commented Oct 1, 2023

uweigand commented Oct 1, 2023

uweigand commented Oct 1, 2023

briansmith commented Oct 1, 2023

uweigand commented Oct 1, 2023

briansmith commented Sep 30, 2023 •

edited