Skip to content

Conversation

@PlaidCat
Copy link
Collaborator

Update process (This kernel CentOS base for 5.14.0-570)

  • Kernel History Rebuild Process for all src.rpms hosted by RESF
  • Create fips-9-compliant/5.14.0-570.X.1.el8_10 branch
  • Check if any maintained code is included in the new el release.
  • Cherry-pick all code from previous branch into new branch (skipping unneeded code)
    • Fix conflicts as they arise
  • Build and Test

FIPS checks

[rolling release update] Checking for FIPS protected changes between the common tag and HEAD
[rolling release update] Checking for FIPS protected changes
[rolling release update] Getting SHAS c18abab0e029..HEAD
[rolling release update] Number of commits to check:  15
[rolling release update] Checking modifications of shas
[rolling release update] Checked 1 of 15 commits
[rolling release update] Checked 2 of 15 commits
[rolling release update] Checked 3 of 15 commits
[rolling release update] Checked 4 of 15 commits
[rolling release update] Checked 5 of 15 commits
[rolling release update] Checked 6 of 15 commits
[rolling release update] Checked 7 of 15 commits
[rolling release update] Checked 8 of 15 commits
[rolling release update] Checked 9 of 15 commits
[rolling release update] Checked 10 of 15 commits
[rolling release update] Checked 11 of 15 commits
[rolling release update] Checked 12 of 15 commits
[rolling release update] Checked 13 of 15 commits
[rolling release update] Checked 14 of 15 commits
[rolling release update] Checked 15 of 15 commits
[rolling release update] 0 of 15 commits have FIPS protected changes

Rolling Release

[jmaple@devbox code]$ cat RR.FIPS.resf_kernel-5.14.0-570.55.1.el9_6.log
[rolling release update] Rolling Product:  fips-9-compliant
[rolling release update] Checking out branch:  fips-9-compliant/5.14.0-570.52.1.el9_6
[rolling release update] Gathering all the RESF kernel Tags
[rolling release update] Found 23 RESF kernel tags
[rolling release update] Checking out branch:  rocky9_6
[rolling release update] Gathering all the RESF kernel Tags
[rolling release update] Found 24 RESF kernel tags
[rolling release update] Latest RESF tag sha:  b'c18abab0e029'
"c18abab0e0297b9f20d92e476a57f82b62b79ed3 Rebuild rocky9_6 with kernel-5.14.0-570.52.1.el9_6"
[rolling release update] Checking for FIPS protected changes between the common tag and HEAD
[rolling release update] Checking for FIPS protected changes
[rolling release update] Getting SHAS c18abab0e029..HEAD
[rolling release update] Number of commits to check:  15
[rolling release update] Checking modifications of shas
[rolling release update] Checked 1 of 15 commits
[rolling release update] Checked 2 of 15 commits
[rolling release update] Checked 3 of 15 commits
[rolling release update] Checked 4 of 15 commits
[rolling release update] Checked 5 of 15 commits
[rolling release update] Checked 6 of 15 commits
[rolling release update] Checked 7 of 15 commits
[rolling release update] Checked 8 of 15 commits
[rolling release update] Checked 9 of 15 commits
[rolling release update] Checked 10 of 15 commits
[rolling release update] Checked 11 of 15 commits
[rolling release update] Checked 12 of 15 commits
[rolling release update] Checked 13 of 15 commits
[rolling release update] Checked 14 of 15 commits
[rolling release update] Checked 15 of 15 commits
[rolling release update] 0 of 15 commits have FIPS protected changes
[rolling release update] Checking out old rolling branch:  fips-9-compliant/5.14.0-570.52.1.el9_6
[rolling release update] Finding the CIQ Kernel and Associated Upstream commits between the last resf tag and HEAD
[rolling release update] Last RESF tag sha:  b'c18abab0e029'
[rolling release update] Total commits in old branch: 15
[rolling release update] Checking out new base branch:  rocky9_6
[rolling release update] Finding the kernel version for the new rolling release
[rolling release update] New Branch to create: fips-9-compliant/5.14.0-570.55.1.el9_6
[rolling release update] Creating new branch: fips-9-compliant/5.14.0-570.55.1.el9_6
[rolling release update] Creating new branch for PR:  jmaple_fips-9-compliant/5.14.0-570.55.1.el9_6
[rolling release update] Creating Map of all new commits from last rolling release fork
[rolling release update] Total commits in new branch: 14
[rolling release update] Checking if any of the commits from the old rolling release are already present in the new base branch
[rolling release update] Found 0 duplicate commits to remove
[rolling release update] Applying 15 remaining commits to the new branch
  [1/15] 8f81cf5b92fb SUSE: patch: crypto-ecdh-implement-FIPS-PCT.patch
  [2/15] 33e6e21bc41b crypto: essiv - Zeroize keys on exit in essiv_aead_setkey()
  [3/15] f4c8c495bbda crypto: jitter - replace LFSR with SHA3-256
  [4/15] f753b75b8218 crypto: aead,cipher - zeroize key buffer after use
  [5/15] 7d87d16ac8eb crypto: ecdh - explicitly zeroize private_key
  [6/15] 0960806acec2 crypto: lib/mpi - Fix unexpected pointer access in mpi_ec_init
  [7/15] 76f4821d4b21 crypto: Kconfig - Make CRYPTO_FIPS depend on the DRBG being built-in
  [8/15] 83dd3ced0319 random: Restrict extrng registration to init time
  [9/15] 1e13146c5ea1 crypto: rng - Convert crypto_default_rng_refcnt into an unsigned int
  [10/15] 54084d69accf crypto: rng - Only allow the DRBG to register as "stdrng" in FIPS mode
  [11/15] c8f8f6a21a6a crypto: drbg - Align buffers to at least a cache line
  [12/15] d4aa0f529a45 crypto: rng - Fix priority inversions due to mutex locks
  [13/15] 5890b0f7d191 mm/gup: reintroduce pin_user_pages_fast_only()
  [14/15] 8ac02cebf7a1 crypto: rng - Implement fast per-CPU DRBG instances
  [15/15] e165cc8ceee1 configs: Ensure FIPS settings defined
[rolling release update] Successfully applied all 15 commits

Build

[jmaple@devbox code]$ egrep -B 5 -A 5 "\[TIMER\]|^Starting Build" $(ls -t kbuild* | head -n1)
/mnt/code/kernel-src-tree-build
Running make mrproper...
[TIMER]{MRPROPER}: 6s
x86_64 architecture detected, copying config
'configs/kernel-x86_64-rhel.config' -> '.config'
Setting Local Version for build
CONFIG_LOCALVERSION="-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f"
Making olddefconfig
--
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
Starting Build
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_32.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_64.h
  SYSHDR  arch/x86/include/generated/uapi/asm/unistd_x32.h
  SYSTBL  arch/x86/include/generated/asm/syscalls_32.h
  SYSHDR  arch/x86/include/generated/asm/unistd_32_ia32.h
--
  BTF [M] sound/usb/snd-usb-audio.ko
  LD [M]  sound/x86/snd-hdmi-lpe-audio.ko
  LD [M]  sound/xen/snd_xen_front.ko
  BTF [M] sound/x86/snd-hdmi-lpe-audio.ko
  BTF [M] sound/xen/snd_xen_front.ko
[TIMER]{BUILD}: 1703s
Making Modules
  INSTALL /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/arch/x86/crypto/blake2s-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/arch/x86/crypto/blowfish-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko
  INSTALL /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/arch/x86/crypto/camellia-aesni-avx2.ko
--
  INSTALL /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/sound/xen/snd_xen_front.ko
  STRIP   /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/sound/xen/snd_xen_front.ko
  SIGN    /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/sound/virtio/virtio_snd.ko
  SIGN    /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+/kernel/sound/xen/snd_xen_front.ko
  DEPMOD  /lib/modules/5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+
[TIMER]{MODULES}: 7s
Making Install
sh ./arch/x86/boot/install.sh 5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+ \
        arch/x86/boot/bzImage System.map "/boot"
[TIMER]{INSTALL}: 22s
Checking kABI
kABI check passed
Setting Default Kernel to /boot/vmlinuz-5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+ and Index to 1
Hopefully Grub2.0 took everything ... rebooting after time metrices
[TIMER]{MRPROPER}: 6s
[TIMER]{BUILD}: 1703s
[TIMER]{MODULES}: 7s
[TIMER]{INSTALL}: 22s
[TIMER]{TOTAL} 1744s
Rebooting in 10 seconds

KSelfTests

[jmaple@devbox code]$ ~/workspace/auto_kernel_history_rebuild/Rocky10/rocky10/code/get_kselftest_diff.sh
kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.33.2.el9_6-b4790268d6a9+.log
317
kselftest.5.14.0-jmaple_sig-cloud-9_5.14.0-570.39.1.el9_6-f6a810230c4c+.log
318
kselftest.5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+.log
318
kselftest.5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+.log
318
Before: kselftest.5.14.0-jmaple_rlc-9_5.14.0-570.55.1.el9_6-3f4cd04f0c2c+.log
After: kselftest.5.14.0-jmaple_fips-9-compliant_5.14.0-570.55.1.el9_6-516395f+.log
Diff:
-ok 7 selftests: timers: raw_skew
+ok 7 selftests: timers: raw_skew # SKIP

jallisonciq and others added 15 commits October 28, 2025 15:18
Signed-off-by: Jeremy Allison <jallison@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
In essiv_aead_setkey(), use the same logic as crypto_authenc_esn_setkey()
to zeroize keys on exit.

[Sultan: touched up commit message]

Signed-off-by: Jason Rodriguez <jrodriguez@ciq.com>
Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
        Using the kernel crypto API, the SHA3-256 algorithm is used as
        conditioning element to replace the LFSR in the Jitter RNG. All other
        parts of the Jitter RNG are unchanged.

        The application and use of the SHA-3 conditioning operation is identical
        to the user space Jitter RNG 3.4.0 by applying the following concept:

        - the Jitter RNG initializes a SHA-3 state which acts as the "entropy
          pool" when the Jitter RNG is allocated.

        - When a new time delta is obtained, it is inserted into the "entropy
          pool" with a SHA-3 update operation. Note, this operation in most of
          the cases is a simple memcpy() onto the SHA-3 stack.

        - To cause a true SHA-3 operation for each time delta operation, a
          second SHA-3 operation is performed hashing Jitter RNG status
          information. The final message digest is also inserted into the
          "entropy pool" with a SHA-3 update operation. Yet, this data is not
          considered to provide any entropy, but it shall stir the entropy pool.

        - To generate a random number, a SHA-3 final operation is performed to
          calculate a message digest followed by an immediate SHA-3 init to
          re-initialize the "entropy pool". The obtained message digest is one
          block of the Jitter RNG that is returned to the caller.

        Mathematically speaking, the random number generated by the Jitter RNG
        is:

        aux_t = SHA-3(Jitter RNG state data)

        Jitter RNG block = SHA-3(time_i || aux_i || time_(i-1) || aux_(i-1) ||
                                 ... || time_(i-255) || aux_(i-255))

        when assuming that the OSR = 1, i.e. the default value.

        This operation implies that the Jitter RNG has an output-blocksize of
        256 bits instead of the 64 bits of the LFSR-based Jitter RNG that is
        replaced with this patch.

        The patch also replaces the varying number of invocations of the
        conditioning function with one fixed number of invocations. The use
        of the conditioning function consistent with the userspace Jitter RNG
        library version 3.4.0.

        The code is tested with a system that exhibited the least amount of
        entropy generated by the Jitter RNG: the SiFive Unmatched RISC-V
        system. The measured entropy rate is well above the heuristically
        implied entropy value of 1 bit of entropy per time delta. On all other
        tested systems, the measured entropy rate is even higher by orders
        of magnitude. The measurement was performed using updated tooling
        provided with the user space Jitter RNG library test framework.

        The performance of the Jitter RNG with this patch is about en par
        with the performance of the Jitter RNG without the patch.

        Signed-off-by: Stephan Mueller <smueller@chronox.de>
        Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

            Back-port of commit bb897c5
            Author: Stephan Müller <smueller@chronox.de>
            Date:   Fri Apr 21 08:08:04 2023 +0200

Signed-off-by: Jeremy Allison <jallison@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
    I.G 9.7.B for FIPS 140-3 specifies that variables temporarily holding
    cryptographic information should be zeroized once they are no longer
    needed. Accomplish this by using kfree_sensitive for buffers that
    previously held the private key.

    Signed-off-by: Hailey Mothershead <hailmo@amazon.com>
    Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

        Back-ported from commit 23e4099
        Author: Hailey Mothershead <hailmo@amazon.com>
        Date:   Mon Apr 15 22:19:15 2024 +0000

Signed-off-by: Jeremy Allison <jallison@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
private_key is overwritten with the key parameter passed in by the
caller (if present), or alternatively a newly generated private key.
However, it is possible that the caller provides a key (or the newly
generated key) which is shorter than the previous key. In that
scenario, some key material from the previous key would not be
overwritten. The easiest solution is to explicitly zeroize the entire
private_key array first.

Note that this patch slightly changes the behavior of this function:
previously, if the ecc_gen_privkey failed, the old private_key would
remain. Now, the private_key is always zeroized. This behavior is
consistent with the case where params.key is set and ecc_is_key_valid
fails.

Signed-off-by: Joachim Vandersmissen <git@jvdsn.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
[ Upstream commit ba3c557 ]

When the mpi_ec_ctx structure is initialized, some fields are not
cleared, causing a crash when referencing the field when the
structure was released. Initially, this issue was ignored because
memory for mpi_ec_ctx is allocated with the __GFP_ZERO flag.
For example, this error will be triggered when calculating the
Za value for SM2 separately.

Fixes: d58bb7e ("lib/mpi: Introduce ec implementation to MPI library")
Cc: stable@vger.kernel.org # v6.5
Signed-off-by: Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
When FIPS mode is enabled (via fips=1), there is an absolute need for the
DRBG to be available. This is at odds with the fact that the DRBG can be
built as a module when in FIPS mode, leaving critical RNG functionality at
the whims of userspace.

Userspace could simply rmmod the DRBG module, or not provide it at all and
thus a different stdrng algorithm could be used without anyone noticing.

Additionally, when running a FIPS-enabled userspace, modprobe itself may
perform a getrandom() syscall _before_ loading a given module. As a result,
there's a possible deadlock scenario where the RNG core (crypto/rng.c)
initializes _before_ the DRBG, thereby installing its getrandom() override
without having an stdrng algorithm available. Then, when userspace calls
getrandom() which redirects to the override in crypto/rng.c,
crypto_alloc_rng("stdrng") invokes the UMH (modprobe) to load the DRBG
(which is aliased to stdrng). And *then* that modprobe invocation gets
stuck at getrandom() because there's no stdrng algorithm available!

There are too many risks that come with allowing the DRBG and RNG core to
be modular for FIPS mode. Therefore, make CRYPTO_FIPS require the DRBG to
be built-in, which in turn makes the DRBG require the RNG core to be
built-in. That way, it's guaranteed for these drivers to be built-in when
running in FIPS mode.

Also clean up the CRYPTO_FIPS option name and remove the CRYPTO_ANSI_CPRNG
dependency since it's obsolete for FIPS now.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>

Signed-off-by: Jonathan Maple <jmaple@ciq.com>
It is technically a risk to permit extrng registration by modules after
kernel init completes. Since there is only one user of the extrng interface
and it is imperative that it is the _only_ registered extrng for FIPS
compliance, restrict the extrng registration interface to only permit
registration during kernel init and only from built-in drivers.

This also eliminates the risks associated with the extrng interface itself
being designed to solely accommodate a single registration, which would
therefore permit the registered extrng to be overridden or even removed by
an unrelated module.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
There is no reason this refcount should be a signed int. Convert it to an
unsigned int, thereby also making it less likely to ever overflow.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
In FIPS mode, the DRBG must take precedence over all stdrng algorithms.
The only problem standing in the way of this is that a different stdrng
algorithm could get registered and utilized before the DRBG is registered,
and since crypto_alloc_rng() only allocates an stdrng algorithm when
there's no existing allocation, this means that it's possible for the wrong
stdrng algorithm to remain in use indefinitely.

This issue is also often impossible to observe from userspace; an RNG other
than the DRBG could be used somewhere in the kernel and userspace would be
none the wiser.

To ensure this can never happen, only allow stdrng instances from the DRBG
to be registered when running in FIPS mode. This works since the previous
commit forces the DRBG to be built into the kernel when CONFIG_CRYPTO_FIPS
is enabled, so the DRBG's presence is guaranteed when fips_enabled is true.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
None of the ciphers used by the DRBG have an alignment requirement; thus,
they all return 0 from .crypto_init, resulting in inconsistent alignment
across all buffers.

Align all buffers to at least a cache line to improve performance. This is
especially useful when multiple DRBG instances are used, since it prevents
false sharing of cache lines between the different instances.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Since crypto_devrandom_read_iter() is invoked directly by user tasks and is
accessible by every task in the system, there are glaring priority
inversions on crypto_reseed_rng_lock and crypto_default_rng_lock.

Tasks of arbitrary scheduling priority access crypto_devrandom_read_iter().
When a low-priority task owns one of the mutex locks, higher-priority tasks
waiting on that mutex lock are stalled until the low-priority task is done.

Fix the priority inversions by converting the mutex locks into rt_mutex
locks which have PI support.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
Like pin_user_pages_fast(), but with the internal-only FOLL_FAST_ONLY flag.

This complements the get_user_pages*() API, which already has
get_user_pages_fast_only().

Note that pin_user_pages_fast_only() used to exist but was removed in
upstream commit edad1bb ("mm/gup: remove pin_user_pages_fast_only()")
due to it not having any users.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
When the kernel is booted with fips=1, the RNG exposed to userspace is
hijacked away from the CRNG and redirects to crypto_devrandom_read_iter(),
which utilizes the DRBG.

Notably, crypto_devrandom_read_iter() maintains just two global DRBG
instances _for the entire system_, and the two instances serve separate
request types: one instance for GRND_RANDOM requests (crypto_reseed_rng),
and one instance for non-GRND_RANDOM requests (crypto_default_rng). So in
essence, for requests of a single type, there is just one global RNG for
all CPUs in the entire system, which scales _very_ poorly.

To make matters worse, the temporary buffer used to ferry data between the
DRBG and userspace is woefully small at only 256 bytes, which doesn't do a
good job of maximizing throughput from the DRBG. This results in lost
performance when userspace requests >256 bytes; it is observed that DRBG
throughput improves by 70% on an i9-13900H when the buffer size is
increased to 4096 bytes (one page). Going beyond the size of one page up to
the DRBG maximum request limit of 65536 bytes produces diminishing returns
of only 3% improved throughput in comparison. And going below the size of
one page produces progressively less throughput at each power of 2: there's
a 5% loss going from 4096 bytes to 2048 bytes and a 9% loss going from 2048
bytes to 1024 bytes.

Thus, this implements per-CPU DRBG instances utilizing a page-sized buffer
for each CPU to utilize the DRBG itself more effectively. On top of that,
for non-GRND_RANDOM requests, the DRBG's operations now occur under a local
lock that disables preemption on non-PREEMPT_RT kernels, which not only
keeps each CPU's DRBG instance isolated from another, but also improves
temporal cache locality while the DRBG actively generates a new string of
random bytes.

Prefaulting one user destination page at a time is also employed to prevent
a DRBG instance from getting blocked on page faults, thereby maximizing the
use of the DRBG so that the only bottleneck is the DRBG itself.

Signed-off-by: Sultan Alsawaf <sultan@ciq.com>
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
We want to hard set the x86_64 FIPS required configs rather than rely on
default settings in the kernel, should these ever change without our
knowing it would not be something we would have actively checked.

The configs are a limited set of configs that is expanded out when
building using `make olddefconfig` a common practice in kernel building.

Note had to manually add the following since its normaly set by the RPM
build process.
CONFIG_CRYPTO_FIPS_NAME="Rocky Linux 9 Kernel Cryptographic API"

Signed-off-by: Jonathan Maple <jmaple@ciq.com>
@PlaidCat PlaidCat requested a review from a team October 28, 2025 21:20
@PlaidCat PlaidCat self-assigned this Oct 28, 2025
@bmastbergen bmastbergen self-requested a review October 29, 2025 13:52
@PlaidCat PlaidCat merged commit 516395f into fips-9-compliant/5.14.0-570.55.1.el9_6 Oct 29, 2025
4 checks passed
@PlaidCat PlaidCat deleted the jmaple_fips-9-compliant/5.14.0-570.55.1.el9_6 branch October 29, 2025 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

9 participants