[BUG] RPM add repo_gpgcheck=1 in YUM repo file and enable installation on FIPS mode

[peterzhuamazon]

Initially discussed in the community forum:
https://forum.opensearch.org/t/rpm-distributions-delayed-again/6994/45

1. Our YUM repo file does not include repo_gpgcheck=1 means our signed repomd.xml is not being verified during installation.
2. As of now FIPS mode enabled machine cannot enable installation of our signed rpm, testing centos8/rockylinux8:

$ fips-mode-setup --check
Installation of FIPS modules is not completed.

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
    Header SHA1 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
    MD5 digest: OK

$ sudo fips-mode-setup --enable
Kernel initramdisks are being regenerated. This might take some time.
/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.

/etc/dracut.conf.d/xen.conf:add_drivers+="xen-netfront xen-blkfront "

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.

Setting system policy to FIPS
Note: System-wide crypto policies are applied on application start-up.
It is recommended to restart the system for the change of policies
to fully take place.
FIPS mode will be enabled.
Please reboot the system for the setting to take effect.

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
    Header SHA1 digest: OK
    Payload SHA256 digest: NOTFOUND
    V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
    MD5 digest: NOTFOUND

Confirmed that in FIPS enabled mode we indeed need payload SHA256 to be there and probably MD5 as well.
It shows NOKEY because I havent import the public subkey into gpg.

Thanks.

[peterzhuamazon]

The 1st task of adding repo_gpgcheck=1 is done and updated in production bucket.
This allows the repo meta to be verified before the download of rpm even happens.

Thanks.

[peterzhuamazon]

Seems like our signing method is ok, but during the build with rpmbuild through spec files these macros are missing for the digest to be generated:

%define _source_filedigest_algorithm 8
%define _binary_filedigest_algorithm 8

This also confirms that signing must happen on a rpm version 4.12+ which rockylinux8 is fine with 4.14.

[peterzhuamazon]

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    MD5 digest: OK

[peterzhuamazon]

Test install FIPS mode enabled on CentOS8:

$ fips-mode-setup --check
FIPS mode is enabled.

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: OK

$ sudo yum install opensearch-1.3.2-linux-x64.rpm
Last metadata expiration check: 2:05:24 ago on Fri 06 May 2022 11:33:46 PM UTC.
Dependencies resolved.
============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Architecture                                                              Version                                                                      Repository                                                                       Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                    x86_64                                                                    1.3.2-1                                                                      @commandline                                                                    369 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total size: 369 M
Installed size: 604 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                                                                    1/1
  Running scriptlet: opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Installing       : opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                          1/1
  Running scriptlet: opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                          1/1
[opensearch.conf:1] Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log

[/usr/lib/tmpfiles.d/opensearch.conf:1] Line references path below legacy directory /var/run/, updating /var/run/opensearch → /run/opensearch; please update the tmpfiles.d/ drop-in file accordingly.

  Verifying        : opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                          1/1

Installed:
  opensearch-1.3.2-1.x86_64

Complete!

[peterzhuamazon]

After many tries seems resolved now.

Manually build based on OS 2713 OSD 3032 on 1.3.2 version for x64:

OS: https://ci.opensearch.org/ci/dbc/tests/rpm-fips/opensearch-1.3.2-linux-x64-fips-enabled.rpm
OSD: https://ci.opensearch.org/ci/dbc/tests/rpm-fips/opensearch-dashboards-1.3.2-linux-x64-fips-enabled.rpm

@justchris1 please try these out and let me know if they runs on your system.

cc: @CEHENKLE @bbarani

[peterzhuamazon]

PR:

• Add rpm digest generation so FIPS enabled host can install rpm #2102

[justchris1]

@justchris1 please try these out and let me know if they runs on your system.

Those RPMs pass verification and installed! Thanks for the updates!

[peterzhuamazon]

Next Step:

1. Get signing pipeline ready with rpm signing.
2. Assemble on opensearch needs to separate from building like in dashboards, as the assemble part needs to happen in RockyLinux8 for higher RPM version while our build image needs to happen in CentOS7 for k-NN compatibility.
3. All staging rpm and yum needs to be signed, and the rpm validation workflow needs to support verifiying signature and install on a fips-enabled server.