FIPS module checksums: add scripts and Makefile rule

[levitte]

This adds the following scripts:

util/lang-compress.pl:

Compress source code, which language is determined by the first argument.
For the moment, we know 'perl' (perlasm source code), 'C' (C source code)
and 'S' (Assembler with C preprocessor directives).
This removes comments and empty lines, and compresses series of horizontal
spaces to one single space in the languages where that's appropriate.

util/fips-checksums.sh:

Takes source file names as arguments, pushes them through
util/lang-compress.pl and unifdef with FIPS_MODE defined, and calculates
the checksum on the result.

Fixes #13130

[levitte]

Ping @mattcaswell, @t-j-h, @paulidale

[mspncp]

FWIW: I was able to build unifdef using Microsoft Visual Studio Professional 2019.

After cloning,

git clone http://dotat.at/git/unifdef.git

you need to run the reversion.sh script (e.g. using git bash) as follows,

msp@msp-nbw10 MINGW64 /c/src/unifdef (master)
$ ./scripts/reversion.sh

before building the solution

unifdef\win32\unifdef.sln

otherwise Visual Studio will complain about a missing version.h file.

[levitte]

I've done a major rework of this branch. Instead of magically figuring out the source files for any configuration, we only collect source files for the default and the no-asm configuration and calculates checksums over those files, period.

This should be more or less done at this point. I'll watch the CIs for confirmation, then I'll make this non-WIP.

[mattcaswell]

Hmmm...the "make update" CI failure in fips-sources.checksums is presumably because of changes in the master branch??

[levitte]

Ah! I was wondering what was going on...

[levitte]

BTW, how come we have some sources from ssl/ in the FIPS module?

[levitte]

Rebasing wasn't enough, of course, I should have re-generated the checksums [facepalm]

[levitte]

Yay, check-update much happier now.

Quick, approve before master changes again! 😉

[mattcaswell]

BTW, how come we have some sources from ssl/ in the FIPS module?

Because there is now some "padding" related code which we now do provider side. Partly for the reason that it needs to be inside the FIPS boundary. However we can't entirely remove the same code from libssl, e.g. for backwards compatibility in case we are using an engine and no provider is involved.

[t8m]

I suppose we do not want to merge this earlier than just before the beta1 release. Otherwise we would have to update the checksums whenever we touch the fips module code, which still can commonly happen in PRs before beta1.

[mattcaswell]

Otherwise we would have to update the checksums whenever we touch the fips module code, which still can commonly happen in PRs before beta1.

Is that really too much of a burden? I'd prefer to get this in as soon as possible.

[levitte]

I suppose we do not want to merge this earlier than just before the beta1 release. Otherwise we would have to update the checksums whenever we touch the fips module code, which still can commonly happen in PRs before beta1.

I'm ambivalent. After all, it's not a bad idea to exercise this a few times before things go to stillness...

[t8m]

LGTM. I would wait with merging this at least for discussion in OTC as any two PRs touching the FIPS source files will inevitably have merge conflict.

[levitte]

Merged

841a438 Add OpenSSL::Config::Query and use it in configdata.pm
27ca03e Unix build file: Add a target to create providers/fips.module.sources
be22315 FIPS module checksums: add scripts and Makefile rule
49f699b GitHub CI: ensure that unifdef is installed
f97bc7c [TEMPORARY] make 'make update' verbose in ci.yml

[h-vetinari]

Perhaps one of the most august 3.0 PRs merged 🥳
Even managed to celebrate its 2nd birthday yesterday 😅