cryptography: safely vendor and port modern libraries

[Shamar]

Given we want to replace libsec, we need a modern well established crypto library that will replace it.
Since it's required by the kernel (random number generation, devtls and so on), it's going to be part of our core system.

Thus we need

1. to native port it, with the long term goal of contributing the port to the upstream community
   • to native port its dependencies, with the long term goal of contributing the port the respective upstream communities
   • to keep control on the version we use, avoiding to automatically include the "latest" version
2. to minimize our impact on code that we do not really understand
3. to be able to add our own regression tests
4. to be able to run all tests provided by the upstream libraries that apply to Harvey

All this should be achieved

1. preserving reproducible builds and our ability to use git bisect
2. adopting a trust-less semi automatic and verifiable process during the updates of the library, to prevent
   • the introduction of security flaws in the imported sources (that could pass unnoticed during a large update to a new upstream version)
   • any delay of critical security updates that could advantage an attacker
   • the introduction of security flaws in the port scripts (that could pass unnoticed during a large update to a new upstream version)

To this aim we decided to code a set of Go utilities that can automatically vendor a source archive according to directives described in a json file (from download to checksum, including a final git commit with a standardized message) and automatically verify that the code imported matches the one contained in the archive declared in the json,
This tool would prevent the first obvious threat described above and will reduce the risk of the second, as it is pointless to discuss an automated commit that builds, passes the regression tests and pass the source verification: the only check that remains to reviewers (on this regard) is to ensure that the URI of the archive prescribed in the json and its digests actually match the intended one.

To address the third possible threat in an (almost) trust-less fashion, we need to be able to easily review the port scripts when the are updated. This unfortunately precludes the use of patch files because they are sensible to many small non-semantic changes that can occur during a version update and that, all together, can hide to a human reviewer a single semantic change (think for example to the removal of a single null check hidden in a large patch-set addressing whitespace changes, #include replacements and so on).

For this reason we are going to use dedicated easy to read scripts for every single transformations that the sources will require: for example during a review it is much easier to ensure that a remove-stdio-include.sed script behaves correctly, than the same transformation spread in dozen of patch files (or in one patch targeting multiple files).

In the long term, when Harvey will be a mainstream OS supported like any other by the upstream developers, the vendor tool and its automatic checker will be still useful for updates: we will only need to remove the porting scripts from the build,json.

[louy2]

A todo list of the libraries in need of porting may serve a better issue. Like so:

• zlib
• liblts

Source:

- [ ] zlib
- [ ] liblts

[rminnich]

no longer needed.