Include digital signatures on project packages

[Gormartsen]

I propose to implement MD5 checksum for each packaged project.

It can be provided as a part of the title.

We need to use it via project installer to make sure that package is not hacked and replaced.

---

Advocates: @jlfranklin @quicksketch

---

Weekly update?

---

Proof of concept by quicksketch: backdrop-ops/backdropcms.org#456
Initial core PR by @jlfranklin: backdrop/backdrop#2449

Other related issues that also need to be done:

• Project Module API improvements: Allow other modules to include data in the project XML files backdrop-contrib/project#25
• Project Module Code Sign Support: Add support for Code Sign backdrop-contrib/project#29
• Update Installer Module: Installer should check digital signatures when downloading a package #3714

[Gormartsen]

Nice example is here:
https://github.com/syncthing/syncthing/releases/

[klonos]

👍 Security++

[jenlampton]

👍

[jlfranklin]

Ooh. SyncThing also GPG signs their releases.

MD5, while good for a lot of things, has been weakened enough that it shouldn't be used for making sure a package hasn't been intentionally hacked.

I like the idea, but I suggest a stronger hash algorithm... SHA256, maybe. Even better, GPG sign them or encourage module maintainers to do so.

[quicksketch]

Retitling this slightly. Like @jlfranklin, I think we can do better than md5, but if want to start with a simple hash and expand it out later, we can do that.

Research over in #2018 led me to https://paragonie.com/blog/2016/10/guide-automatic-security-updates-for-php-developers, which included many, many excellent ideas. One of my favorite ideas is using private/public keys for digitally signing the packages. Backdrop's packager would sign the packages, which would then be stored on GitHub as they are now. Backdrop core would include the public key for validating the packages.

The message of the package could even then be the md5 of the zip archive. So you'd both verify that the package came from a legitimate location, and that the zip file had not been tampered with in transit.

For the public/private keys, the author recommends using libsodium (https://download.libsodium.org/doc/) which is built into PHP 7.2 and higher (if included in the compilation). But as we support many older versions of PHP and even new installations might not have it built in, the author amazingly created a bundleable version of libsodium as a pure PHP package that can be included as a .phar file (660KB): https://github.com/paragonie/sodium_compat

Perhaps that's too much to handle all in one go, or perhaps not. If we're going through the work of updating the XML feeds on BackdropCMS.org to include an MD5, it could nearly as easily include a digital signature instead.

[klonos]

...GPG sign them or encourage module maintainers to do so.

If our packager can do that for them, then we should do that for them :) ...basically the plan that @quicksketch has laid out.

[quicksketch]

I've been making progress on this functionality. Adding MD5 capability to the Project Release module is fairly trivial. So that shouldn't be difficult at all.

Digital signing also doesn't appear to be too difficult, though we need to make some architecture decisions.

Using the Sodium Compat library at https://github.com/paragonie/sodium_compat, the basics of signing are provided in the README.md file:

<?php
require_once "/path/to/sodium_compat/autoload.php";

$alice_kp = sodium_crypto_sign_keypair();
$alice_sk = sodium_crypto_sign_secretkey($alice_kp);
$alice_pk = sodium_crypto_sign_publickey($alice_kp);

$message = 'This is a test message.';
$signature = sodium_crypto_sign_detached($message, $alice_sk);
if (sodium_crypto_sign_verify_detached($signature, $message, $alice_pk)) {
    echo 'OK', PHP_EOL;
} else {
    throw new Exception('Invalid signature');
}

Applying this to Backdrop would mean the following:

• We would generate a key-pair per the above code.
• The private key would be stored in a locked-down location on backdropcms.org server.
• The public key would be bundled directly into Backdrop core, probably under /core/misc/keys or in /core/modules/installer.
• When the packager runs on backdropcms.org, it would md5 hash the archive file, and store this md5 value in the project_release table.
• After doing the md5, the packager would also create a digital signature, using $signature = sodium_crypto_sign_detached($message, $secret_key). The "message" would be the md5 value of the archive. It would also store this signature in the project_release table.
• Both the MD5 and the signature will be added to the project XML feed, such as the one at https://updates.backdropcms.org/release-history/backdrop/1.x
• When the Installer module downloads a new module or release, it will first validate the MD5 of the archive. Then it will take the signature value, and decode it with sodium_crypto_sign_verify_detached($signature, $message, $alice_pk). Where the $message is the MD5 string again, and the public key is the one bundled with Backdrop core.
• Now we know that the package both has not been tampered with, and it was packaged exclusively through the private key that only exists on the BackdropCMS.org server.

Some additional thoughts:

• We should probably create not just one, but two key pairs. The first private key is stored on backdropcms.org. The second private key is a backup we would store only in an offline location. Both public keys would be bundled with Backdrop core. The backup would be used in the event the main private key becomes compromised, in which case we would regenerate all the signatures using the new key.
• sodium_crypto_sign_detached() creates a string that is made up of random characters, so it doesn't seem to have any real sane representation when viewed in a text file. Here's what a public key looks like:

çhú=!M�/Åv¼�{D½˜—æ>þœ¯Èfç�üð·N-ðµ˜B-ÀÂÙæ�CØs(Œé�‘4/C�PÜ¡�
�Ä�†þLµ˜B-ÀÂÙæ�CØs(Œé�‘4/C�PÜ¡�
�Ä�†þL

• As this string is likely to be interpretted incorrectly when transferred as text, we should encode it in some way before we store it. The AirShip CMS made by Paragonie provides some example as to how to do this, where they use base64 URL-safe encoding before storing and writing keys to disk.

In \ParagonIE\Halite\Asymmetric\Crypto::sign, the code to do the actual signing is:

    public static function sign(
        string $message,
        SignatureSecretKey $privateKey,
        $encoding = Halite::ENCODE_BASE64URLSAFE
    ): string {
        $signed = \sodium_crypto_sign_detached(
            $message,
            $privateKey->getRawKeyMaterial()
        );
        $encoder = Halite::chooseEncoder($encoding);
        if ($encoder) {
            return (string) $encoder($signed);
        }
        return (string) $signed;
    }

Where $message is an MD5 string, and $encoding is "base64urlsafe". They provide the option to use a different encoding, but I think for practical purposes, we can just assume a Base64 encoding in the BackdropCMS.org packager. I don't think we need to worry about making the encoded version URL-safe, but Backdrop does include a backdrop_base64_encode() function for URL-safe strings, but it does not have an equivalent function for decoding them.

[docwilmot]

Sounds good.

[jlfranklin]

Sounds like you're reinventing SSL signing, minus the identity features.

Where you have an offline key, SSL would use a CA key and cert. (It's easy enough to create a CA cert using OpenSSL.) The CA's key (private part) is kept offline, the cert itself (public part) is baked into Backdrop. A signing key is generated and that key/cert lives on backdropcms.org for signing packages. (Actually, it should live in a more secure signing server that has limited access to the internet, and only listens for signing requests from whitelisted sites like backdropcms.org.)

The signing cert can be recreated if (when) it is compromised, and the individual package certs can carry the identity of the package and author(?) or signer(?). The date in the signed package's cert should match the release date of the module, and if it isn't refreshed in n years when the cert expires, the module should be considered abandoned.

Separate signing certs can be generated based on characteristics of the module. For example, regular modules would get signed by the main cert, sandbox modules would be signed with a lower-grade cert.

Using a certificate revocation list, modules with (>= serious) security releases have their signing certs revoked, as will modules that are considered abandoned before their cert expires.

There will need to be some work done to verify the CA cert is the one that signed the package, but that code exists.

The Backdrop project could then sell signing certs to major web shops or (cough) forks of the project as a way to generate some revenue.

[quicksketch]

Sounds like you're reinventing SSL signing, minus the identity features.

Well, I'm not reinventing SSL signing, I'm just using libsodium key-pair signing instead of SSL certificate signing.

You bring up a good point though, we might be able use OpenSSL signed certificates instead of Sodium signing. I could see benefits and downsides: The existing Sodium functions seem to be made for this exact purpose, they're fairly trivial to implement, are (supposedly) quite secure, and are built into newer versions of PHP. On the downside, this technology is relatively new, and requires an backwards compatibility shim for older PHP versions.

OpenSSL is available for all versions of PHP, but it requires that OpenSSL be installed on the host computer and PHP configured to use it correctly. I would expect that having PHP correctly configured and --with-openssl compilation flag might turn into a common installation problem.

I'm also not clear on how an OpenSSL implementation would work. @jlfranklin could you attempt to stub out code you would expect to validate the signature of a downloaded package?

The Backdrop project could then sell signing certs to major web shops

They would only need signing certs if they were redistributing their modules through another means than backdropcms.org, because the packager will already do the signing for everything in the contrib group. At this point we're just looking to make sure auto-updates come from a valid authority (backdropcms.org), so this validation only occurs when downloading a module, we wouldn't check anything for modules that already exist on disk (i.e. custom modules).

Overall on the question of OpenSSL vs Sodium (or any other option), I'm fine with any of them as long as it's secure and relatively simple. I think we should choose whichever solution is going to be the least hassle for inexperienced end-users.

[jlfranklin]

I'm also not clear on how an OpenSSL implementation would work. @jlfranklin could you attempt to stub out code you would expect to validate the signature of a downloaded package?

I'd be happy to. Give me through the weekend. I'm not sure how much free time I have this week.

[quicksketch]

I haven't tested all the pieces together, but I've put up a proof of concept against the backdropcms.org repository: backdrop-ops/backdropcms.org#456

For the time being I've just written it to use the sodium functions (though I have not included the sodium library itself yet). So far I've only tested creating a release and verifying that the MD5 is added correctly and that the MD5 + signature columns are added to the project_release table. Alternatively, we could put the signature as an attachment (asset) on the GitHub release, but it will make invalidating/regenerating signatures much easier if they're all stored centrally on backdropcms.org.

[klonos]

I would expect that having PHP correctly configured and --with-openssl compilation flag might turn into a common installation problem.

I am sharing the same concern.

The Backdrop project could then sell signing certs to major web shops or (cough) forks of the project as a way to generate some revenue.

They would only need signing certs if they were redistributing their modules through another means than backdropcms.org

Sorry @jlfranklin, I am strongly against anything like this. What makes sense to me the way @quicksketch puts it, is that if a 3rd party is distributing their custom modules from another source, then perhaps we should be making profit out of that. Rationale:

• b.org contrib-land is free and has low/no-barrier to enter, so people can bring their code there for distribution.
• I see no substantial reason for one to use another source, unless they plan to make money out of it (paid-for modules/themes)
• If people are making money directly from the product our community is volunteering time/energy/love to build, then at the very least b.org deserves a "donation" 😄...which could be used to cover our expenses and/or boost our marketing ahem, "outreach" ... efforts.

my 2c

[jenlampton]

My vote would be for using Sodium because 1) I like things that are easy, for learnability reasons. and 2) the shim may be useful for servers that don't have the Sodium module compiled with PHP, even after version 7.2, and 3) since we've already started going down this road this approach may also be the shortest route to the finish line. my .02 ;)

[jlfranklin]

I've worked through enough of the coding that I think I have a workable architecture for SSL-based signatures on modules. This post is pretty dense and technical. I don't recommend starting unless you have some time and spare cycles to devote to it.

Before we start, I want to emphasize that SSL is about identity, not encryption. Encryption is a prerequisite for SSL, not a benefit. The same encryption algorithms used by SSL are also used by SSH. You could use an SSH tunnel to connect to a website securely, but you would have no assurance you're talking to the right web site. SSL provides the identity mechanism to know who you're talking to.

Each SSL cert is a document that says, "I [insert signer's name here], hereby certify [insert subject's name here] is the owner of the private key matching the public key in this cert," and a digital signature that can be verified with the signer's public key. The cert contains the Common Name or CN of both the subject's name and the signer's name. (If you recognize CN from LDAP, it's because SSL and LDAP both derive from X.500 and share a lot of the same terminology.)

SSL certs define a one-to-one, "A signs B" relationship. If you need multiple entities to sign a key, you need multiple certs.

When I talk about creating certs below, I implicitly include the creation of a public-private key pair for the cert.

With all that in mind, let's begin.

As an overview, below is the certificate chain that I envision, with the CN of each cert in parenthesis.

Root CA Cert (Backdrop Codesign CA) signs
Class n Signing Cert (Backdrop Codesign Intermediate Class n) signs
Developer Signing Cert (Edward Xample, Developer) signs
Module Signing Cert (My Module, v1.0) creates
signature block (encrypted digest of some hash).

The Root CA Cert, Backdrop Codesign Intermediate cert, and any other CA certs are kept in core/misc/certs. The Root CA's private key should be kept offline in a secure location, such as a bank deposit box. The Codesign private key is on a hardend server that listens for signing requests from Backdrop.org and other whitelisted servers.

The developer applies for his signing cert from the Backdrop Project. For the core code and core modules, the developer cert will have something like "Backdrop Core Team" as the CN, and be managed by the release team.

Each module has a MANIFEST.txt file with the file names of all module files, including submodules. I considered using the module's .info file to store the manifest, but decided against it as module packages that include submodules would have multiple info files, and I thought the signature should be for the full module package. If the MANIFEST.txt isn't created by the author, it should be created by the signing process.

Now that the developer has his own signing cert, we're ready to sign a module.

The signing process creates a module cert signed by the developer's cert. The name of the module and its version is the CN, and the subjectAlternativeName has the machine names and versions of the modules and submodules. (E.g., my_module-1.0, my_submodule-1.0.) The subjectAlternativeName allows Backdrop to easily match the modules to the signatures.

The signing process reads the manifest file and concatenates the contents of each file named in order, finally adding the manifest itself to the end. This is the "data" that will be signed.

The signing process then uses the private key associated with the module cert to sign the module "data". The signature -- a hash of the "data", encrypted with the signer's private key -- is written to a codesign.pem file in a format that saves the hashing algorithm (e.g. sha256) and the key ID of the module's signing key. The module's cert, the developer's cert, and any other intermediate certs are added to the codesign.pem file.

When Backdrop attempts to verify the signature, it reads the codesign.pem file to get the signature block and the cert with the public key needed to verify the signature. If the signature checks out, the verification process follows the cert chain back until it finds one of the CA certs (success) or the chain is broken (failure.)

Multiple signatures can be in the codesign.pem file. A second signature from the "Backdrop Module Publishing Service" can be present in codesign.pem, with the Publishing Service replacing the developer's cert in the chain.

Finally, a Codesign module can add signing information to the admin/modules page or its own admin/reports/codesign page, and a cron hook to verify the code from time to time.

That's it.

So far, it's about 400 lines of PHP. With proper docs and tests, I expect it to be around 1,000 lines total. I've run into a couple dead ends, because some of the OpenSSL functions were introduced later than I thought, some as recently as 7.2, and I wanted to make sure we had a solution that would work with PHP 5.x.

Obviously, the code will need some thorough vetting, but I expect the hardest part will be the infrastructure to support it.

[jlfranklin]

I would have loved to get this into 1.12, but there is no way that is going to happen. Here is a quick update on my progress.

To get this to work requires changes in multiple areas:

• A new core codesign module to handle the crypto.
• Changes to the module installer to use codesign to verify the downloaded modules.
• Changes to the Project module (specifically project_release) to allow other modules to add data to a release.
• An additional project_codesign module to generate and add the signatures to module releases.

Changes to Project are already done. (See backdrop-contrib/project#25.)

The codesign module is mostly done, but I keep refactoring major pieces of it as I develop other parts of the system. (Track https://github.com/sd-backdrop/backdrop/commits/BD-1992 for progress.) The current codesign module is crypto-engine agnostic, allowing contrib modules to handle the signing and key management for OpenSSL, GnuPG, Sodium, or whatever the new hotness is next year.

I'm writing two contrib modules to ensure codesign is crypto-engine agnostic, one for OpenSSL, one for GnuPG. They are evolving with codesign.

The project_codesign module is my current focus. I'm debating adding it as a submodule to Project or creating a separate contrib module.

And, of course, there will need to be a suite of tests written for all the different parts.

[klonos]

Thanks for all your hard work and energy/time spent on this @jlfranklin 👍

Do we absolutely need to have a separate module for this? When we merged diff module in core (for use in the config diff in /admin/config/development/configuration), we have done that as an include file. I mean, is this something that we will need to turn on/off?

[jlfranklin]

It implements hook_menu() and will likely have some of its own configuration and tables by the time it's done. Codesign is not required for a site to function properly, so it could be turned off.

[klonos]

Thanks for taking the time to respond @jlfranklin ...have not gotten my head around everything involved yet, but this helps.

[jlfranklin]

See also https://github.com/sd-backdrop/codesign_gnupg as the first functional signing module.

[quicksketch]

Awesome @jlfranklin! I haven't checked in on this in a long time! It's great to see progress here.

I'm debating adding it as a submodule to Project or creating a separate contrib module.

Perhaps start separate, but if we use this on backdropcms.org we'll want it directly in project module. The situations where project module is not part of backdropcms.org are our secondary use-case, and it's already hard enough to manage with project module being separate from the main backdropcms.org repository.

[herbdool]

Wow, this looks awesome so far. Thanks for the hard work!

[quicksketch]

Thanks @jlfranklin!!! I haven't yet had the time to review this. It sounds great though! I'll get to this when I can.

[quicksketch]

I've been reviewing this one piece at a time. It's looks really good. So far I'm stuck in project module changes, but I'll get to reviewing the code_sign modules soon.

[jlfranklin]

I rerolled this patch against 1.x this morning.

[jlfranklin]

I've opened #3714 to track updates to the Installer module to check the digital signatures.

[quicksketch]

Thanks @jlfranklin! With all the moving pieces between BackdropCMS.org, project module, and core, I don't think this is possible for 1.13.0, not your fault though, reviewing is where we got hung up here.

Let's continue pushing forward on the infrastructure changes and getting the feeds themselves updated with including signatures in the short-term, and shoot for having them validated by core in 1.14.0.

[klonos]

Which is the PR to be reviewed here?

[quicksketch]

The PR for this issue is backdrop/backdrop#2449. I updated the summary to cross-link all the related issues as well.

[quicksketch]

@jlfranklin I'm working on updating the PR for Project Support of Code Sign (backdrop-contrib/project#29). And found an area where the API may need to be updated.

It looks like in order to test this I'd have to do the following:

• Apply the PR Issue #1992: Add support for signing and verifying modules backdrop#2449 to Backdrop get Code Sign module
• Install https://github.com/sd-backdrop/code_sign_openssl or https://github.com/sd-backdrop/code_sign_gnupg
• Apply the PR Add support for Code Sign backdrop-contrib/project#29 to Project module

If there's an API issue, we'd have to update 2 PRs and 2 projects in your personal account. To make this all easier, can we put all 4 modules into contrib (code_sign, code_sign_project, code_sign_gnupg, and code_sign_openssl)? For simplicity just putting them all in a single project (code_sign) would make changes across all 4 of them a lot easier.

[klonos]

Yeah, a single code_sign project, with the rest of the things being submodules seems more appropriate.

[jlfranklin]

Code Sign itself needs to be in core, because Installer calls it to verify the packages are legit.

[quicksketch]

Code Sign itself needs to be in core, because Installer calls it to verify the packages are legit.

I'd like to get it on BackdropCMS.org first so it can sign the packages and update the XML feed. But as it doesn't look like we'll likely put it in 1.14, I'd like to have it available so we can install it separately. It also gives us a good opportunity to change it without any repercussions.

Though currently the only issue I've found is that the API for exposing signing profiles just has unnecessary hash sign prefixes: https://github.com/backdrop/backdrop/pull/2449/files#r317851694

[klonos]

...as also discussed during the last dev meeting, this should not have a release when moved to contrib.

[quicksketch]

Hash signs removed in backdrop-contrib/project@b3ed711

The Contrib module (with sub-modules) for Code Sign is now pushed up to https://github.com/backdrop-contrib/code_sign, also with the hash signs removed.

[quicksketch]

I'm taking this out of the 1.15.0 milestone, as today is feature freeze and this is still not ready (mostly my fault I know). I'll continue to take this forward when I can but if there are any volunteers who are interested I'm happy to turn over the work.