Extensible way to GPG-sign repository metadata. #1093

mibanescu · 2018-03-08T22:02:14Z

In certain environments, GPG private keys are secured in an HSM,
instead of being in a keyring, unprotected by a passphrase.

This allows one to change the signing command, and passes
repository ID information as an envronment variable into
the signing command, in case different keys need to be used.

Fixes: #3444
https://pulp.plan.io/issues/3444

dkliban · 2018-03-08T22:27:49Z

@ehelms I think you may be interested in this PR.

ehelms · 2018-03-09T20:32:13Z

To clarify for me, this keeps GPG signing key management external to Pulp but increases the flexibility for deployment of that key via configurable signing mechanism that lets me have full control over how I deploy my signing key?

mibanescu · 2018-03-09T20:34:45Z

Yes, that is the intention. We use relic (https://github.com/sassoftware/relic) and a signing server with an HSM.

ehelms · 2018-03-09T20:38:22Z

@PaulSD I think you will find this very interesting given it would affect the work in (theforeman/puppet-certs#188) potentially changing it or making it less complicated for your scenario.

PaulSD · 2018-03-12T16:59:05Z

Cool!

For reference, I wrote the initial repo metadata signing stuff here: f73805f
Some HSMs can be seamlessly used by the gpg command in place of a keyring, hence why I didn't bother making the command configurable. But this PR is definitely a better and more generic solution.

theforeman/puppet-certs#188 is meant to configure this behavior in pulp, distribute the public GPG key to clients, and configure subscription-manager on clients (see candlepin/subscription-manager#1749 ). I should probably update that PR to allow configuration of the new Pulp configuration options in this PR, but I don't otherwise think this should impact that.

PaulSD · 2018-03-12T18:16:34Z

Actually, in the case that multiple keys are used for different Pulp repos, this may significantly complicate things in theforeman/puppet-certs#188 ...

@mibanescu do you have a user story or an actual use case that requires signing the repo metadata on separate repos with separate keys? Would a Foreman user ever need to do that, or is this only needed for more generic/independent uses of Pulp? Can I just ignore that use case for the purposes of theforeman/puppet-certs#188 ?

mibanescu · 2018-03-12T18:40:26Z

I am not familiar with Foreman or the other PR, but here is my use case.

We use a dev/test/prod model for our software lifecycle.

We want our customers, consuming prod packages (and metadata) to validate that the software comes from us. So we sign that with a key.

However, we don't want dev or test packages to be signed with the prod key, because that software is not ready for public consumption.

We would want a test deployment to act like a prod deployment, including the signature validation part, so do want signatures, just not with the prod key.

PaulSD · 2018-03-12T19:06:07Z

Thanks!

In that case, you definitely wouldn't want all of the keys automatically distributed to all of the clients, which is part of the point of theforeman/puppet-certs#188 ... So, I think I am safe ignoring this use case for now in that PR.

mibanescu · 2018-03-12T19:58:27Z

I should also mention that, if gpg_cmd is not set in the plugin config, the sign command defaults to what was in the original patch.

rohanpm · 2018-03-13T05:43:57Z

docs/tech-reference/yum-plugins.rst

+
+The signing command will be passed the following environment variables:
+* ``GPG_CMD``
+* ``GPG_KEY_ID`` (if specified in the configuration file)


This says "the configuration file", but the key ID can also be stored on the yum distributor object, correct?

Yes, that is correct. All these can be done via distributor configurations as well, although GPG_CMD is probably not hugely helpful. GPG_KEY_ID is.

Would you like me to update the documentation to include this?

Also, is it possible for this to be included in pulp 2.16? I saw an email mentioning that it would need to be merged today.

I'm commenting as an interested user only. We would use this feature, but we'd probably set the key ID on each repo's distributor object.

If written as, "if specified in the configuration", i.e. drop the "file", then I'd have understood it as meaning all of the usual ways config can be provided to a distributor: on the distributor itself, on the config file for that distributor type, or in override_config during a call to publish.

Agreed, and fixed.

bmbouter · 2018-03-13T19:00:24Z

@mibanescu Can you add a 2.16 release note to pulp_rpm for this? I may be able to test some today before the 2.16 deadline, but it needs a release note.

In certain environments, GPG private keys are secured in an HSM, instead of being in a keyring, unprotected by a passphrase. This allows one to change the signing command, and passes repository ID information as an envronment variable into the signing command, in case different keys need to be used. Fixes: #3444 https://pulp.plan.io/issues/3444

bmbouter · 2018-03-14T13:25:35Z

Started from a vanilla pulp2 dev box w/ @misa's PR checked out
I then update the distributor with: curl -u admin:admin -k https://localhost/pulp/api/v2/repositories/zoo/distributors/yum_distributor/ -XPUT --data-binary '{"distributor_config":{"gpg_key_id": "some-name-here"}}'
Then I get this error

Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032) Task pulp.server.tasks.repository.distributor_update[89b59096-802c-4438-83dc-58375c0a644d] raised unexpected: PulpDataException(u'Configuration key [gpg_key_id] is not supported',)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032) Traceback (most recent call last):
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)     R = retval = fun(*args, **kwargs)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 529, in __call__
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)     return super(Task, self).__call__(*args, **kwargs)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)   File "/home/vagrant/devel/pulp/server/pulp/server/async/tasks.py", line 107, in __call__
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)     return super(PulpTask, self).__call__(*args, **kwargs)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)     return self.run(*args, **kwargs)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)   File "/home/vagrant/devel/pulp/server/pulp/server/controllers/distributor.py", line 239, in update
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032)     raise exceptions.PulpDataException(message)
Mar 13 21:13:03 pulp2.dev pulp[9367]: celery.app.trace:ERROR: [89b59096] (9367-36032) PulpDataException: Configuration key [gpg_key_id] is not supported

bmbouter · 2018-03-14T18:11:16Z

After pulling the latest code and making a few changes to testing I believe this feature is demonstrated to be working. Here is what I did:

Write the following contents to /etc/pulp/server/plugins.conf.d/yum_distributor.json

{
    "gpg_sign_metadata": true,
    "gpg_cmd": "/usr/local/bin/sign.sh"
}

Write a simple utility that will be the thing called. This writes out the environment variables the signing script can use. I write this here: /usr/local/bin/sign.sh and it contains these contents and chmod 0755 it.

#!/bin/bash -e

set > ${1}.asc

I then post the distributor update to the yum_distributor to set gpg_key_id with this command: curl -u admin:admin -k https://localhost/pulp/api/v2/repositories/zoo/distributors/yum_distributor/ -XPUT --data-binary '{"distributor_config":{"gpg_key_id": "some-name-here"}}'
I then run the sync+publish with: pulp-admin rpm repo sync run --repo-id zoo. Watch the sync and publish complete successfully.
I cat the .asc file that was created and contains the following:

[vagrant@pulp2 ~]$ sudo cat /var/lib/pulp/published/yum/master/yum_distributor/zoo/1521050158.47/repodata/repomd.xml.asc
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=([0]="1")
BASH_ARGV=([0]="/var/cache/pulp/reserved_resource_worker-0@pulp2.dev/2bab9400-82fa-40cb-944a-c478019cde43/repodata/repomd.xml")
BASH_CMDS=()
BASH_LINENO=([0]="0")
BASH_SOURCE=([0]="/usr/local/bin/sign.sh")
BASH_VERSINFO=([0]="4" [1]="4" [2]="12" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='4.4.12(1)-release'
DIRSTACK=()
EUID=48
GPG_CMD=/usr/local/bin/sign.sh
GPG_KEY_ID=some-name-here
GPG_REPOSITORY_NAME=zoo
GROUPS=()
HOSTNAME=pulp2.dev
HOSTTYPE=x86_64
IFS=$' \t\n'
MACHTYPE=x86_64-redhat-linux-gnu
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin
PPID=14326
PS4='+ '
PWD=/run/pulp
SHELL=/sbin/nologin
SHELLOPTS=braceexpand:errexit:hashall:interactive-comments
SHLVL=1
TERM=dumb
UID=48
_=/bin/bash

Specifically you can see that:

the signing call was made as expected
The variables GPG_CMD, GPG_KEY_ID, and GPG_REPOSITORY_NAME are all present and contain the expected values.

bmbouter

I've tested it and I posted my results. This works as expected and there are no other outstanding changes that I know of.

mibanescu force-pushed the master branch 2 times, most recently from 45d93d5 to f575f24 Compare March 9, 2018 19:35

rohanpm reviewed Mar 13, 2018

View reviewed changes

mibanescu force-pushed the master branch 2 times, most recently from fd7cfc6 to 9e241f8 Compare March 13, 2018 18:54

mibanescu force-pushed the master branch from 9e241f8 to 37aaf04 Compare March 13, 2018 19:46

mibanescu force-pushed the master branch from 37aaf04 to 09ba819 Compare March 14, 2018 01:39

bmbouter approved these changes Mar 14, 2018

View reviewed changes

bmbouter merged commit 3d77b81 into pulp:master Mar 14, 2018

PaulSD mentioned this pull request Jun 13, 2018

Use gpg_key_id even when gpg_cmd is not configured #1117

Merged

Extensible way to GPG-sign repository metadata. #1093

Extensible way to GPG-sign repository metadata. #1093

mibanescu commented Mar 8, 2018

dkliban commented Mar 8, 2018

ehelms commented Mar 9, 2018

mibanescu commented Mar 9, 2018

ehelms commented Mar 9, 2018

PaulSD commented Mar 12, 2018

PaulSD commented Mar 12, 2018 •

edited

mibanescu commented Mar 12, 2018

PaulSD commented Mar 12, 2018

mibanescu commented Mar 12, 2018

rohanpm Mar 13, 2018

mibanescu Mar 13, 2018

rohanpm Mar 13, 2018

mibanescu Mar 14, 2018

bmbouter commented Mar 13, 2018

bmbouter commented Mar 14, 2018

bmbouter commented Mar 14, 2018

bmbouter left a comment

Extensible way to GPG-sign repository metadata. #1093

Extensible way to GPG-sign repository metadata. #1093

Conversation

mibanescu commented Mar 8, 2018

dkliban commented Mar 8, 2018

ehelms commented Mar 9, 2018

mibanescu commented Mar 9, 2018

ehelms commented Mar 9, 2018

PaulSD commented Mar 12, 2018

PaulSD commented Mar 12, 2018 • edited

mibanescu commented Mar 12, 2018

PaulSD commented Mar 12, 2018

mibanescu commented Mar 12, 2018

rohanpm Mar 13, 2018

Choose a reason for hiding this comment

mibanescu Mar 13, 2018

Choose a reason for hiding this comment

rohanpm Mar 13, 2018

Choose a reason for hiding this comment

mibanescu Mar 14, 2018

Choose a reason for hiding this comment

bmbouter commented Mar 13, 2018

bmbouter commented Mar 14, 2018

bmbouter commented Mar 14, 2018

bmbouter left a comment

Choose a reason for hiding this comment

PaulSD commented Mar 12, 2018 •

edited