Allow choosing between a gpgme and openpgp signature backend using a build tag #207

mtrmac · 2017-01-09T18:40:01Z

This demonstrates an alternative to #198 , which percolates the mechanism choice through the API and does not permit file-format-dependent mechanism selection.

In here, the default mechanism is gpgme; a containers_image_openpgp build tag can be used to use openpgp instead.

openpgp does not currently support signing, and is based on mfojtik's implementation (adding GPG home directory support, parsing of unarmored keys, and fixing ImportKeysFromBytes semantics).

NOTE: The openpgp backend is not really fleshed out yet. Some of the mechanism_test.go tests may be better mechanism-specific, and openpgp definitely needs more tests e.g. for optionalDir.

mtrmac · 2017-01-09T18:48:48Z

Note to self: this would also need a .travis.yml extension testing both variants so that we notice any breakage in the non-default path.

runcom · 2017-01-09T18:55:30Z

I'm all for build tags, it looks really nice 👍

mfojtik · 2017-01-10T09:44:31Z

signature/mechanism_openpgp.go

+		}
+	}
+
+	pubring, err := ioutil.ReadFile(path.Join(gpgHome, "pubring.gpg"))


Do we want to do this? (I guess this mimic what gpgme does).

(Repeating a personal conversation) this allows skopeo standalone-verify to work. Otherwise I agree that it is not likely to be that useful in common use (verifying against arbitrary keys including keys collected from people who have ever sent me an PGP-signed email)

mfojtik · 2017-01-10T09:45:37Z

signature/mechanism_openpgp.go

+}
+
+func (m *openpgpSigningMechanism) Sign(input []byte, keyIdentity string) ([]byte, error) {
+	return nil, errors.New("signing is not supported in github.com/containers/image built with the containers_image_openpgp build tag")


why not just return SupportSigning() here as an error?

I was considering that, but it felt dirty. Perhaps the SupportsSigning() error interface itself is just too ugly; it could just return a bool as well. Or we can define the shared error string as a constant. I don’t have a strong opinion.

mfojtik · 2017-01-10T09:45:56Z

signature/mechanism_openpgp.go

+}
+
+func (m *openpgpSigningMechanism) Verify(unverifiedSignature []byte) (contents []byte, keyIdentity string, err error) {
+	if len(m.keyring) == 0 {


not needed as we agreed.

Thanks, will drop.

mtrmac · 2017-01-11T13:55:29Z

Updated, see the first commit for an API change eliminating a publc SigningMechanism.ImportKeysFromBytes; the original commit has been updated only to implement it in the openpgp implementation.

runcom · 2017-01-11T13:58:12Z

signature/mechanism.go

-	// Sign creates a (non-detached) signature of input using keyidentity
+	Close() error
+	// SupportsSigning returns nil if the mechanism supports signing, or an error message.
+	SupportsSigning() error


Can we get rid of this method on the interface and instead Sign returns a ErrSigningNotSupported error? I don't really like this additional method where Sign could already say not supported by returning a typed error (I also believe it's more clean if this can be done). Are there any downsides with my approach?

Having an extra check which does not require you to get a key ID etc. is convenient for things like

if err := mech.SupportsSigning(); err != nil { t.Skipf("Signing not supported: %v", err) }

which this PR adds to tests, or perhaps to modify the UI of a tool (replace signing description in --help with a message saying that signing is not supported in this build).

We could, of course, also define a SigningNotSupported error type, and return that from Sign(), for those who don’t care to check in advance but still want to detect the situation.

alright, fine with that if that's used mainly to present stuff in the UI, I would rather use the typed error internally though.

Added a SigningNotSupportedError; in openpgpSigningMechanism both SupportsSigning and Sign now return that type.

runcom · 2017-01-11T16:58:27Z

signature/mechanism.go

-type gpgSigningMechanism struct {
-	ctx *gpgme.Context
+// SigningNotSupportedError is returned when trying to sign using a mechanism which does not support that.
+type SigningNotSupportedError string


sorry for nit-picking, this should be called ErrSigningNotSupported according to golang's standards

Pointer to the standards?

Both https://blog.golang.org/error-handling-and-go and https://golang.org/doc/effective_go.html#errors and e.g. types in https://golang.org/pkg/os/ use the ...Error form.

forget this, https://github.com/golang/go/wiki/Errors#naming (I swapped types and variables)

Hum, interesting. (And confusing that both types and values are recommended.)

mfojtik · 2017-01-12T11:22:38Z

LGTM

runcom · 2017-01-12T11:31:17Z

@mtrmac is there anything left to be done here? Otherwise let's pull this in.

runcom · 2017-01-12T11:32:19Z

LGTM

mtrmac · 2017-01-12T13:57:46Z

@mtrmac is there anything left to be done here?

A bit of test coverage for the new logic (e.g. $GNUPGHOME), test automation, build tag documentation, at least. And a corresponding skopeo update.

mfojtik · 2017-01-16T15:34:33Z

@mtrmac can those be done as a follow up?

mfojtik · 2017-01-23T10:27:44Z

@runcom @mtrmac how far is this from merging? I would like to pick up this work for Origin 1.5 which has feature freeze this week.

mfojtik · 2017-01-23T12:47:29Z

@mtrmac @runcom i'm taking the last comment back... I tried to pick up container/image as godep in openshift and this is what I get:

[@dev] .../openshift/origin # ./hack/build-go.sh cmd/openshift
++ Building go targets for linux/amd64: cmd/openshift
../../containers/image/openshift/openshift-copies.go:21:2: cannot find package "k8s.io/apimachinery/pkg/util/errors" in any of:
	/opt/go/src/k8s.io/apimachinery/pkg/util/errors (from $GOROOT)
	/data/src/k8s.io/apimachinery/pkg/util/errors (from $GOPATH)
../../containers/image/openshift/openshift-copies.go:22:2: cannot find package "k8s.io/apimachinery/pkg/util/net" in any of:
	/opt/go/src/k8s.io/apimachinery/pkg/util/net (from $GOROOT)
	/data/src/k8s.io/apimachinery/pkg/util/net (from $GOPATH)

This is because containers/image use "newer" version of Kubernetes than we actually have in OpenShift. This comes back to not mixing multiple things into one package and use build tags to control what gets compiled...

For OpenShift, all I need is the OpenPGP signing mechanism, that I can use to call Verify() and get back true/false and also the unverified data... That seems super-easy to do if you have a separate signature/openpgp package that does not import stuff from the signature package that has dependency on transport which has dependency on Kubernetes...

WDYT?

mfojtik · 2017-01-23T12:53:37Z

What I propose here is to break the signature/ package into pieces:

signature/policy that will be responsible for verifying policies
signature/mechanism/openpgp and
signature/mechanism/gpgme that will implement the "signature engine"

Then glue these together using the Go interfaces.

runcom · 2017-01-23T12:56:19Z

Then glue these together using the Go interfaces.

+1 to this.

mfojtik · 2017-01-23T13:00:54Z

@runcom also the openshift dependency (AKA openshift transport) have to go away because:

we now have registry endpoint to deal with signatures
it will cause mess when we import this into openshift (outdated types/etc)
openshift does not run on latest kube (it will never be).

runcom · 2017-01-23T13:03:00Z

@runcom also the openshift dependency (AKA openshift transport) have to go away because:

we now have registry endpoint to deal with signatures
it will cause mess when we import this into openshift (outdated types/etc)
openshift does not run on latest kube (it will never be).

I see, I understand those points. Let's see what @mtrmac thinks about this (since he wrote the openshift transport at the beginning).

mtrmac · 2017-01-23T16:27:33Z

This is because containers/image use "newer" version of Kubernetes than we actually have in OpenShift. This comes back to not mixing multiple things into one package and use build tags to control what gets compiled...

No, that really comes back to Go being utterly unhelpful for maintaining stable APIs and sharing referenced libraries across projects. This kind of breakage could just as well have happened in any other dependency like logrus or x/crypto/openpgp.

(@runcom is working on allowing to use signature with fewer dependencies, in #216 . But that would still not be obviously helpful here; it would be perfectly natural for an OpenShift caller of containers/image to depend on the OpenShift transport, and for that transport to use OpenShift client libraries. The underlying issue really is the version skew / API breakage.)

For OpenShift, all I need is the OpenPGP signing mechanism, that I can use to call Verify() and get back true/false and also the unverified data...

Getting back “true/false” really won’t work that way, with future signatures of base images etc. “Does the signature verify” and “does the signature apply to this image” will still not have a sufficient semantic meaning, and something like that policy will be necessary (perhaps not as a policy.json file format, but as a data structure, a statement of “what is expected of the signature I am submitting”.)

That seems super-easy to do if you have a separate signature/openpgp package that does not import stuff from the signature package that has dependency on transport which has dependency on Kubernetes...

For the above reason, it is fairly likely that the fairly low-level signature.VerifyDockerManifestSignature will either go away completely, or become a front-end for the policy evaluation mechanism within the next year or so.

mtrmac · 2017-01-23T16:34:01Z

@runcom also the openshift dependency (AKA openshift transport) have to go away because:

we now have registry endpoint to deal with signatures

We do need to have that conversation, but it can’t go away right now, while registries which only support the OpenShift-like API are deployed. This will need a deprecation plan, compatibility transition mechanism, and so on.

it will cause mess when we import this into openshift (outdated types/etc)

openshift does not run on latest kube (it will never be).

Really there is another, perhaps pretty good alternative: #188 , just depend on the OpenShift-provided client implementation (if that has a stable API). Right now the size of the added dependency seems to make it prohibitive.

Right now, the simplest way to clean this up is to stop using the k8s.io/apimachinery-provided functionality, and fold even more code into openshift-copies.go. sigh Working on that.

mtrmac · 2017-01-23T16:44:44Z

@runcom @mtrmac how far is this from merging? I would like to pick up this work for Origin 1.5 which has feature freeze this week.

I’ll get rid of k8s.io/apimachinery, and those tests etc. still need to be written (sorry, last week we had several build breakages caused by dependencies which have delayed that work).

But given @runcom ’s LGTM, I think you can safely rely on the API and build tag name not changing.

mtrmac · 2017-01-23T18:07:55Z

Right now, the simplest way to clean this up is to stop using the k8s.io/apimachinery-provided functionality, and fold even more code into openshift-copies.go. *sigh* Working on that.

#225

mtrmac · 2017-01-30T21:34:29Z

it seemed a build tag is a no-go for openshift 😕

*sigh* What other alternative is there for 1) not having a mandatory gpgme dependency, 2) keeping the API capable of handling multiple formats in prSignedBy, 3) keeping the default build capable of signing, 4) using a best-practice implementation by default?

It seems that 1) is critical for OpenShift, 2) is critical for containers/image; and both 3 and 4 seem very desirable. We can give up on 3 by defaulting to openpgp, or on 4 by writing an OpenPGP signing implementation; but really is the cost of a few lines somewhere in a Makefile that great? Can we help with that?

mfojtik · 2017-01-31T09:44:09Z

The containers/image is not doing the signing, it is the projects that are using this library, right?
If CRI is going to pickup containers/imageand enforce signatures in Kubernetes (@runcom ?) then
I think the use of build tags or CGO will be a strong "no-go" there.

I see the build tags as a band-aid to make this work now, but what I think should really happen is a
proper refactoring of the "signature" package. The signing mechanisms (implementations) should be
moved to separate packages (maybe the policies as well) and we should use the interfaces and not depend on a single implementation.
If the tests are the problem here, we can have build tag for tests (a go file that will override the openpgp -> gpgme if the build tag is set).

Then whoever is importing the github.com/containers/image should decide which mechanism he wants to use (depending whether he wants signing/verify or just verify?).

@smarterclayton FYI

mtrmac · 2017-03-06T21:51:18Z

@mfojtik

The containers/image is not doing the signing, it is the projects that are using this library, right?

I am not sure what you mean by drawing this distinction. Signing happens by callers of the library invoking code within the library.

If CRI is going to pickup containers/imageand enforce signatures in Kubernetes (@runcom ?) then
I think the use of build tags or CGO will be a strong "no-go" there.

AFAICT it is a complete non-issue:

It is already using build tags: https://github.com/kubernetes-incubator/cri-o/blob/master/Makefile#L14
It is already using CGO: https://github.com/kubernetes-incubator/cri-o/blob/master/vendor/github.com/seccomp/libseccomp-golang/seccomp.go#L23 .
And it is already using containers/image/signature: https://github.com/kubernetes-incubator/cri-o/blob/master/pkg/storage/image.go#L107

Or do you mean some other CRI project?

I see the build tags as a band-aid to make this work now, but what I think should really happen is a
proper refactoring of the "signature" package.

I still can’t see any way to refactor this to individual Lego bricks while doing at least 1) and 2) from #207 (comment) . For 2), containers/image/signature must reference mechanisms for all defined file formats; not only for the policy, but also for #208 (right now that is only GPG but that will change, at least to add X.509). Am I missing anything?

The signing mechanisms (implementations) should be
moved to separate packages (maybe the policies as well) and we should use the interfaces and not > depend on a single implementation.

(BTW exposing the mechanism and making them pluggable is fairly undesirable. Every time the interface allows it, “helpful” recipes like http://www.rgagnon.com/javadetails/java-fix-certificate-problem-in-HTTPS.html start appearing on the internet. I am very tempted to make the SigningMechanism interface use private methods.

Sure, in the ideal world, and all that… But anyway, 2) above makes this interface design/nudge aspect secondary.)

FWIW https://github.com/mtrmac/origin/tree/c-i-verify-image-signature , and mtrmac/origin@e45af45 in particular, seems to work well enough as a proof of concept for using the build tag in OpenShift: at least both make and make build-cross works just fine. This also demonstrates a possible use of policy.json instead of dealing with GPG directly.

(Disclaimers:

I never quite got godeps to behave, which AFAICT has nothing to do with the build tag. I ended up manually commiting a copy to vendor for one dependency.
Other parts of the build system like test-go.sh will probably also need updating for build tags. OTOH, with the documented gssapi tag, it seems that such infrastructure should ideally exist anyway.
This still copy&pastes some code it shouldn’t, to get rid of containers/image/docker. A correct solution would be to handle the version skew of docker/…/strslice and github.com/docker/go-connections/tlsconfig, which seems quite plausible but I didn’t want to deal with it for the prototype.

)

mfojtik · 2017-03-16T11:38:43Z

@mtrmac i think I'm ok with the current state as long as we can move forward with the image verification in openshift.

mtrmac · 2017-03-18T18:39:27Z

@runcom PTAL again

runcom · 2017-03-20T13:30:56Z

seems like tests are failing with the new build tag https://travis-ci.org/containers/image/jobs/212314846

mtrmac · 2017-03-20T14:50:03Z

seems like tests are failing with the new build tag

Yes, because skopeo tests expect signing to be possible, see #207 (comment) for more discussion.

A manual make .gitvalidation validate test test-skopeo BUILDTAGS="btrfs_noversion libdm_no_deferred_remove containers_image_openpgp" SKOPEO_REPO=mtrmac/skopeo SKOPEO_BRANCH=openpgp does work fine.

…hanism objects Instead of SigningMechanism.ImportKeysFromBytes, which may modify long-term state in the user’s home directory, provide NewEphemeralGPGSigningMechanism, which combines creation of a temporary directory and key import; users of NewGPGSigningMechanism (using $GNUPGHOME etc.) and (test-suite) users of newGPGSigningMechanismInDirectory can no longer import keys. To support this, all users of SigningMechanism must call .Close() on the object. Signed-off-by: Miloslav Trmač <mitr@redhat.com>

…build tag The default is gpgme; a containers_image_openpgp build tag can be used to use openpgp instead. openpgp does not currently support signing, and is based on mfojtik's implementation (adding GPG home directory support, parsing of unarmored keys, and fixing ImportKeysFromBytes semantics). Also adds build documentation, including the new containers_image_openpgp build tag, to README.md. This does not yet hook this into Travis, because that needs a corresponding skopeo build infrastructure and test update to make it possible. That is a separate commit (perhaps a separate PR)? Signed-off-by: Miloslav Trmač <mitr@redhat.com>

Signed-off-by: Miloslav Trmač <mitr@redhat.com>

This uses the Travis test matrix support; arguably the total time might be smaller if setting up the VM only once and running both variants within the shell script. Signed-off-by: Miloslav Trmač <mitr@redhat.com>

mfojtik · 2017-03-29T08:03:42Z

@runcom @mtrmac any chance we can have this merged soon? i would like to go forward with the oc adm verify-image-signature in openshift for 3.6

runcom · 2017-03-29T08:11:16Z

@mfojtik containers/skopeo#298 passes tests with this PR vendored so I'd say it's ok to merge this if @mtrmac agrees.

mtrmac · 2017-03-29T18:47:21Z

👍

mtrmac mentioned this pull request Jan 9, 2017

Add second signing mechanism based on native 'openpgp' Go library #198

Closed

mfojtik reviewed Jan 10, 2017

View reviewed changes

mtrmac force-pushed the openpgp branch from b35edc2 to ba7aa74 Compare January 11, 2017 13:54

runcom reviewed Jan 11, 2017

View reviewed changes

mtrmac force-pushed the openpgp branch from ba7aa74 to 628aa2a Compare January 11, 2017 16:51

runcom reviewed Jan 11, 2017

View reviewed changes

mtrmac mentioned this pull request Jan 18, 2017

API for providing untrusted data about unverified signatures #208

Merged

mtrmac force-pushed the openpgp branch from 628aa2a to 034d3ad Compare January 19, 2017 23:25

mfojtik mentioned this pull request Jan 23, 2017

Verify image signatures openshift/origin#12616

Closed

mtrmac mentioned this pull request Jan 23, 2017

Remove dependencies on k8s.io/apimachinery #225

Closed

mtrmac force-pushed the openpgp branch from 91a2ccb to eb2edc0 Compare January 30, 2017 21:25

mtrmac force-pushed the openpgp branch 2 times, most recently from 474e19a to 0eeca14 Compare January 31, 2017 17:29

mtrmac mentioned this pull request Jan 31, 2017

Drop c/i/docker/reference (again) #221

Merged

mtrmac force-pushed the openpgp branch 2 times, most recently from 108e23d to bc7933d Compare February 6, 2017 20:36

mtrmac force-pushed the openpgp branch from bc7933d to 935f4d5 Compare February 14, 2017 16:17

mtrmac force-pushed the openpgp branch 3 times, most recently from c2c51e0 to 4683181 Compare March 2, 2017 20:19

mtrmac force-pushed the openpgp branch from 4683181 to c07f8fd Compare March 3, 2017 23:49

mtrmac force-pushed the openpgp branch from c07f8fd to a0e643f Compare March 17, 2017 23:18

mtrmac added 4 commits March 28, 2017 18:57

Propagate BUILDTAGS also to skopeo's (make check)

8c17b7b

Signed-off-by: Miloslav Trmač <mitr@redhat.com>

Test both the default and openpgp variants in Travis.

1a37631

This uses the Travis test matrix support; arguably the total time might be smaller if setting up the VM only once and running both variants within the shell script. Signed-off-by: Miloslav Trmač <mitr@redhat.com>

mtrmac force-pushed the openpgp branch from a0e643f to 1a37631 Compare March 28, 2017 16:59

mtrmac merged commit 819d16e into containers:master Mar 29, 2017

mtrmac deleted the openpgp branch March 29, 2017 18:48

mtrmac mentioned this pull request May 10, 2017

Switch to use native go opengpg instead of cgo containers/skopeo#274

Closed

Allow choosing between a gpgme and openpgp signature backend using a build tag #207

Allow choosing between a gpgme and openpgp signature backend using a build tag #207

Conversation

mtrmac commented Jan 9, 2017

mtrmac commented Jan 9, 2017

runcom commented Jan 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Jan 11, 2017

runcom Jan 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfojtik commented Jan 12, 2017

runcom commented Jan 12, 2017 • edited Loading

runcom commented Jan 12, 2017 • edited Loading

mtrmac commented Jan 12, 2017

mfojtik commented Jan 16, 2017

mfojtik commented Jan 23, 2017

mfojtik commented Jan 23, 2017

mfojtik commented Jan 23, 2017

runcom commented Jan 23, 2017

mfojtik commented Jan 23, 2017 • edited Loading

runcom commented Jan 23, 2017

mtrmac commented Jan 23, 2017

mtrmac commented Jan 23, 2017

mtrmac commented Jan 23, 2017

mtrmac commented Jan 23, 2017

mtrmac commented Jan 30, 2017

mfojtik commented Jan 31, 2017

mtrmac commented Mar 6, 2017

mfojtik commented Mar 16, 2017

mtrmac commented Mar 18, 2017

runcom commented Mar 20, 2017

mtrmac commented Mar 20, 2017

mfojtik commented Mar 29, 2017

runcom commented Mar 29, 2017

mtrmac commented Mar 29, 2017 • edited by runcom Loading

runcom Jan 11, 2017 •

edited

Loading

runcom commented Jan 12, 2017 •

edited

Loading

runcom commented Jan 12, 2017 •

edited

Loading

mfojtik commented Jan 23, 2017 •

edited

Loading

mtrmac commented Mar 29, 2017 •

edited by runcom

Loading