Pluggable secret backend #2239

liron-l · 2017-06-11T09:55:20Z

This commit extends SwarmKit secret management with pluggable secret
backends support. The solution uses the existing docker plugin
framework for loading plugins and the existing SwarmKit data backend for
storing them.

The approach is to add a new driver parameter to existing secrets,
which defines whether the values are taken as is or fetched from one of
the secret plugins. The loading of secrets is done using the standard
docker plugin infrastructure, which is already accessible in SwarmKit
and used in other flows (e.g., networking).
The fetched values are are stored as regular SwarmKit secrets.

Remarks:

I've added support for mocking the plugin subsystem when settings up
the controlapi server.
I preferred this approach over loading the full plugin subsystem in UT.

Work still needed in this CR:

More unit tests (pending initial iteration)
Customized error handling (e.g., customize error string for Not
Found)

Work still needed to complete this feature:

Inject secrets as part of plugin initialization
CLI support in docker
Docs
Support scheduling plugins in swarm
Plugins on swarm moby#33575

Signed-off-by: liron liron@twistlock.com

codecov · 2017-06-11T11:48:23Z

Codecov Report

Merging #2239 into master will increase coverage by 0.03%.
The diff coverage is 81.25%.

@@            Coverage Diff            @@
##           master   #2239      +/-   ##
=========================================
+ Coverage   61.07%   61.1%   +0.03%     
=========================================
  Files         128     128              
  Lines       20556   20579      +23     
=========================================
+ Hits        12554   12575      +21     
+ Misses       6627    6619       -8     
- Partials     1375    1385      +10

aaronlehmann · 2017-06-12T14:04:03Z

api/specs.proto

@@ -386,6 +386,9 @@ message SecretSpec {

 	// Data is the secret payload - the maximum size is 500KB (that is, 500*1024 bytes)
 	bytes data = 2;
+
+	// Driver is the name of the secret driver that is used to store the specified secret
+	string driver = 3;


I wonder if it makes sense to use the Driver type here, to allow the future possibility of passing secret-specific options to the driver.

cpuguy83 · 2017-06-11T18:06:08Z

manager/controlapi/secret.go

-const MaxSecretSize = 500 * 1024 // 500KB
+const (
+	// SecretsPluginAPI is the endpoint for fetching secrets from plugins
+	SecretsPluginAPI = "/SecretsDriver.GetSecret"


SecretProvider? Same with the capability name.

aaronlehmann · 2017-06-12T14:13:16Z

manager/controlapi/secret.go

@@ -157,12 +169,16 @@ func (s *Server) ListSecrets(ctx context.Context, request *api.ListSecretsReques
 //   or if the secret data is too long or contains invalid characters.
 // - Returns an error if the creation fails.
 func (s *Server) CreateSecret(ctx context.Context, request *api.CreateSecretRequest) (*api.CreateSecretResponse, error) {
+	err := s.populateSecretFromPlugin(ctx, request.Spec)


Another alternative would be to put this resolution in the dispatcher, so the secret is fetched at the time the task is sent to the node where it will run. This is what I suggested earlier. The advantages would be:

swarmkit would not be modifying the secret Spec. Specs are meant to be under the control of the user, and so far swarmkit never changes them. We're considering signing specs with a user-controlled key in the future.

Calling UpdateSecret (for example, to change the secret's labels) would not have side effects on the payload, if it happened to change in the backend.

We would avoid storing a copy of the secret payload inside the Raft datastore, which people may not want to do for security reasons.

The disadvantages would be:

Inability to access the secrets backend could block deploying tasks, not just creating/updating secrets.

To avoid redundant queries to the backend for each of N tasks that reference a secret, it would make sense to have an in-memory cache of secret payloads with a time-to-live, but this would add complexity.

@diogomonica @cpuguy83 what do you think?
@aaronlehmann i think that caching should be the plugin responsibility. WDYT?

SGMT @aaronlehmann

One way around issues is to have a separate call that the the manager runs here to ensure that the secret is accessible before sending it for dispatch. Doesn't need to store anything.

I also think we should create a driver interface that have implementations for the built-in store and plugins.
Then it's a simple d := GetDriver(spec.Driver); d.<method>

Thanks @aaronlehmann, the assignment solution still means that we need to modify the secret spec payload inside Assignment_Secret before we send the assignment (that is, after the secret query in addTaskDependencies). Is this a valid approach?
I think your first concern might cause debugging issues with secret plugins, @diogomonica WDYT?

@cpuguy83 I've abstracted the plugin/driver initialization and setup with a new drivers package. This package can be used to load any type of plugin in Swarmkit. Let me know what you think.

Thanks @aaronlehmann, the assignment solution still means that we need to modify the secret spec payload inside Assignment_Secret before we send the assignment (that is, after the secret query in addTaskDependencies). Is this a valid approach?

That's a fair point, however I think it's preferable to modify the spec for last-mile delivery versus storing a modified version in the data store. One possibility to avoid modifying the spec at all would be to introduce another Data field outside of Spec that could be freely modified by the manager. When the worker receives a secret, it would check both fields to see which one is populated. @diogomonica WDYT?

@aaronlehmann I like that approach. Would still allow us to sign the spec itself, and if the secret is external, that component would be unsigned.

liron-l · 2017-06-18T13:21:27Z

@aaronlehmann @cpuguy83, @diogomonica I moved the secret resolution to the assignmentSet flow. Let me know if this makes sense and I will add dedicated UT.

diogomonica · 2017-06-18T23:54:11Z

manager/drivers/secrets.go

+}
+
+// Get gets a secret from the secret provider
+func (d *SecretDriver) Get(spec *api.SecretSpec) ([]byte, error) {


Do we want to provide an api.SecretSpec here? Would be great if the driver got information of which service is requesting this secret.

@diogomonica @cpuguy83 @aaronlehmann, I can think about two options to pass service parameters to plugin in a deterministic backward compatible way:

Send the Task as binary blob inside the request

Use the ServiceAnnotations
Do you have other ideas?

Riyaz and I came up with a list. For now let's not worry about passing more stuff and agree on the general API.

Great, thanks!

diogomonica · 2017-06-18T23:55:41Z

manager/dispatcher/assignments.go

+
+// populateSecretFromPlugin populates the secret value for the given specification using the secret plugin subsystem.
+func (a *assignmentSet) populateSecretFromDriver(spec *api.SecretSpec) error {
+	if spec == nil || spec.Driver.Name == "" {


Why not use validateSecretSpec

Thanks, I've added shared code to validate the secret payload, after the value is populated.

diogomonica · 2017-06-18T23:58:14Z

manager/drivers/secrets.go

+}
+
+// SecretsProviderRequest is the request specification for retrieving secrets from plugins.
+type SecretsProviderRequest struct {


Would be great if we have extra metadata on the caller of the request, so that external plugin can issue customized secrets.

I passed the ServiceAnnotations, I hope this makes sense.

from discussion in slack: it might be more useful to pass the full ServiceSpec, though that would require converting into JSON

diogomonica · 2017-06-18T23:59:59Z

Overall I think this is a good start. We need to figure out the best way of providing external metadata on the ultimate service this secret is being requested for, such that an external plugin can issue secrets for a specific service.

An example would be a TLS certificate that gets issued on-the-fly by the secrets plugin, and needs to include the service name/other metadata on the x509 certificate that depends on the service itself.

liron-l · 2017-06-19T08:07:31Z

@diogomonica @cpuguy83 @aaronlehmann
I published another iteration with the following changes:

Moved the secret resolution to the dispatcher next to fetching the secret values from raft store.
Add the ServiceAnnotations metadata to the driver request
Added a simple dispatcher UT for resolving the secret values

aaronlehmann · 2017-06-19T17:32:24Z

manager/dispatcher/dispatcher_test.go

+	assert.NoError(t, err)
+	defer stream.CloseSend()
+
+	time.Sleep(500 * time.Millisecond)


Is the sleep necessary? Recv is a blocking function.

Thanks, fixed

aaronlehmann · 2017-06-19T17:34:27Z

manager/controlapi/secret.go

@@ -238,16 +238,26 @@ func (s *Server) RemoveSecret(ctx context.Context, request *api.RemoveSecretRequ
 	}
 }

+// ValidateSecretPayload validates the secret payload size
+func ValidateSecretPayload(data []byte) error {


I'd prefer to move this to a subpackage like api/secret or api/validation, so that dispatcher doesn't import controlapi.

Moved to api/validation/secrets.go, I kept the prefix validation in the function name, let me know if you prefer validation.SecretPayload

aaronlehmann · 2017-06-19T17:35:19Z

manager/dispatcher/dispatcher_test.go

@@ -15,16 +15,23 @@ import (
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/credentials"

+	"encoding/json"


nit: Standard library imports such as this are typically put in the top section of the import statement, sorted alphabetically.

Thanks fixed

liron-l · 2017-06-22T09:07:51Z

Thanks @aaronlehmann I've updated the review according to your comments.

liron-l · 2017-06-27T04:11:20Z

Thanks @aaronlehmann! I've fixed the test and added validation that the ServiceSpec propagated correctly to the secrets driver.

liron-l · 2017-07-02T10:48:12Z

Thanks for all comments, @aaronlehmann, @diogomonica, @cpuguy83.
I've update the review according to the comments, please take a look.

liron-l · 2017-07-05T19:19:45Z

Per discussion with @diogomonica, I removed ServerSpec from plugin request. Additional service properties will be added if required.

cpuguy83 · 2017-07-05T19:23:09Z

manager/controlapi/secret.go

-
-	if len(spec.Data) >= MaxSecretSize || len(spec.Data) < 1 {
-		return grpc.Errorf(codes.InvalidArgument, "secret data must be larger than 0 and less than %d bytes", MaxSecretSize)
+	if spec.Driver.Name != "" {


I assume the built-in backend must have a name, or is likely to have a name at some point just to be explicit... maybe this check is not sufficient?

Thanks @cpuguy83, I made this type nullable, and if defined, validation will throw an error if no name is specified.

cpuguy83 · 2017-07-05T19:36:11Z

api/specs.proto

@@ -393,6 +393,9 @@ message SecretSpec {
 	// The currently recognized values are:
 	// - golang: Go templating
 	Driver templating = 3;
+
+	// Driver is the the secret driver that is used to store the specified secret
+	Driver driver = 4 [(gogoproto.nullable) = false];


Are we sure this shouldn't be nullable?
I see a lot of checks for if spec.Driver.Name != "" {}

Thanks @cpuguy83, I made this nullable. Now, I separate between nullable driver (OK) to initialized value with name (added validation)

cpuguy83 · 2017-07-05T19:37:48Z

manager/dispatcher/assignments.go

@@ -245,3 +260,24 @@ func (a *assignmentSet) message() api.AssignmentsMessage {

 	return message
 }
+
+// populateSecretFromPlugin populates the secret value for the given specification using the secret plugin subsystem.
+func (a *assignmentSet) populateSecretFromDriver(spec *api.SecretSpec, readTx store.ReadTx) error {


This function still confuses me a bit. How is the secret populated for the built-in raft store?

Thanks @cpuguy83, the secret spec was populated by the calling method. I melded everything to a single function (secret, which fetch the value from raft store and populate the secret value if needed). To keep flow consistent, I've change the Debug message in case the secret is not found in raft-store to Error, hope this makes more sense now.

liron-l · 2017-07-05T20:35:35Z

Thanks for the comments @cpuguy83, I've updated the review based on your feedback.

riyazdf · 2017-07-05T20:16:08Z

cmd/swarmctl/secret/create.go

@@ -63,4 +69,5 @@ var createCmd = &cobra.Command{

 func init() {
 	createCmd.Flags().StringP("file", "f", "", "Rather than read the secret from STDIN, read from the given file")
+	createCmd.Flags().StringP("driver", "d", "", "The secret driver")


nit: I think we should note that it is STDIN if not specified

Thanks @riyazdf, I changed it according to the file flag format.

riyazdf · 2017-07-05T20:40:50Z

manager/controlapi/secret.go

+	// Check if secret driver is defined
+	if spec.Driver != nil {
+		// Ensure secret driver has a name
+		if spec.Driver.Name == "" {


I might be tracing the code incorrectly but it seems that we could have a non-nil driver with an empty name? See: https://github.com/docker/swarmkit/pull/2239/files#diff-b7cdf7ddfbe8b31d75bc99e8d2d0fa78R58

Thanks @riyazdf, you are right, I modified the create flow accordingly.

liron-l · 2017-07-05T20:56:15Z

Thanks @riyazdf, let me know if you think we should make the Driver non-nullable type.

cpuguy83 · 2017-07-05T20:45:19Z

manager/dispatcher/assignments.go

+
+// secret populates the secret value from raft store. For external secrets, the value is populated
+// from the secret driver.
+func (a *assignmentSet) secret(secretID string, readTx store.ReadTx) (*api.Secret, error) {


Just a nit, I like to pass the transaction first.

Thanks, makes sense, I didn't notice the order.

cpuguy83 · 2017-07-05T20:59:40Z

manager/dispatcher/assignments.go

+// secret populates the secret value from raft store. For external secrets, the value is populated
+// from the secret driver.
+func (a *assignmentSet) secret(secretID string, readTx store.ReadTx) (*api.Secret, error) {
+	secret := store.GetSecret(readTx, secretID)


Should we only do this if the driver is nil?

Thanks @cpuguy83, as far as I understand no, for two reasons:

We need to fetch the actual Driver object in case additional Driver metadata is needed for initiating the plugin

We need the api.Secret to initiate the assignment. Since I have the secret ID, name and value I can probably generate this object. However, I feel that using the raft store is more robust.
WDYT?

riyazdf · 2017-07-05T22:18:21Z

@liron-l: thanks! I think nullable makes sense, though I'll let you know if I think of a reason we should make it non-nullable 👍

liron-l · 2017-07-08T13:13:16Z

Thanks @aaronlehmann, @riyazdf and @cpuguy83, I hope the last iteration satisfies all requirements.

cpuguy83 · 2017-07-10T17:59:41Z

manager/dispatcher/assignments.go

+// from the secret driver.
+func (a *assignmentSet) secret(readTx store.ReadTx, secretID string) (*api.Secret, error) {
+	secret := store.GetSecret(readTx, secretID)
+	if secret == nil {


Looks like this is returning any time the local store doesn't have the secret.

I still think it's better to check the local store only if driver was not specified.

@cpuguy83 but if a driver was specified, I still need to query the api.Secret object to fetch the api.Driver (to correctly materialize the secret).

Oh, I see now.
Thanks.

I'm not sure I follow: doesn't this function return if the store.GetSecret returns nil?

The secret metadata is still stored in the raft store and must exist there.

@riyazdf, just to clarify, following @cpuguy83 comments, I consolidate all secret fetching functionality to a single function.
The flow:

Fetch the secret (since the secret is required by the task, return error if not found)

If secrets driver is defined, fetch the secret value from the driver (otherwise return the secret value)
Let me know if additional refactoring is needed.

got it. Thanks @cpuguy83 and @liron-l for clarifying. :)

cpuguy83 · 2017-07-10T18:04:41Z

manager/drivers/secrets.go

+
+// SecretsProviderRequest is the request specification for retrieving secrets from plugins.
+type SecretsProviderRequest struct {
+	Name string `json:"name"` // Name is the name of the secret plugin


Thanks fixed.

cpuguy83

Not all that familiar with the swarmkit side of things, but this code LGTM.

Not a fan of plugingetter, but it's what's available from docker right now.

riyazdf

LGTM, thank you @liron-l for the hard work!

liron-l · 2017-07-11T18:20:58Z

Thanks so much @riyazdf @cpuguy83 @aaronlehmann, @diogomonica who else needs to review this commit before merging?

aaronlehmann · 2017-07-11T21:58:49Z

manager/dispatcher/assignments.go

@@ -1,9 +1,12 @@
 package dispatcher

 import (
+	"fmt"


minor nit: normally there would be a blank line here

Thanks @aaronlehmann fixed.

This commit extends SwarmKit secret management with pluggable secret backends support. The solution uses the existing docker plugin framework for loading plugins and the existing SwarmKit data backend for storing them. The approach is to add a new `driver` parameter to existing secrets, which defines whether the values are taken as is or fetched from one of the secret plugins. The loading of secrets is done using the standard docker plugin infrastructure, which is already accessible in SwarmKit and used in other flows (e.g., networking). The fetched values are evaluated before assigning them to worker nodes, so the payload is not stored in the raft store. Remarks: * I've added support for mocking the plugin subsystem when settings up the controlapi server. I preferred this approach over loading the full plugin subsystem in UT. Work still needed in this CR: - [ ] More unit tests (pending initial iteration) - [ ] Customized error handling (e.g., customize error string for Not Found) Work still needed to complete this feature: - [ ] Inject secrets as part of plugin initialization - [ ] CLI support in docker - [ ] Docs - [ ] Support scheduling plugins in swarm moby/moby#33575 Signed-off-by: liron <liron@twistlock.com>

liron-l · 2017-07-14T17:43:03Z

Thanks @aaronlehmann, @cpuguy83, @riyazdf, @diogomonica.
What is the process of merging this change?

liron-l force-pushed the pluggable_secret_backend branch 3 times, most recently from 23a2ee6 to f966946 Compare June 11, 2017 11:38

aaronlehmann reviewed Jun 12, 2017

View reviewed changes

cpuguy83 suggested changes Jun 12, 2017

View reviewed changes

aaronlehmann reviewed Jun 12, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch 13 times, most recently from 6d827fc to 1ac46c1 Compare June 18, 2017 13:03

diogomonica reviewed Jun 18, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from 1ac46c1 to 4a4f588 Compare June 19, 2017 08:04

aaronlehmann reviewed Jun 19, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from 4a4f588 to 8b88006 Compare June 22, 2017 08:53

liron-l force-pushed the pluggable_secret_backend branch from 1ab8d83 to 8533e89 Compare July 2, 2017 10:27

liron-l force-pushed the pluggable_secret_backend branch from 8533e89 to 84bd7cc Compare July 5, 2017 17:42

cpuguy83 suggested changes Jul 5, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from 56b9b2f to dc7d83a Compare July 5, 2017 20:31

liron-l force-pushed the pluggable_secret_backend branch from dc7d83a to e776b92 Compare July 5, 2017 20:37

riyazdf reviewed Jul 5, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from e776b92 to 7df28e3 Compare July 5, 2017 20:53

cpuguy83 suggested changes Jul 5, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from 7df28e3 to b400d18 Compare July 5, 2017 21:04

cpuguy83 reviewed Jul 10, 2017

View reviewed changes

cpuguy83 approved these changes Jul 10, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from b400d18 to 184a88a Compare July 10, 2017 18:23

riyazdf approved these changes Jul 10, 2017

View reviewed changes

aaronlehmann approved these changes Jul 11, 2017

View reviewed changes

liron-l force-pushed the pluggable_secret_backend branch from 184a88a to 93a76cb Compare July 12, 2017 03:31

liron-l force-pushed the pluggable_secret_backend branch from 93a76cb to e9a7bc0 Compare July 12, 2017 03:40

thaJeztah added the impact/changelog label Jul 13, 2017

diogomonica approved these changes Jul 14, 2017

View reviewed changes

diogomonica merged commit eebac27 into moby:master Jul 14, 2017

Pluggable secret backend #2239

Pluggable secret backend #2239

Conversation

liron-l commented Jun 11, 2017 • edited by diogomonica Loading

codecov bot commented Jun 11, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l commented Jun 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diogomonica commented Jun 18, 2017

liron-l commented Jun 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l commented Jun 22, 2017

liron-l commented Jun 27, 2017 • edited Loading

liron-l commented Jul 2, 2017

liron-l commented Jul 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l commented Jul 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l commented Jul 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l Jul 5, 2017 • edited Loading

Choose a reason for hiding this comment

riyazdf commented Jul 5, 2017

liron-l commented Jul 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpuguy83 left a comment

Choose a reason for hiding this comment

riyazdf left a comment

Choose a reason for hiding this comment

liron-l commented Jul 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liron-l commented Jul 14, 2017

liron-l commented Jun 11, 2017 •

edited by diogomonica

Loading

codecov bot commented Jun 11, 2017 •

edited

Loading

liron-l commented Jun 27, 2017 •

edited

Loading

liron-l commented Jul 5, 2017 •

edited

Loading

liron-l Jul 5, 2017 •

edited

Loading