Skip to content

Commit

Permalink
Merge pull request dexidp#1062 from rithujohn191/crd-migration
Browse files Browse the repository at this point in the history
storage/kubernetes: add CRD support
  • Loading branch information
rithujohn191 committed Sep 14, 2017
2 parents 5006dbe + e7f0a55 commit bb54077
Show file tree
Hide file tree
Showing 5 changed files with 388 additions and 30 deletions.
64 changes: 60 additions & 4 deletions Documentation/storage.md
Expand Up @@ -4,13 +4,68 @@ Dex requires persisting state to perform various tasks such as track refresh tok

Storage breaches are serious as they can affect applications that rely on dex. Dex saves sensitive data in its backing storage, including signing keys and bcrypt'd passwords. As such, transport security and database ACLs should both be used, no matter which storage option is chosen.

## Kubernetes third party resources
## Kubernetes custom resource definitions (CRDs)

__NOTE:__ Dex requires Kubernetes version 1.4+.
__NOTE:__ CRDs are only supported by Kubernetes version 1.7+.

Kubernetes third party resources are a way for applications to create new resources types in the Kubernetes API. This allows dex to run on top of an existing Kubernetes cluster without the need for an external database. While this storage may not be appropriate for a large number of users, it's extremely effective for many Kubernetes use cases.
Kubernetes [custom resource definitions](crd) are a way for applications to create new resources types in the Kubernetes API. The Custom Resource Definition (CRD) API object was introduced in Kubernetes version 1.7 to replace the Third Party Resource (TPR) extension. CRDs allows dex to run on top of an existing Kubernetes cluster without the need for an external database. While this storage may not be appropriate for a large number of users, it's extremely effective for many Kubernetes use cases.

The rest of this section will explore internal details of how dex uses `ThirdPartyResources`. __Admins should not interact with these resources directly__, except when debugging. These resources are only designed to store state and aren't meant to be consumed by humans. For modifying dex's state dynamically see the [API documentation](api.md).
The rest of this section will explore internal details of how dex uses CRDs. __Admins should not interact with these resources directly__, except while debugging. These resources are only designed to store state and aren't meant to be consumed by end users. For modifying dex's state dynamically see the [API documentation](api.md).

The following is an example of the AuthCode resource managed by dex:

```
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
creationTimestamp: 2017-09-13T19:56:28Z
name: authcodes.dex.coreos.com
resourceVersion: "288893"
selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/authcodes.dex.coreos.com
uid: a1cb72dc-98bd-11e7-8f6a-02d13336a01e
spec:
group: dex.coreos.com
names:
kind: AuthCode
listKind: AuthCodeList
plural: authcodes
singular: authcode
scope: Namespaced
version: v1
status:
acceptedNames:
kind: AuthCode
listKind: AuthCodeList
plural: authcodes
singular: authcode
conditions:
- lastTransitionTime: null
message: no conflicts found
reason: NoConflicts
status: "True"
type: NamesAccepted
- lastTransitionTime: 2017-09-13T19:56:28Z
message: the initial names have been accepted
reason: InitialNamesAccepted
status: "True"
type: Established
```

Once the `CustomResourceDefinition` is created, custom resources can be created and stored at a namespace level. The CRD type and the custom resources can be queried, deleted, and edited like any other resource using `kubectl`.

## Kubernetes third party resources(TPRs)

__NOTE:__ TPRs will be deprecated by Kubernetes version 1.8.

The default behavior of dex from release v2.7.0 onwards is to utitlize CRDs to manage its custom resources. If users would like to use dex with a Kubernetes version lower than 1.7, they will have to force dex to use TPRs instead of CRDs by setting the `UseTPR` flag in the storage configuration as shown below:

```
storage:
type: kubernetes
config:
kubeConfigFile: kubeconfig
useTPR: true
```

The `ThirdPartyResource` type acts as a description for the new resource a user wishes to create. The following an example of a resource managed by dex:

Expand Down Expand Up @@ -166,3 +221,4 @@ Any proposal to add a new implementation must address the following:
[issues-transaction-tests]: https://github.com/coreos/dex/issues/600
[k8s-api]: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#concurrency-control-and-consistency
[psql-conn-options]: https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
[crd]: https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/
12 changes: 9 additions & 3 deletions storage/kubernetes/client.go
Expand Up @@ -249,7 +249,7 @@ func (c *client) put(resource, name string, v interface{}) error {
return checkHTTPErr(resp, http.StatusOK)
}

func newClient(cluster k8sapi.Cluster, user k8sapi.AuthInfo, namespace string, logger logrus.FieldLogger) (*client, error) {
func newClient(cluster k8sapi.Cluster, user k8sapi.AuthInfo, namespace string, logger logrus.FieldLogger, useTPR bool) (*client, error) {
tlsConfig := cryptopasta.DefaultTLSConfig()
data := func(b string, file string) ([]byte, error) {
if b != "" {
Expand Down Expand Up @@ -325,13 +325,19 @@ func newClient(cluster k8sapi.Cluster, user k8sapi.AuthInfo, namespace string, l
}
}

// TODO(ericchiang): make API Group and version configurable.
// the API Group and version differ depending on if CRDs or TPRs are used.
apiVersion := "dex.coreos.com/v1"
if useTPR {
apiVersion = "oidc.coreos.com/v1"
}

logger.Infof("kubernetes client apiVersion = %s", apiVersion)
return &client{
client: &http.Client{Transport: t},
baseURL: cluster.Server,
hash: func() hash.Hash { return fnv.New64() },
namespace: namespace,
apiVersion: "oidc.coreos.com/v1",
apiVersion: apiVersion,
logger: logger,
}, nil
}
Expand Down
141 changes: 141 additions & 0 deletions storage/kubernetes/k8sapi/crd_extensions.go
@@ -0,0 +1,141 @@
/*
Copyright 2017 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package k8sapi

// CustomResourceDefinitionSpec describes how a user wants their resource to appear
type CustomResourceDefinitionSpec struct {
// Group is the group this resource belongs in
Group string `json:"group" protobuf:"bytes,1,opt,name=group"`
// Version is the version this resource belongs in
Version string `json:"version" protobuf:"bytes,2,opt,name=version"`
// Names are the names used to describe this custom resource
Names CustomResourceDefinitionNames `json:"names" protobuf:"bytes,3,opt,name=names"`

// Scope indicates whether this resource is cluster or namespace scoped. Default is namespaced
Scope ResourceScope `json:"scope" protobuf:"bytes,4,opt,name=scope,casttype=ResourceScope"`
}

// CustomResourceDefinitionNames indicates the names to serve this CustomResourceDefinition
type CustomResourceDefinitionNames struct {
// Plural is the plural name of the resource to serve. It must match the name of the CustomResourceDefinition-registration
// too: plural.group and it must be all lowercase.
Plural string `json:"plural" protobuf:"bytes,1,opt,name=plural"`
// Singular is the singular name of the resource. It must be all lowercase Defaults to lowercased <kind>
Singular string `json:"singular,omitempty" protobuf:"bytes,2,opt,name=singular"`
// ShortNames are short names for the resource. It must be all lowercase.
ShortNames []string `json:"shortNames,omitempty" protobuf:"bytes,3,opt,name=shortNames"`
// Kind is the serialized kind of the resource. It is normally CamelCase and singular.
Kind string `json:"kind" protobuf:"bytes,4,opt,name=kind"`
// ListKind is the serialized kind of the list for this resource. Defaults to <kind>List.
ListKind string `json:"listKind,omitempty" protobuf:"bytes,5,opt,name=listKind"`
}

// ResourceScope is an enum defining the different scopes availabe to a custom resource
type ResourceScope string

const (
// ClusterScoped is the `cluster` scope for a custom resource.
ClusterScoped ResourceScope = "Cluster"
// NamespaceScoped is the `namespaced` scope for a custom resource.
NamespaceScoped ResourceScope = "Namespaced"
)

// ConditionStatus reflects if a resource
type ConditionStatus string

// These are valid condition statuses. "ConditionTrue" means a resource is in the condition.
// "ConditionFalse" means a resource is not in the condition. "ConditionUnknown" means kubernetes
// can't decide if a resource is in the condition or not. In the future, we could add other
// intermediate conditions, e.g. ConditionDegraded.
const (
ConditionTrue ConditionStatus = "True"
ConditionFalse ConditionStatus = "False"
ConditionUnknown ConditionStatus = "Unknown"
)

// CustomResourceDefinitionConditionType is a valid value for CustomResourceDefinitionCondition.Type
type CustomResourceDefinitionConditionType string

const (
// Established means that the resource has become active. A resource is established when all names are
// accepted without a conflict for the first time. A resource stays established until deleted, even during
// a later NamesAccepted due to changed names. Note that not all names can be changed.
Established CustomResourceDefinitionConditionType = "Established"
// NamesAccepted means the names chosen for this CustomResourceDefinition do not conflict with others in
// the group and are therefore accepted.
NamesAccepted CustomResourceDefinitionConditionType = "NamesAccepted"
// Terminating means that the CustomResourceDefinition has been deleted and is cleaning up.
Terminating CustomResourceDefinitionConditionType = "Terminating"
)

// CustomResourceDefinitionCondition contains details for the current condition of this pod.
type CustomResourceDefinitionCondition struct {
// Type is the type of the condition.
Type CustomResourceDefinitionConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=CustomResourceDefinitionConditionType"`
// Status is the status of the condition.
// Can be True, False, Unknown.
Status ConditionStatus `json:"status" protobuf:"bytes,2,opt,name=status,casttype=ConditionStatus"`
// Last time the condition transitioned from one status to another.
// +optional
LastTransitionTime Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,3,opt,name=lastTransitionTime"`
// Unique, one-word, CamelCase reason for the condition's last transition.
// +optional
Reason string `json:"reason,omitempty" protobuf:"bytes,4,opt,name=reason"`
// Human-readable message indicating details about last transition.
// +optional
Message string `json:"message,omitempty" protobuf:"bytes,5,opt,name=message"`
}

// CustomResourceDefinitionStatus indicates the state of the CustomResourceDefinition
type CustomResourceDefinitionStatus struct {
// Conditions indicate state for particular aspects of a CustomResourceDefinition
Conditions []CustomResourceDefinitionCondition `json:"conditions" protobuf:"bytes,1,opt,name=conditions"`

// AcceptedNames are the names that are actually being used to serve discovery
// They may be different than the names in spec.
AcceptedNames CustomResourceDefinitionNames `json:"acceptedNames" protobuf:"bytes,2,opt,name=acceptedNames"`
}

// CustomResourceCleanupFinalizer is the name of the finalizer which will delete instances of
// a CustomResourceDefinition
const CustomResourceCleanupFinalizer = "customresourcecleanup.apiextensions.k8s.io"

// +genclient
// +genclient:nonNamespaced
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format
// <.spec.name>.<.spec.group>.
type CustomResourceDefinition struct {
TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

// Spec describes how the user wants the resources to appear
Spec CustomResourceDefinitionSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// Status indicates the actual state of the CustomResourceDefinition
Status CustomResourceDefinitionStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// CustomResourceDefinitionList is a list of CustomResourceDefinition objects.
type CustomResourceDefinitionList struct {
TypeMeta `json:",inline"`
ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

// Items individual CustomResourceDefinitions
Items []CustomResourceDefinition `json:"items" protobuf:"bytes,2,rep,name=items"`
}
66 changes: 43 additions & 23 deletions storage/kubernetes/storage.go
Expand Up @@ -38,6 +38,7 @@ const (
type Config struct {
InCluster bool `json:"inCluster"`
KubeConfigFile string `json:"kubeConfigFile"`
UseTPR bool `json:"useTPR"` // Flag option to use TPRs instead of CRDs
}

// Open returns a storage using Kubernetes third party resource.
Expand All @@ -52,9 +53,9 @@ func (c *Config) Open(logger logrus.FieldLogger) (storage.Storage, error) {
// open returns a kubernetes client, initializing the third party resources used
// by dex.
//
// errOnTPRs controls if errors creating the resources cause this method to return
// errOnResources controls if errors creating the resources cause this method to return
// immediately (used during testing), or if the client will asynchronously retry.
func (c *Config) open(logger logrus.FieldLogger, errOnTPRs bool) (*client, error) {
func (c *Config) open(logger logrus.FieldLogger, errOnResources bool) (*client, error) {
if c.InCluster && (c.KubeConfigFile != "") {
return nil, errors.New("cannot specify both 'inCluster' and 'kubeConfigFile'")
}
Expand All @@ -77,26 +78,26 @@ func (c *Config) open(logger logrus.FieldLogger, errOnTPRs bool) (*client, error
return nil, err
}

cli, err := newClient(cluster, user, namespace, logger)
cli, err := newClient(cluster, user, namespace, logger, c.UseTPR)
if err != nil {
return nil, fmt.Errorf("create client: %v", err)
}

ctx, cancel := context.WithCancel(context.Background())

if !cli.createThirdPartyResources() {
if errOnTPRs {
if !cli.registerCustomResources(c.UseTPR) {
if errOnResources {
cancel()
return nil, fmt.Errorf("failed creating third party resources")
return nil, fmt.Errorf("failed creating custom resources")
}

// Try to synchronously create the third party resources once. This doesn't mean
// Try to synchronously create the custom resources once. This doesn't mean
// they'll immediately be available, but ensures that the client will actually try
// once.
logger.Errorf("failed creating third party resources: %v", err)
logger.Errorf("failed creating custom resources: %v", err)
go func() {
for {
if cli.createThirdPartyResources() {
if cli.registerCustomResources(c.UseTPR) {
return
}

Expand All @@ -109,37 +110,56 @@ func (c *Config) open(logger logrus.FieldLogger, errOnTPRs bool) (*client, error
}()
}

// If the client is closed, stop trying to create third party resources.
// If the client is closed, stop trying to create resources.
cli.cancel = cancel
return cli, nil
}

// createThirdPartyResources attempts to create the third party resources dex
// requires or identifies that they're already enabled. It logs all errors,
// returning true if the third party resources were created successfully.
// registerCustomResources attempts to create the custom resources dex
// requires or identifies that they're already enabled. This function creates
// third party resources(TPRs) or custom resource definitions(CRDs) depending
// on the `useTPR` flag passed in as an argument.
// It logs all errors, returning true if the resources were created successfully.
//
// Creating a third party resource does not mean that they'll be immediately available.
// Creating a custom resource does not mean that they'll be immediately available.
//
// TODO(ericchiang): Provide an option to wait for the third party resources
// to actually be available.
func (cli *client) createThirdPartyResources() (ok bool) {
// TODO(ericchiang): Provide an option to wait for the resources to actually
// be available.
func (cli *client) registerCustomResources(useTPR bool) (ok bool) {
ok = true
for _, r := range thirdPartyResources {
err := cli.postResource("extensions/v1beta1", "", "thirdpartyresources", r)
length := len(customResourceDefinitions)
if useTPR {
length = len(thirdPartyResources)
}

for i := 0; i < length; i++ {
var err error
var resourceName string

if useTPR {
r := thirdPartyResources[i]
err = cli.postResource("extensions/v1beta1", "", "thirdpartyresources", r)
resourceName = r.ObjectMeta.Name
} else {
r := customResourceDefinitions[i]
err = cli.postResource("apiextensions.k8s.io/v1beta1", "", "customresourcedefinitions", r)
resourceName = r.ObjectMeta.Name
}

if err != nil {
switch err {
case storage.ErrAlreadyExists:
cli.logger.Infof("third party resource already created %s", r.ObjectMeta.Name)
cli.logger.Infof("custom resource already created %s", resourceName)
case storage.ErrNotFound:
cli.logger.Errorf("third party resources not found, please enable API group extensions/v1beta1")
cli.logger.Errorf("custom resources not found, please enable the respective API group")
ok = false
default:
cli.logger.Errorf("creating third party resource %s: %v", r.ObjectMeta.Name, err)
cli.logger.Errorf("creating custom resource %s: %v", resourceName, err)
ok = false
}
continue
}
cli.logger.Errorf("create third party resource %s", r.ObjectMeta.Name)
cli.logger.Errorf("create custom resource %s", resourceName)
}
return ok
}
Expand Down

0 comments on commit bb54077

Please sign in to comment.