Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
fbeadf2
pkg/test/context.go: use nanoseconds instead of seconds
AlexNPavel Mar 12, 2019
5df3a9e
doc/test-framework/writing...: update doc
AlexNPavel Mar 13, 2019
bb52ce2
pkg/test,test/e2e: clean up and expose test framework Setup func
AlexNPavel Mar 13, 2019
05535c9
pkg/test,test/e2e: expose CreateFromYAML and clean up e2e tests
AlexNPavel Mar 13, 2019
f0fee70
pkg/test: add godocs
AlexNPavel Mar 13, 2019
06f4086
pkg/test/resource_creator: add SetNamespace function
AlexNPavel Mar 13, 2019
bd12703
pkg/test: expose SingleNamespace and prevent nil pointer
AlexNPavel Mar 14, 2019
ee6c9dc
pkg/test/context: don't make cleanup error fatal
AlexNPavel Mar 14, 2019
b73cd05
test/test-framework/.../namespace-init: reset file
AlexNPavel Mar 18, 2019
c7b682b
pkg/test: fix godoc punctuation
AlexNPavel Mar 18, 2019
3687a63
Merge branch 'master' into test-framework-expose
AlexNPavel Apr 3, 2019
f0d12ce
Update pkg/test/resource_creator.go
lilic Apr 10, 2019
2bb88b1
pkg/test: don't make SingleNamespace a pointer
AlexNPavel Apr 10, 2019
3ad807f
test/e2e: rearrange an import
AlexNPavel Apr 10, 2019
5a174b1
Merge branch 'test-framework-expose' of github.com:AlexNPavel/operato…
AlexNPavel Apr 10, 2019
f09bb2b
pkg/test: change Setup signature and unexpose singleNamespace
AlexNPavel Apr 10, 2019
a1faddf
Merge branch 'master' into test-framework-expose
AlexNPavel Apr 10, 2019
735c154
pkg/test: fix in cluster testing mode
AlexNPavel Apr 10, 2019
17f3575
Merge branch 'master' into test-framework-expose
AlexNPavel Apr 10, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 18 additions & 16 deletions doc/test-framework/writing-e2e-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ defer ctx.Cleanup()
```

Now that there is a `TestCtx`, the test's Kubernetes resources (specifically the test namespace,
Service Account, RBAC, and Operator deployment in `local` testing; just the Operator deployment
Service Account, Role, Role Binding, and Operator deployment in `local` testing; just the Operator deployment
in `cluster` testing) can be initialized:

```go
Expand Down Expand Up @@ -226,9 +226,9 @@ functions will automatically be run since they were deferred when the TestCtx wa
## Running the Tests

To make running the tests simpler, the `operator-sdk` CLI tool has a `test` subcommand that can configure
default test settings, such as locations of your global resource manifest file (by default
`deploy/crd.yaml`) and your namespaced resource manifest file (by default `deploy/service_account.yaml` concatenated with
`deploy/rbac.yaml` and `deploy/operator.yaml`), and allows the user to configure runtime options. There are 2 ways to use the
default test settings, such as locations of your global resource manifest file (by default all CRDs in
`deploy/crds`) and your namespaced resource manifest file (by default `deploy/service_account.yaml` concatenated with
`deploy/role.yaml`, `deploy/role_binding.yaml`, and `deploy/operator.yaml`), and allows the user to configure runtime options. There are 2 ways to use the
subcommand: local and cluster.

### Local
Expand Down Expand Up @@ -295,7 +295,9 @@ will result in undefined behavior. This is an example `go test` equivalent to th
# Combine service_account, rbac, operator manifest into namespaced manifest
$ cp deploy/service_account.yaml deploy/namespace-init.yaml
$ echo -e "\n---\n" >> deploy/namespace-init.yaml
$ cat deploy/rbac.yaml >> deploy/namespace-init.yaml
$ cat deploy/role.yaml >> deploy/namespace-init.yaml
$ echo -e "\n---\n" >> deploy/namespace-init.yaml
$ cat deploy/role_binding.yaml >> deploy/namespace-init.yaml
$ echo -e "\n---\n" >> deploy/namespace-init.yaml
$ cat deploy/operator.yaml >> deploy/namespace-init.yaml
# Run tests
Expand Down Expand Up @@ -352,24 +354,24 @@ in your cluster. You can do this with `kubectl`:
$ kubectl get namespaces

Example Output:
NAME STATUS AGE
default Active 2h
kube-public Active 2h
kube-system Active 2h
main-1534287036 Active 23s
memcached-memcached-group-cluster-1534287037 Active 22s
memcached-memcached-group-cluster2-1534287037 Active 22s
NAME STATUS AGE
default Active 2h
kube-public Active 2h
kube-system Active 2h
memcached-memcached-group-cluster-1552500058464380681 Active 5s
memcached-memcached-group-cluster2-1552500058464409898 Active 5s
operator-sdk-1552500057336429125 Active 6s
```

The names of the namespaces will be either start with `main` or with the name of the tests and the suffix will
be a Unix timestamp (number of seconds since January 1, 1970 00:00 UTC). Kubectl can be used to delete these
The names of the namespaces will be either start with `operator-sdk` or with the name of the tests and the suffix will
be a Unix nanosecond timestamp (number of nanoseconds since January 1, 1970 00:00 UTC). Kubectl can be used to delete these
namespaces and the resources in those namespaces:

```shell
$ kubectl delete namespace main-153428703
$ kubectl delete namespace operator-sdk-1552500057336429125
```

Since the CRD is not namespaced, it must be deleted separately. Clean up the CRD created by the tests using the CRD manifest `deploy/crd.yaml`:
Since the CRD is not namespaced, it must be deleted separately. Clean up the CRD created by the tests using the CRD manifest(s) in `deploy/crds`:

```shell
$ kubectl delete -f deploy/crds/cache_v1alpha1_memcached_crd.yaml
Expand Down
6 changes: 6 additions & 0 deletions pkg/test/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ type frameworkClient struct {

var _ FrameworkClient = &frameworkClient{}

// FrameworkClient is a wrapper for the controller-runtime client with a modified Create function
// that automatically adds a cleanup function for the created resource.
type FrameworkClient interface {
Get(gCtx goctx.Context, key dynclient.ObjectKey, obj runtime.Object) error
List(gCtx goctx.Context, opts *dynclient.ListOptions, list runtime.Object) error
Expand Down Expand Up @@ -89,18 +91,22 @@ func (f *frameworkClient) Create(gCtx goctx.Context, obj runtime.Object, cleanup
return nil
}

// Get is a simple wrapper for the controller-runtime client's Get function.
func (f *frameworkClient) Get(gCtx goctx.Context, key dynclient.ObjectKey, obj runtime.Object) error {
return f.Client.Get(gCtx, key, obj)
}

// List is a simple wrapper for the controller-runtime client's List function.
func (f *frameworkClient) List(gCtx goctx.Context, opts *dynclient.ListOptions, list runtime.Object) error {
return f.Client.List(gCtx, opts, list)
}

// Delete is a simple wrapper for the controller-runtime client's Delete function.
func (f *frameworkClient) Delete(gCtx goctx.Context, obj runtime.Object, opts ...dynclient.DeleteOptionFunc) error {
return f.Client.Delete(gCtx, obj, opts...)
}

// Update is a simple wrapper for the controller-runtime client's Update function.
func (f *frameworkClient) Update(gCtx goctx.Context, obj runtime.Object) error {
return f.Client.Update(gCtx, obj)
}
17 changes: 10 additions & 7 deletions pkg/test/context.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,15 @@ import (
log "github.com/sirupsen/logrus"
)

// TestCtx contains the state of a test, which includes ID, namespace, and cleanup functions.
type TestCtx struct {
id string
cleanupFns []cleanupFn
namespace string
t *testing.T
}

// CleanupOptions allows for configuration of resource cleanup functions.
type CleanupOptions struct {
TestContext *TestCtx
Timeout time.Duration
Expand All @@ -38,9 +40,11 @@ type CleanupOptions struct {

type cleanupFn func() error

// NewTestCtx returns a new TestCtx object.
func NewTestCtx(t *testing.T) *TestCtx {
var prefix string
if t != nil {
// Use the name of the test as the prefix
// TestCtx is used among others for namespace names where '/' is forbidden
prefix = strings.TrimPrefix(
strings.Replace(
Expand All @@ -52,38 +56,37 @@ func NewTestCtx(t *testing.T) *TestCtx {
"test",
)
} else {
prefix = "main"
prefix = "operator-sdk"
}

id := prefix + "-" + strconv.FormatInt(time.Now().Unix(), 10)
// add a creation timestamp to the ID
id := prefix + "-" + strconv.FormatInt(time.Now().UnixNano(), 10)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason for switching to UnixNano?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To prevent collisions. Collisions shouldn't have happened before due to the namespace being based on the name of the test, but now it could be more likely to happen if users run create multiple contexts in parallel since those would use the same name and potentially be created in the same second. Using nanoseconds should prevent collisions.

return &TestCtx{
id: id,
t: t,
}
}

// GetID returns the ID of the TestCtx.
func (ctx *TestCtx) GetID() string {
return ctx.id
}

// Cleanup runs all the TestCtx's cleanup function in reverse order of their insertion.
func (ctx *TestCtx) Cleanup() {
failed := false
for i := len(ctx.cleanupFns) - 1; i >= 0; i-- {
err := ctx.cleanupFns[i]()
if err != nil {
failed = true
if ctx.t != nil {
ctx.t.Errorf("A cleanup function failed with error: (%v)\n", err)
} else {
log.Errorf("A cleanup function failed with error: (%v)", err)
}
}
}
if ctx.t == nil && failed {
log.Fatal("A cleanup function failed")
}
}

// AddCleanupFn adds a new cleanup function to the TestCtx.
func (ctx *TestCtx) AddCleanupFn(fn cleanupFn) {
ctx.cleanupFns = append(ctx.cleanupFns, fn)
}
3 changes: 2 additions & 1 deletion pkg/test/e2eutil/wait_util.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func WaitForDeployment(t *testing.T, kubeclient kubernetes.Interface, namespace,
}

// WaitForOperatorDeployment has the same functionality as WaitForDeployment but will no wait for the deployment if the
// test was run with a locally run operator (--up-local flag)
// test was run with a locally run operator (--up-local flag).
func WaitForOperatorDeployment(t *testing.T, kubeclient kubernetes.Interface, namespace, name string, replicas int, retryInterval, timeout time.Duration) error {
return waitForDeployment(t, kubeclient, namespace, name, replicas, retryInterval, timeout, true)
}
Expand Down Expand Up @@ -71,6 +71,7 @@ func waitForDeployment(t *testing.T, kubeclient kubernetes.Interface, namespace,
return nil
}

// WaitForDeletion waits for a given object to be fully deleted according to the apiserver before returning.
func WaitForDeletion(t *testing.T, dynclient client.Client, obj runtime.Object, retryInterval, timeout time.Duration) error {
key, err := client.ObjectKeyFromObject(obj)
if err != nil {
Expand Down
53 changes: 26 additions & 27 deletions pkg/test/framework.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,36 +40,45 @@ import (
)

var (
// Global framework struct
// Global is a global framework struct that the test can use.
Global *Framework
// mutex for AddToFrameworkScheme
mutex = sync.Mutex{}
// whether to run tests in a single namespace
singleNamespace *bool
// singleNamespaceInternal determines whether tests are to be run in a single namespace or not.
singleNamespaceInternal bool
// decoder used by createFromYaml
dynamicDecoder runtime.Decoder
// restMapper for the dynamic client
restMapper *restmapper.DeferredDiscoveryRESTMapper
)

// Framework contains all relevant variables needed for running tests with the operator-sdk.
type Framework struct {
Client *frameworkClient
KubeConfig *rest.Config
KubeClient kubernetes.Interface
Scheme *runtime.Scheme
NamespacedManPath *string
NamespacedManPath string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this done initially in case we could have a nil and do we not have that case anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why it was a pointer. We don't have any nil checks anywhere, so this doesn't affect anything. Just makes more sense to have it as a normal string IMO

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, will this be a breaking change for the users, should we explicitly mention it in the changelog.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's something the users should have been accessing directly in the first place. It might make sense to unexport this field (and maybe a few others)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @AlexNPavel,

I think that namespace can be null, is not possible the user/dev use it as an lib/imported in their project?

Namespace string
LocalOperator bool
}

func setup(kubeconfigPath, namespacedManPath *string, localOperator bool) error {
namespace := ""
if *singleNamespace {
namespace = os.Getenv(TestNamespaceEnv)
// Setup initializes the Global.Framework variable and its fields.
func Setup(kubeconfigPath, namespacedManPath, namespace string, singleNamespace, localOperator bool) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if there are problematic implications, but can this function signature be refactored to something like:

type FrameworkOptions struct {
    KubeconfigPath string
    NamespacedManPath string
    Namespace string
    SingleNamespace bool
    LocalOperator bool
}

func NewFramework(opts FrameworkOptions) (*Framework, error)

Then AddToFrameworkScheme() could be a method on the Framework struct and renamed to AddToScheme().

I realize this would be a breaking change, so that's worth some discussion.

I think we should try to avoid having global mutable state as much as possible and favor returning and passing values around instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The biggest difficulty with not having the global Framework variable is how the framework would be passed around in the tests. We don't have a simple way to pass down variables through tests without global variables and having the global variable makes it easier for us to implement the main entry that sets up a single framework that all tests run after the entry can use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One way to avoid the global state in pkg/test would be to have a global framework variable in the user's main_test.go file that gets set to the result of NewFramework. But this would be a pretty important breaking change that we would have to make very clear to users.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I think I'm understanding this a little bit better now having taken a look at https://github.com/operator-framework/operator-sdk/blob/master/test/test-framework/test/e2e/memcached_test.go.

If I understand correctly, right now it's basically required for tests to call f.MainEntry(m) to get Global setup properly. My concern is that if we open up more of the test framework and make it possible to run tests without calling f.MainEntry, then users will have to know the right order of calls to make before Global is safe to use.

If getting rid of the exported Global variable will cause too big of a breaking change, would it be possible to make its zero value (or initialized value) usable?

if namespace != "" && !singleNamespace {
return fmt.Errorf("oneNamespace must be set to true if namespace is set")
}
singleNamespaceInternal = singleNamespace
var err error
var kubeconfig *rest.Config
if *kubeconfigPath == "incluster" {
if kubeconfigPath == "incluster" {
// when running with an InCluster config, we don't have permission to create new namespaces, so we must be in single namespace mode
if singleNamespaceInternal != true {
return fmt.Errorf("singleNamespace must be set to true for in cluster testing mode")
}
if len(namespace) == 0 {
return fmt.Errorf("namespace must be set for in cluster testing mode")
}
// Work around https://github.com/kubernetes/kubernetes/issues/40973
if len(os.Getenv("KUBERNETES_SERVICE_HOST")) == 0 {
addrs, err := net.LookupHost("kubernetes.default.svc")
Expand All @@ -86,15 +95,10 @@ func setup(kubeconfigPath, namespacedManPath *string, localOperator bool) error
}
}
kubeconfig, err = rest.InClusterConfig()
*singleNamespace = true
namespace = os.Getenv(TestNamespaceEnv)
if len(namespace) == 0 {
return fmt.Errorf("test namespace env not set")
}
} else {
var kcNamespace string
kubeconfig, kcNamespace, err = k8sInternal.GetKubeconfigAndNamespace(*kubeconfigPath)
if *singleNamespace && namespace == "" {
kubeconfig, kcNamespace, err = k8sInternal.GetKubeconfigAndNamespace(kubeconfigPath)
if singleNamespaceInternal && namespace == "" {
namespace = kcNamespace
}
}
Expand Down Expand Up @@ -147,34 +151,29 @@ type addToSchemeFunc func(*runtime.Scheme) error
// }
// The List object is needed because the CRD has not always been fully registered
// by the time this function is called. If the CRD takes more than 5 seconds to
// become ready, this function throws an error
// become ready, this function throws an error.
func AddToFrameworkScheme(addToScheme addToSchemeFunc, obj runtime.Object) error {
mutex.Lock()
defer mutex.Unlock()
err := addToScheme(Global.Scheme)
if err != nil {
return err
return fmt.Errorf("failed to update global scheme: %v", err)
}
restMapper.Reset()
dynClient, err := dynclient.New(Global.KubeConfig, dynclient.Options{Scheme: Global.Scheme, Mapper: restMapper})
if err != nil {
return fmt.Errorf("failed to initialize new dynamic client: (%v)", err)
}
err = wait.PollImmediate(time.Second, time.Second*10, func() (done bool, err error) {
if *singleNamespace {
err = dynClient.List(goctx.TODO(), &dynclient.ListOptions{Namespace: Global.Namespace}, obj)
if singleNamespaceInternal {
err = Global.Client.List(goctx.TODO(), &dynclient.ListOptions{Namespace: Global.Namespace}, obj)
} else {
err = dynClient.List(goctx.TODO(), &dynclient.ListOptions{Namespace: "default"}, obj)
err = Global.Client.List(goctx.TODO(), &dynclient.ListOptions{Namespace: "default"}, obj)
}
if err != nil {
restMapper.Reset()
return false, nil
}
Global.Client = &frameworkClient{Client: dynClient}
return true, nil
})
if err != nil {
return fmt.Errorf("failed to build the dynamic client: %v", err)
return fmt.Errorf("failed to update the client restmapper: %v", err)
}
dynamicDecoder = serializer.NewCodecFactory(Global.Scheme).UniversalDeserializer()
return nil
Expand Down
14 changes: 11 additions & 3 deletions pkg/test/main_entry.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,28 @@ const (
LocalOperatorFlag = "localOperator"
)

// MainEntry parses the flags set by the operator-sdk test command, configures the testing environment, and then
// runs the tests.
func MainEntry(m *testing.M) {
projRoot := flag.String(ProjRootFlag, "", "path to project root")
kubeconfigPath := flag.String(KubeConfigFlag, "", "path to kubeconfig")
globalManPath := flag.String(GlobalManPathFlag, "", "path to operator manifest")
namespacedManPath := flag.String(NamespacedManPathFlag, "", "path to rbac manifest")
singleNamespace = flag.Bool(SingleNamespaceFlag, false, "enable single namespace mode")
localOperator := flag.Bool(LocalOperatorFlag, false, "enable if operator is running locally (not in cluster)")
flag.BoolVar(&singleNamespaceInternal, SingleNamespaceFlag, false, "enable single namespace mode")
flag.Parse()
// go test always runs from the test directory; change to project root
err := os.Chdir(*projRoot)
if err != nil {
log.Fatalf("Failed to change directory to project root: %v", err)
}
if err := setup(kubeconfigPath, namespacedManPath, *localOperator); err != nil {
namespace := ""
if singleNamespaceInternal || *kubeconfigPath == "incluster" {
namespace = os.Getenv(TestNamespaceEnv)
// if kubeconfig is set to incluster, make sure single namespace mode is set
singleNamespaceInternal = true
}
if err := Setup(*kubeconfigPath, *namespacedManPath, namespace, singleNamespaceInternal, *localOperator); err != nil {
log.Fatalf("Failed to set up framework: %v", err)
}
// setup local operator command, but don't start it yet
Expand Down Expand Up @@ -126,7 +134,7 @@ func MainEntry(m *testing.M) {
if err != nil {
log.Fatalf("Failed to read global resource manifest: %v", err)
}
err = ctx.createFromYAML(globalYAML, true, &CleanupOptions{TestContext: ctx})
err = ctx.CreateFromYAML(globalYAML, true, &CleanupOptions{TestContext: ctx})
if err != nil {
log.Fatalf("Failed to create resource(s) in global resource manifest: %v", err)
}
Expand Down
Loading