Is it possible to use a third-party test framework like Ginkgo ?

[kanekoh]

Type of question

Are you asking about community best practices, how to implement a specific feature, or about general context and help around the operator-sdk?

About how to implement e2e test using a third party test framework.

Question

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).

I'm new into operator-sdk and I'd like to know how I implement e2e test using Ginkgo.

Trying to use Ginkgo for e2e test, however I couldn't find any way to pass testing.T into framework.NewContext() and functions like e2e/util/WaitForOperatorDeployment.

Are there any example or documentations for my understanding?

[mkimuram]

I had the exact same use case and managed to implement it like this.

I've tested it with my kind environment like below way and confirmed that it seems to work well.

sed -i "s/REPLACE_IMAGE/memcached:v0.0.1/" deploy/operator.yaml
sed -i "s/Always/IfNotPresent/" deploy/operator.yaml
operator-sdk build memcached:v0.0.1
kind load docker-image memcached:v0.0.1

operator-sdk test local ./test/e2e_ginkgo --debug --go-test-flags="-v -ginkgo.v"
operator-sdk test local ./test/e2e_ginkgo --debug --go-test-flags="-v -ginkgo.v -ginkgo.skip=\[slow\]"

Note that passing testing.T like this way is not recommended in ginkgo. So, it would be nice to avoid passing testing.T in operator-sdk's e2e framework side, in the future. (I guess that just using it for logging won't be harm, although it is too noisy in the last output of the tests. So, I hope my current implementation is still good enough, practically.)

@estroz @kanekoh
Any feedback is welcomed, for I'm planning to implement e2e tests in my project by using this way.
Also, I can create a PR for this sample to operator-sdk-samples, if it makes sense.

[kanekoh]

I am not the expert of Ginkgo/Gomega however, I don't think that is good idea implementing e2e tests using operator-sdk with the unrecommended way which was mentioned the issue you linked.

I would like to chose a recommend way to implement the test cases.

[mkimuram]

So, it would be nice to avoid passing testing.T in operator-sdk's e2e framework side, in the future.

Just for removing dependency to testing.T, changes like this in operator-sdk will be fine, as similar approach was done here. Then, it will allow passing nil to testing.T, so we don't need to access to it in ginkgo test like this.

I guess that it will be difficult to change the interface not to pass testing.T in framework, because there must be already many existing users using current one. Therefore, above approach would be the realistic solution. (Until someone decide to define v2 interface, which I don't think there are such motivations.)

One remaining issue is --skip-cleanup-error won't work well if testing.T is nil, for it relies on t.Failed(). (However, from source code, it allows such a case, for it check t != nil.)

@estroz

Any comments on this?

[estroz]

Even though passing testing.T like you have done is not recommended, it is probably the best way to use Ginkgo/Gomega in conjunction with the SDK's e2e testing library for the foreseeable future.

Instead of continuing development of the SDK's e2e framework, we are going to start promoting envtest as the go-to operator e2e test framework. It has better UX, is under active development, and is already suggested for use with kubebuilder projects (which we are migrating to for Go-based operators). We recently discussed this switch at our issue triage meeting this morning. Docs on this to come.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[camilamacedo86]

It shows solved. in this way, I am closing this one and please feel free to re-open if you need and/or raise new issues as well.