CRD support

[ytsarev]

Is there any suggested way how to test Custom Resources with terratest ?

I would like to have ability to at least assert specific Status of custom resource.

Is there support or maybe some workaround to do so in the terratest based tests ?

[yorinasub17]

Ideally you can use the struct from the operator/custom resource definition source to render it out (e.g., see this test, where we use the prometheus operator source). Barring that, you can use a map[string]interface{} type to read out the json into a unstructured map for doing the checks. E.g.,

	rendered := map[string]interface{}
	require.NoError(t, yaml.Unmarshal([]byte(out), &rendered))
        assert.Equal(t, rendered["status"], "some-status-check")

You might have to do some type conversions for this to work, e.g. rendered["status"].(string).

[yorinasub17]

Oh wait, do you mean how to query the custom resource using the API as opposed to helm template? If so, I am not sure if we have a nice way to do that yet. I unfortunately don't have a lot of experience working with custom resources so haven't gotten too deep into it.

You can try using the dynamic client to query it out. See the example from client-go. You should be able to get the authenticated client from a KubectlOptions object with the following code:

func getDynamicClient(t testing.TestingT, options *KubectlOptions) (dynamic.Interface, error) {
	var err error

	kubeConfigPath, err := options.GetConfigPath(t)
	if err != nil {
		return nil, err
	}
	logger.Logf(t, "Configuring kubectl using config file %s with context %s", kubeConfigPath, options.ContextName)
	// Load API config (instead of more low level ClientConfig)
	config, err := LoadApiClientConfigE(kubeConfigPath, options.ContextName)
	if err != nil {
		return nil, err
	}

	clientset, err := dynamic.NewForConfig(config)
	if err != nil {
		return nil, err
	}

	return clientset, nil
}

[ytsarev]

@yorinasub17 thanks a lot for suggestions!

Dynamic client looks interesting, I was also thinking to hook in client-go based api query in the tests...
But it appeared a little bit too verbose.
So currently I'm working it around with kubectl + jsonpath like https://github.com/AbsaOSS/ohmyglb/pull/76/files#diff-002d5e43a8db27fc6389f813c803d55fR89

It also follow the idea of terratest to be very end to end and use the same tooling for tests in backend as in standard CLI interaction.

Works fine for simple checks and queries but is not incredibly nice :)

Assuming I want to extend terratest with some abstracted CRD support - which place in the codebase should I look at first ? Could you suggest some best examples of the new modules creation?

I feel I will hit some roof soon with my current approach and will have to extend it and obviously eventually contribute.

[yorinasub17]

Assuming I want to extend terratest with some abstracted CRD support - which place in the codebase should I look at first ? Could you suggest some best examples of the new modules creation?

I would suggest adding the functions into the k8s module first, since it will be related to k8s. I don't think it will be necessary to have a dedicated new module for it, and I assume you will want to use the same KubectlOptions object to configure it.

I think the challenge with a module for CRDs is the fact that they are dynamic objects to being with, which makes it hard to get the static types out if you want to use the API directly. So the options for a clean, flexible abstraction are probably limited.

[ytsarev]

So I guess everything is more or less clear. Thanks a ton for clarification, @yorinasub17 ! Closing the issue