E2E TestEnterpriseTrialLicense fails

[sebgl]

=== RUN   TestEnterpriseTrialLicense/Elasticsearch_license_should_be_[basic]
Retries (5m0s timeout): ....................................................................................................
    TestEnterpriseTrialLicense/Elasticsearch_license_should_be_[basic]: utils.go:84: 
        	Error Trace:	utils.go:84
        	Error:      	Received unexpected error:
        	            	expectedTypes license type [ basic] got platinum
        	Test:       	TestEnterpriseTrialLicense/Elasticsearch_license_should_be_[basic]
{"log.level":"error","@timestamp":"2020-05-29T01:38:11.724Z","message":"stopping early","service.version":"0.0.0-00000000","service.type":"eck","ecs.version":"1.4.0","error":"test failure","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/elastic/cloud-on-k8s/test/e2e/test.StepList.RunSequential\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:43\ngithub.com/elastic/cloud-on-k8s/test/e2e/es.TestEnterpriseTrialLicense\n\t/go/src/github.com/elastic/cloud-on-k8s/test/e2e/es/license_test.go:130\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:991"}
    --- FAIL: TestEnterpriseTrialLicense/Elasticsearch_license_should_be_[basic] (300.00s)

https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-stack-versions/101/testReport/

[sebgl]

So I managed to run the test locally (after spending some time figuring out which pubkey/privkey/license I need to extract), and... it succeeded! :|
It seemed to reliably fail in CI though.
Still investigating.

[sebgl]

From the operator logs:

{"log.level":"info","@timestamp":"2020-05-29T01:38:35.699Z","log.logger":"license-controller","message":"Starting reconciliation run","service.version":"1.2.0-57fdf901","service.type":"eck","ecs.version":"1.4.0","iteration":279,"namespace":"e2e-7uo4r-mercury","es_name":"test-es-trial-license-wk7b"}
{"log.level":"debug","@timestamp":"2020-05-29T01:38:35.700Z","log.logger":"license-controller","message":"No enterprise license found. Attempting to remove cluster license secret","service.version":"1.2.0-57fdf901","service.type":"eck","ecs.version":"1.4.0","namespace":"e2e-7uo4r-mercury","es_name":"test-es-trial-license-wk7b"}
{"log.level":"debug","@timestamp":"2020-05-29T01:38:35.705Z","log.logger":"license-controller","message":"Reconcile result","service.version":"1.2.0-57fdf901","service.type":"eck","ecs.version":"1.4.0","requeue":false,"requeueAfter":3600}

It looks like the license controller did its job of removing the cluster license secret.
Then I would expect the Elasticsearch controller to revert back to basic since the secret cannot be found:

	err := c.Get(
		types.NamespacedName{
			Namespace: esCluster.Namespace,
			Name:      esv1.LicenseSecretName(esCluster.Name),
		},
		&license,
	)
	if err != nil {
		if apierrors.IsNotFound(err) {
			if isTrial(current) {
				// this is the case if the user has started a trial on the ES cluster level
				// e.g. from Kibana when trying to access a commercial feature.
				// While this is not a supported use case we tolerate it to avoid a bad user
				// experience because trials can only be started once on the ES cluster level
				return nil
			}
			// no license linked to this cluster. Revert to basic.
			return startBasic(ctx, updater)
		}
		return err
	}

But it didn't happen so I guess it may have taken the isTrial(current) path. Some logs here would have helped debugging.

current comes from the observer, and is based on a request made to Elasticsearch every 10 seconds.

I think the following race condition could have happened:

1. E2E test applies a trial license
2. Observer observes the trial license
3. E2E test applies an enterprise license
4. E2E test removes the enterprise license and the trial license
5. Operator detects there's no license anymore and the last observed license is trial, so there's nothing to do
6. Observer observes the enterprise license, but no more ES reconciliation gets triggered

This seems to be confirmed by the logs, I see a bunch of elasticsearch-controller logs (pasted above), then the observer doing its job but no reconciliation anymore for the elasticsearch-controller.

[sebgl]

I see 2 ways to solve this:

1. In the observer, we trigger a reconciliation whenever there's a change in the observed license.
2. We remove the license API call from the observer altogether, instead we check the license as part of the reconciliation (all the time) - a reconciliation is triggered when the watched license secret gets created/updated/removed already.

I think I prefer option 2.