[PoC] Launch test YAMLs directly

[josvazg]

PoC not to merge but to be used for API testing.

All Submissions:

• Have you signed our CLA?

[s-urbaniak]

stylistic nit: remove those extra newlines, const blocks are tight.

[s-urbaniak]

wrap all errors below. In this case:

Suggested change

	return err
	return fmt.Errorf("error ensuring k8s cluster: %w", err)

[s-urbaniak]

@josvazg I see you are inventing your own testing framework here. What about just using https://kuttl.dev/ ?

[s-urbaniak]

See https://kuttl.dev/docs/kuttl-test-harness.html#writing-your-first-test and https://kuttl.dev/docs/api-integration.html#test-harness.

[josvazg]

See https://kuttl.dev/docs/kuttl-test-harness.html#writing-your-first-test and https://kuttl.dev/docs/api-integration.html#test-harness.

This is not a testing framework, it is a simple testing helper. It could be used as a framework maybe, but it is NOT the intent here and for that I agree that KUTTL is a better option.

Think of this as the function I call in Go to get the Atlas infrastructure in the state I need to run a contract test, which is defined in Go code, not in YAML. A test that calls our translation layer that talks to the API using the same code that the reconcile logic uses. If KUTTL could be used in the same unobtrusive way, then maybe we could use it for all, I am not 100% convinced yet.

Sample custom test:

  l := launcher.NewFromEnv(testVersion) // instantiate a launcher
  l.Launch(testYml, launcher.WaitReady("atlasprojects/my-project", 30*time.Second))) // launch the YAML and wait for it
  
  // run custom test code - might not be part of KUTTLE, like a custom go code driven contract test

   l.Cleanup()) // optionally cleanup the infrastructure required for the test (does KUTTL allow for optional cleanup?)

[s-urbaniak]

@josvazg I am leaving the final assertion to you 👍 I didn't have a deeper look at kuttl, just worried about NIH implementations. Again, final call is yours :-)

[josvazg]

@josvazg I am leaving the final assertion to you 👍 I didn't have a deeper look at kuttl, just worried about NIH implementations. Again, final call is yours :-)

I think the best way forward now is to actually force myself to use KUTTL for some real Contract tests, for Atlas Search. If I see no blockers, then we can probably use for everything and forget about this PoC for good.

[helderjs]

It's a mistake or do you really want to set timeout only when it's zero?

[josvazg]

This was a mistake, the logic had to be reversed

[helderjs]

@josvazg This PoC links well to my comment on the CI Revamp document regarding GitHub Actions.

Why didn't you write a Github action workflow for it? I'd say we probably won't need any code for it.
I see your code doing what Github Action should be doing, and I believe that's why @s-urbaniak also saw it as a framework wrapping the tooling needed to achieve the goal. Code is a liability and I prefer to have no code when we already have a toolchain in place to do the job. In the end, you would still use a workflow to trigger this task.
Think that a CI tool like GH action offers us out-of-box multi-stage pipeline and jobs, some level of security, lifecycle control... keep in mind that once you have signaled you will drop this to a new PoC with KUTTL

[josvazg]

Suggested change

	const (
	ExpectedContext = "kind-kind"
	LauncherTestInstall = "test-ako"
	HelmRepoURL = "https://mongodb.github.io/helm-chart"
	RepoRef = "mongodb"
	OperatorChart = "mongodb-atlas-operator"
	AtlasURI = "https://cloud-qa.mongodb.com"
	)
	const (
	ExpectedContext = "kind-kind"
	LauncherTestInstall = "test-ako"
	HelmRepoURL = "https://mongodb.github.io/helm-chart"
	RepoRef = "mongodb"
	OperatorChart = "mongodb-atlas-operator"
	AtlasURI = "https://cloud-qa.mongodb.com"
	)

[josvazg]

@josvazg This PoC links well to my comment on the CI Revamp document regarding GitHub Actions.

Why didn't you write a Github action workflow for it? I'd say we probably won't need any code for it. I see your code doing what Github Action should be doing, and I believe that's why @s-urbaniak also saw it as a framework wrapping the tooling needed to achieve the goal. Code is a liability and I prefer to have no code when we already have a toolchain in place to do the job. In the end, you would still use a workflow to trigger this task. Think that a CI tool like GH action offers us out-of-box multi-stage pipeline and jobs, some level of security, lifecycle control... keep in mind that once you have signaled you will drop this to a new PoC with KUTTL

I want us to be able to do our development without any Github involvement, using the GitHub CI actions or workflow only to automatically re-run tests at the proper time to ensure what we release is correct and did not include any regressions.

The idea is to be able to write contract tests in Go that look like this:

1. Get a particular set of Atlas resources in a particular state.
2. Run one or more contract tests against the Atlas API.
3. Tear down all resources required for the test.

During development, in your laptop and before any code reaches GitHub, you iterate with steps 1-2 until you test all APIs you needed to verify. Then do step 3, when you are done. It is only later on you send a PR and then the CI might also want to run those tests at some point, maybe at merge time only.

This might be doable with KUTTL, because it seems to have a programmatic interface. I will need to test it.
But as a pure GitHub action or workflow it would be completely useless for contract testing.

BTW, even if KUTTL could not do the job fully and this simple launcher would be required, I have the impression it would be able to replace a lot of our current test helper code, which would mean a net removal of code and thus reduced liability instead.

[josvazg]

Going to close this PoC now:

• It was never the intention to merge this at this point.
• It was needed for me as a way to run contract tests, but haven't gotten there yet.
• Seems KUTTL has a programmatic interface that might allow it to be used within tests as a helper, in which case this code will not be needed.
• Should KUTTL prove inadequate later, we can always dig up this PR again.
• The main goal of the PoC is achieved, launching declarative YAML infrastructure for a test OR as a helper to setup before a test is not only doable but also relatively simple.