added sample test config files #740

mjsterckx · 2023-01-06T19:01:16Z

Issue number:

N/A

Description of changes:

Added sample YAML files to be used with cli, as well as a runbook that shows how to populate and use the files.

Testing done:

Populating and running ecs-migration-test.yaml:

 NAME                                TYPE              STATE              PASSED          SKIPPED          FAILED
 aws-ecs-1-initial                   Test              pass               1               0                0
 aws-ecs-2-migrate                   Test              pass               2               0                0
 aws-ecs-3-migrated                  Test              pass               1               0                0
 aws-ecs-4-migrate                   Test              pass               2               0                0
 aws-ecs-5-final                     Test              pass               1               0                0
 external-cluster                    Resource          completed
 external-cluster-instances          Resource          completed

Populating and running ecs-test.yaml:

 NAME                            TYPE               STATE               PASSED           SKIPPED          FAILED
 aws-ecs                         Test               pass                1                0                0
 external-cluster                Resource           completed
 external-cluster-instances      Resource           completed

Populating and running sonobuoy-migration-test.yaml:

 NAME                            TYPE               STATE               PASSED           SKIPPED          FAILED
 aws-k8s-1-initial               Test               pass                1                5772             0
 aws-k8s-2-migrate               Test               pass                2                0                0
 aws-k8s-3-migrated              Test               pass                1                5772             0
 aws-k8s-4-migrate               Test               pass                2                0                0
 aws-k8s-5-final                 Test               pass                1                5772             0
 external-cluster                Resource           completed
 external-cluster-instances      Resource           completed

Populating and running sonobuoy-test.yaml with SONOBUOY-MODE "quick":

 NAME                            TYPE               STATE               PASSED           SKIPPED          FAILED
 aws-k8s                         Test               pass                1                5772             0
 external-cluster                Resource           completed
 external-cluster-instances      Resource           completed

Populating and running vmware-migration-test.yaml:

 NAME                                TYPE              STATE              PASSED          SKIPPED          FAILED
 vmware-k8s-1-initial                Test              pass               1               5772             0
 vmware-k8s-2-migrate                Test              pass               2               0                0
 vmware-k8s-3-migrated               Test              pass               1               5772             0
 vmware-k8s-4-migrate                Test              pass               2               0                0
 vmware-k8s-5-final                  Test              pass               1               5772             0
 vmware-cluster                      Resource          completed
 vmware-cluster-vms                  Resource          completed

Populating and running vmware-sonobuoy-test.yaml:

 NAME                            TYPE               STATE               PASSED           SKIPPED          FAILED
 vmware-k8s                      Test               pass                1                5772             0
 vmware-cluster                  Resource           completed
 vmware-cluster-vms              Resource           completed

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

samples/ecs-test.yaml

ecpullen

Great work. Can we make cluster_name configurable for all crds and have the crds names be related to the cluster name?

samples/RUNBOOK.md

samples/eks/sonobuoy-migration-test.yaml

ecpullen

Just a few more nits. Great work so far.

samples/RUNBOOK.md

ecpullen · 2023-01-19T22:55:24Z

samples/RUNBOOK.md

+export UPGRADE_VERSION="v1.11.1"
+export STARTING_VERSION="v1.11.0"


I'm wondering if these should have similar logic to AGENT_IAMGE_VERSION

It won't be as simple as calling something like cli --version. Maybe a curl call to public.ecr.aws/bottlerocket/<agent-image> and figure out the latest versions that way?

Let's save that for a separate pr.

ecpullen

LGTM 🎉

webern

Are these mostly for documentation purposes?

My concern is that the chance of these being incorrect in the future is about 100%.
Can we generate them?

ecpullen · 2023-01-20T18:01:51Z

Are these mostly for documentation purposes?

My concern is that the chance of these being incorrect in the future is about 100%. Can we generate them?

The samples will be used to validate the agents before a new version is released. So they will always be correct with the latest released version.

webern · 2023-01-20T18:04:06Z

Are these mostly for documentation purposes?
My concern is that the chance of these being incorrect in the future is about 100%. Can we generate them?

The samples will be used to validate the agents before a new version is released. So they will always be correct with the latest released version.

Is there any way to apply them to a kind cluster during CI integ tests? We know that the agents in question will not exist and the controller will report errors, but being able to apply these to an integ cluster would ensure they are not out-of-date.

ecpullen · 2023-01-20T18:08:51Z

Are these mostly for documentation purposes?
My concern is that the chance of these being incorrect in the future is about 100%. Can we generate them?

The samples will be used to validate the agents before a new version is released. So they will always be correct with the latest released version.

Is there any way to apply them to a kind cluster during CI integ tests? We know that the agents in question will not exist and the controller will report errors, but being able to apply these to an integ cluster would ensure they are not out-of-date.

That would just validate the crd, it wouldn't validate the individual agents configurations. A followup to this will include tests, deserializing each piece of the manifest and verifying that the configuration matches the agents configuration. (I think that's what you are asking for)

We could also add automatic generation, but I think this is a good first step.

webern · 2023-01-20T18:11:47Z

That would just validate the crd, it wouldn't validate the individual agents configurations. A followup to this will include tests, deserializing each piece of the manifest and verifying that the configuration matches the agents configuration. (I think that's what you are asking for)

Sorry, yes that's even better. Unit tests that deserialize these is what I'm looking for. Any reason not to include those tests now? I'd hate to see that fall by the wayside.

ecpullen · 2023-01-20T18:18:30Z

That would just validate the crd, it wouldn't validate the individual agents configurations. A followup to this will include tests, deserializing each piece of the manifest and verifying that the configuration matches the agents configuration. (I think that's what you are asking for)

Sorry, yes that's even better. Unit tests that deserialize these is what I'm looking for. Any reason not to include those tests now? I'd hate to see that fall by the wayside.

We'd like to use these for the next release. I'm not sure how much effort the unit tests will require.

etungsten

The bash snippets with all the seds is quite unsightly and a bit to much for something I would be comfortable copying and running directly.

I wonder if there's a better way to go about with the templating short of just going full helm charts.

I think I would prefer using heredocs more so it's cleaner. For example,

First, change the variables like <CLUSTER_NAME> to ${CLUSTER_NAME} in the template yaml file. Then...

### Migration Testing on `aws-ecs` Variants

CLUSTER_NAME="x86-64-aws-ecs-1"
OUTPUT_FILE="${CLUSTER_NAME}-migration.yaml"
VARIANT="aws-ecs-1"
ARCHITECTURE="x86_64"
#....

eval "cat > ${OUTPUT_FILE} <<EOF
$(<eks/ecs-migration-test.yaml)
EOF
" 2> /dev/null

Should give you what you want without all the seds. Please try it out.

webern · 2023-01-20T21:43:10Z

^ rebase https://github.com/bottlerocket-os/bottlerocket-test-system/compare/541b478334f96bdb7f3e00442b2d6e3deb976e05..2a5e6d06a49ee740d106a67c736c5f31afcf6089

webern · 2023-01-20T21:48:10Z

^ Add a unit test that will ensure one of the sample config files stays up-to-date. https://github.com/bottlerocket-os/bottlerocket-test-system/compare/2a5e6d06a49ee740d106a67c736c5f31afcf6089..120970a8e60ff6cacafb58fe5f8c24c368042979

webern · 2023-01-20T21:49:58Z

Should give you what you want without all the seds. Please try it out.

We were looking at this together while writing a unit test. The problem is that we have used this syntax ${blah} to represent substitution by the controller. In other words, using shell substitution would be extremely unwieldy (impossible?) here because we've overloaded that syntax with our own templating syntax.

mjsterckx · 2023-01-20T22:28:31Z

^ Added the unit tests for the remaining sample files.

mjsterckx · 2023-01-23T19:30:57Z

^ Replaced the many sed commands with a single envsubst call.

bottlerocket/samples/RUNBOOK.md

bottlerocket/samples/eks/ecs-migration-test.yaml

mjsterckx · 2023-01-23T20:41:56Z

^ Changed envsubst to eval

stmcginnis reviewed Jan 9, 2023

View reviewed changes

samples/ecs-test.yaml Outdated Show resolved Hide resolved

mjsterckx force-pushed the sample-test-configs branch 8 times, most recently from 3ed2fdb to 865e5a0 Compare January 13, 2023 19:56

mjsterckx force-pushed the sample-test-configs branch 8 times, most recently from 5fd97a7 to d0c37dc Compare January 18, 2023 22:58

mjsterckx marked this pull request as ready for review January 18, 2023 23:06

mjsterckx requested review from stmcginnis, ecpullen, etungsten and webern January 18, 2023 23:06

ecpullen suggested changes Jan 19, 2023

View reviewed changes

mjsterckx force-pushed the sample-test-configs branch 3 times, most recently from 800b54f to 3d4fa18 Compare January 19, 2023 20:26

ecpullen reviewed Jan 19, 2023

View reviewed changes

mjsterckx force-pushed the sample-test-configs branch 3 times, most recently from a12ddd3 to 541b478 Compare January 20, 2023 17:55

ecpullen approved these changes Jan 20, 2023

View reviewed changes

webern reviewed Jan 20, 2023

View reviewed changes

etungsten reviewed Jan 20, 2023

View reviewed changes

webern force-pushed the sample-test-configs branch from 541b478 to 2a5e6d0 Compare January 20, 2023 21:42

webern force-pushed the sample-test-configs branch from 2a5e6d0 to 120970a Compare January 20, 2023 21:47

mjsterckx force-pushed the sample-test-configs branch from 120970a to 15b6604 Compare January 20, 2023 22:26

mjsterckx force-pushed the sample-test-configs branch from 15b6604 to 9675e37 Compare January 23, 2023 19:29

mjsterckx force-pushed the sample-test-configs branch from 9675e37 to 7cc4897 Compare January 23, 2023 19:44

webern approved these changes Jan 23, 2023

View reviewed changes

bottlerocket/samples/RUNBOOK.md Outdated Show resolved Hide resolved

bottlerocket/samples/eks/ecs-migration-test.yaml Outdated Show resolved Hide resolved

added sample test config files

c8d25b6

mjsterckx force-pushed the sample-test-configs branch from 7cc4897 to c8d25b6 Compare January 23, 2023 20:40

etungsten approved these changes Jan 23, 2023

View reviewed changes

mjsterckx merged commit a61ad35 into bottlerocket-os:develop Jan 23, 2023

mjsterckx deleted the sample-test-configs branch January 23, 2023 21:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added sample test config files #740

added sample test config files #740

mjsterckx commented Jan 6, 2023 •

edited

Loading

ecpullen left a comment

ecpullen left a comment

ecpullen Jan 19, 2023

mjsterckx Jan 19, 2023

ecpullen Jan 19, 2023

ecpullen left a comment

webern left a comment

ecpullen commented Jan 20, 2023

webern commented Jan 20, 2023

ecpullen commented Jan 20, 2023 •

edited

Loading

webern commented Jan 20, 2023

ecpullen commented Jan 20, 2023

etungsten left a comment •

edited

Loading

webern commented Jan 20, 2023

webern commented Jan 20, 2023

webern commented Jan 20, 2023

mjsterckx commented Jan 20, 2023

mjsterckx commented Jan 23, 2023

mjsterckx commented Jan 23, 2023

		export UPGRADE_VERSION="v1.11.1"
		export STARTING_VERSION="v1.11.0"

added sample test config files #740

added sample test config files #740

Conversation

mjsterckx commented Jan 6, 2023 • edited Loading

ecpullen left a comment

Choose a reason for hiding this comment

ecpullen left a comment

Choose a reason for hiding this comment

ecpullen Jan 19, 2023

Choose a reason for hiding this comment

mjsterckx Jan 19, 2023

Choose a reason for hiding this comment

ecpullen Jan 19, 2023

Choose a reason for hiding this comment

ecpullen left a comment

Choose a reason for hiding this comment

webern left a comment

Choose a reason for hiding this comment

ecpullen commented Jan 20, 2023

webern commented Jan 20, 2023

ecpullen commented Jan 20, 2023 • edited Loading

webern commented Jan 20, 2023

ecpullen commented Jan 20, 2023

etungsten left a comment • edited Loading

Choose a reason for hiding this comment

webern commented Jan 20, 2023

webern commented Jan 20, 2023

webern commented Jan 20, 2023

mjsterckx commented Jan 20, 2023

mjsterckx commented Jan 23, 2023

mjsterckx commented Jan 23, 2023

mjsterckx commented Jan 6, 2023 •

edited

Loading

ecpullen commented Jan 20, 2023 •

edited

Loading

etungsten left a comment •

edited

Loading