Create a BDD Test Automation Framework with ginkgo #552

alejovicu · 2018-10-26T20:39:01Z

Create a BDD test automation framework using ginkgo and gomega to test the asset generation before launch the cluster installation (https://jira.coreos.com/browse/CORS-923)
Add a test suite for openshift-install create install-configfor libvirt platform (https://jira.coreos.com/browse/CORS-858)

wking · 2018-10-26T21:53:24Z

Do we need this? For the libvirt tests, I was hoping to run the usual origin e2e suite. And if you're only concerned about exercising create install-config, a simpler approach would be to add a cmd/openshift-install/create_test.go for our existing Go unit job.

alejovicu · 2018-10-30T20:58:32Z

Hit @wking
This framework should cover the entire application.
At this moment is just the setup of the framework and the test suite for "create install-config" command. but It will increase adding the test suites for "create ignition-configs, create manifests, create cluster, destroy-cluster...". When the cluster is deployed, the usual origin e2e suite will be executed to certify the cluster health.

Now, in the test workflow, the idea is to run the config tests before the cluster health tests because some issues can be found without the cluster deploy saving some time.

crawford

Overall, this looks good to me. Just a few minor requests.

tests/bdd-smoke/Gopkg.toml

tests/bdd-smoke/suites/config/libvirt/install_config_test.go

tests/bdd-smoke/utils/assertion.go

abhinavdahiya · 2019-02-01T21:25:26Z

@alejovicu

can we quash this into 3 commits, vendor, scripts/docs, bdd code.
also any reason why this has a separate vendor?

alejovicu · 2019-02-01T22:37:44Z

Hi @abhinavdahiya,
The split in 3 commits is done.
I created this as a separate vendor because this framework uses dependencies that only apply for it, like gomega that would be useless by other modules in the installer.

hack/run-bdd-suite.sh

abhinavdahiya · 2019-02-05T00:17:45Z

This looks like a good first step, Thanks a lot @alejovicu :yay:
Lets iterate on this.

/lgtm

openshift-ci-robot · 2019-02-05T00:17:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, alejovicu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abhinavdahiya · 2019-02-05T05:06:07Z

/refresh

wking

I left a few nits inline. Nothing that needs to block a merge, but probably things that are worth fixing post-merge.

wking · 2019-02-05T05:59:32Z

tests/bdd-smoke/suites/config/libvirt/install_config_test.go

+		Specify("Then I expect to see the install-config.yaml file in the path ./scenario1/", func() {
+			Ω(openshiftInstallerPath + "/scenario1/install-config.yaml").Should(BeAnExistingFile())
+		})
+		Specify("And I expect install-config.yaml file contains the info specified in the environment variables", func() {


nit: these aren't environment variables any more (and also in several other Specify strings in this commit).

wking · 2019-02-05T06:01:21Z

tests/bdd-smoke/utils/command_runner.go

+	cmd.Stderr = &errb
+	err := cmd.Run()
+	if err != nil {
+		fmt.Println("Error: failed running the command \"" + command + "\" with the arguments:")


nit: this could be more robust and compact with:

fmt.Printf("Error: failed running the command %q with the arguments:", command)

although printing straight to the terminal feels strange anyway. Does that get captured somewhere? Should we be logging these instead, or attaching it to the error before passing it to log.Fatal?

wking · 2019-02-05T06:05:54Z

tests/bdd-smoke/utils/libvirt_inst_state_file_validator.go

+	var installState installStateConf
+	json.Unmarshal(byteValue, &installState)
+
+	AssertStringContains(installState.InstallConfig.Config.BaseDomain,


This and later parts of ValidateInstallStateConfig feel like a hand-crafted special case of reflect.DeepEqual. Is it doing something else?

wking · 2019-02-05T06:07:06Z

tests/bdd-smoke/utils/libvirt_inst_state_file_validator.go

+		expectedData.ClusterName,
+		"*installconfig.clusterName.ClusterName not found in "+fileName)
+
+	// OPENSHIFT_INSTALL_PLATFORM is implicitly validated by next checks. If its not Libvirt platform, the Name and URI will not be accessible


nit: This references the environment variable dropped in 6be4c25 (#861).

openshift-bot · 2019-02-05T06:34:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2019-02-05T06:54:15Z

@alejovicu: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-openstack	`1b092fe`	link	`/test e2e-openstack`
ci/prow/e2e-libvirt	`1b092fe`	link	`/test e2e-libvirt`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

For regions like eu-west-3 (Paris) that don't support m4. The only instance type available in all AWS regions today is i3. We prefer m4 because of the higher EBS attach limit. But for regions that don't support m4 we use m5. This also adds a CI job which checks the AWS pricing API to keep our region map up to date. The eu-north-1 region is new as of 2018-12-12 [1], and it show up in: $ aws ec2 describe-regions --query 'Regions[].RegionName' --output json | jq -r 'sort[]' ... eu-central-1 eu-north-1 eu-west-1 ... The GovCloud entries didn't show up in that query (maybe they are only listed for accounts that have access?), but they are listed on [2]. The new test is in t/aws to keep it out of our default unit tests. Maintainers can run it manually to verify that the default fallbacks we hard-code are still appropriate for the current AWS offerings. It's not in tests/aws, because 'dep ensure' seems to ignore tests. I think it's fine to have both t and tests in parallel, with the former holding tests who share the project vendor/ and the latter holding tests which have a distinct vendor/. bdd-smoke is in tests with its own vendor/ intentionally to keep gomega and other test-specific libraries out of the project vendor/ [3]. But for these tests we're using Go's built-in test framework, so it's less maintenance to layer our minimal additional dependencies into the project vendor/. [1]: https://aws.amazon.com/blogs/aws/now-open-aws-europe-stockholm-region/ [2]: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ [3]: openshift#552 (comment)

From Trevor: For regions like eu-west-3 (Paris) that don't support m4. The only instance type available in all AWS regions today is i3. We prefer m4 because of the higher EBS attach limit. But for regions that don't support m4 we use m5. This also adds a CI job which checks the AWS pricing API to keep our region map up to date. The eu-north-1 region is new as of 2018-12-12 [1], and it show up in: $ aws ec2 describe-regions --query 'Regions[].RegionName' --output json | jq -r 'sort[]' ... eu-central-1 eu-north-1 eu-west-1 ... The GovCloud entries didn't show up in that query (maybe they are only listed for accounts that have access?), but they are listed on [2]. The new test is in t/aws to keep it out of our default unit tests. Maintainers can run it manually to verify that the default fallbacks we hard-code are still appropriate for the current AWS offerings. It's not in tests/aws, because 'dep ensure' seems to ignore tests. I think it's fine to have both t and tests in parallel, with the former holding tests who share the project vendor/ and the latter holding tests which have a distinct vendor/. bdd-smoke is in tests with its own vendor/ intentionally to keep gomega and other test-specific libraries out of the project vendor/ [3]. But for these tests we're using Go's built-in test framework, so it's less maintenance to layer our minimal additional dependencies into the project vendor/. [1]: https://aws.amazon.com/blogs/aws/now-open-aws-europe-stockholm-region/ [2]: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ [3]: openshift#552 (comment)

From Trevor: For regions like eu-west-3 (Paris) that don't support m4. The only instance type available in all AWS regions today is i3. We prefer m4 because of the higher EBS attach limit. But for regions that don't support m4 we use m5. This also adds a CI job which checks the AWS pricing API to keep our region map up to date. The eu-north-1 region is new as of 2018-12-12 [1], and it show up in: $ aws ec2 describe-regions --query 'Regions[].RegionName' --output json | jq -r 'sort[]' ... eu-central-1 eu-north-1 eu-west-1 ... The GovCloud entries didn't show up in that query (maybe they are only listed for accounts that have access?), but they are listed on [2]. The new test is in platformtests/aws to keep it out of our default unit tests. Maintainers can run it manually to verify that the default fallbacks we hard-code are still appropriate for the current AWS offerings. It's not in tests/aws, because 'dep ensure' seems to ignore tests. I think it's fine to have both platformtests and tests in parallel, with the former holding tests who share the project vendor/ and the latter holding tests which have a distinct vendor/. bdd-smoke is in tests with its own vendor/ intentionally to keep gomega and other test-specific libraries out of the project vendor/ [3]. But for these tests we're using Go's built-in test framework, so it's less maintenance to layer our minimal additional dependencies into the project vendor/. [1]: https://aws.amazon.com/blogs/aws/now-open-aws-europe-stockholm-region/ [2]: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ [3]: openshift#552 (comment)

openshift-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 26, 2018

openshift-ci-robot requested review from smarterclayton and steveej October 26, 2018 20:39

alejovicu changed the title ~~Add test suite for create install config command~~ [WIP] Add test suite for create install config command Nov 13, 2018

alejovicu changed the title ~~[WIP] Add test suite for create install config command~~ Create a BDD test automation framework using ginkgo Dec 6, 2018

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2018

alejovicu changed the title ~~Create a BDD test automation framework using ginkgo~~ Create a BDD Test Automation Framework with ginkgo Dec 6, 2018

crawford reviewed Dec 7, 2018

View reviewed changes

tests/bdd-smoke/Gopkg.toml Show resolved Hide resolved

tests/bdd-smoke/suites/config/libvirt/install_config_test.go Outdated Show resolved Hide resolved

tests/bdd-smoke/utils/assertion.go Outdated Show resolved Hide resolved

tests/bdd-smoke/utils/assertion.go Outdated Show resolved Hide resolved

abhinavdahiya reviewed Feb 1, 2019

View reviewed changes

hack/run-bdd-suite.sh Show resolved Hide resolved

abhinavdahiya reviewed Feb 1, 2019

View reviewed changes

hack/run-bdd-suite.sh Outdated Show resolved Hide resolved

Alejandro Rosero added 3 commits February 4, 2019 12:48

BDD Suite: Add dependencies for BDD ginkgo framework

59e269c

BDD Suite: Add scripts and documentation

b4a3f96

BDD Suite: Add BDD framework code

1b092fe

openshift-ci-robot assigned abhinavdahiya Feb 5, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 5, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 5, 2019

wking reviewed Feb 5, 2019

View reviewed changes

openshift-merge-robot merged commit c5ffe9b into openshift:master Feb 5, 2019

alejovicu deleted the libvirt-tests branch February 6, 2019 16:39

wking mentioned this pull request Feb 8, 2019

pkg/asset/machines: Convert Master to a WriteableAsset #1211

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a BDD Test Automation Framework with ginkgo #552

Create a BDD Test Automation Framework with ginkgo #552

alejovicu commented Oct 26, 2018 •

edited

wking commented Oct 26, 2018

alejovicu commented Oct 30, 2018

crawford left a comment

abhinavdahiya commented Feb 1, 2019

alejovicu commented Feb 1, 2019

abhinavdahiya commented Feb 5, 2019

openshift-ci-robot commented Feb 5, 2019

abhinavdahiya commented Feb 5, 2019

wking left a comment

wking Feb 5, 2019

wking Feb 5, 2019

wking Feb 5, 2019

wking Feb 5, 2019

openshift-bot commented Feb 5, 2019

openshift-ci-robot commented Feb 5, 2019

Create a BDD Test Automation Framework with ginkgo #552

Create a BDD Test Automation Framework with ginkgo #552

Conversation

alejovicu commented Oct 26, 2018 • edited

wking commented Oct 26, 2018

alejovicu commented Oct 30, 2018

crawford left a comment

Choose a reason for hiding this comment

abhinavdahiya commented Feb 1, 2019

alejovicu commented Feb 1, 2019

abhinavdahiya commented Feb 5, 2019

openshift-ci-robot commented Feb 5, 2019

abhinavdahiya commented Feb 5, 2019

wking left a comment

Choose a reason for hiding this comment

wking Feb 5, 2019

Choose a reason for hiding this comment

wking Feb 5, 2019

Choose a reason for hiding this comment

wking Feb 5, 2019

Choose a reason for hiding this comment

wking Feb 5, 2019

Choose a reason for hiding this comment

openshift-bot commented Feb 5, 2019

openshift-ci-robot commented Feb 5, 2019

alejovicu commented Oct 26, 2018 •

edited