Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: run cucumber scenario as go subtest #1244

Merged
merged 2 commits into from Aug 22, 2022

Conversation

lburgazzoli
Copy link
Contributor

@lburgazzoli lburgazzoli commented Aug 12, 2022

Description

This PR is aimed to make cucumber based tests better integrated with the go test framework so each scenario is executed as if it were a standard subtest.

As example, by running the test with go test -p 1 -ldflags -s -v ./internal/connector/test/integration/..., the final test summary would be like:

15 scenarios (15 passed)
232 steps (232 passed)
58.412982833s
--- PASS: TestFeatures (0.00s)
    --- PASS: TestFeatures/connector_cluster_api (8.54s)
        --- PASS: TestFeatures/connector_cluster_api/Greg_creates_lists_and_deletes_a_connector_addon_cluster (8.54s)
    --- PASS: TestFeatures/connector_agent_api (29.95s)
        --- PASS: TestFeatures/connector_agent_api/Deleting_namespaces_with_undeleted_stuck_connectors_get_reconciled (10.88s)
        --- PASS: TestFeatures/connector_agent_api/connector_cluster_is_created_and_agent_processes_assigned_a_deployment. (15.67s)
        --- PASS: TestFeatures/connector_agent_api/Bobby_can_stop_and_start_an_existing_connector (22.33s)
        --- PASS: TestFeatures/connector_agent_api/Admin_API_can_delete_dangling_connector_deployments (29.94s)
    --- PASS: TestFeatures/connector_admin_api (35.11s)
        --- PASS: TestFeatures/connector_admin_api/Ricky_can_lists_all_connector_clusters,_others_can't (26.52s)
        --- PASS: TestFeatures/connector_admin_api/Ricky_lists_connector_types (35.11s)
    --- PASS: TestFeatures/connector_multitenancy_api (38.34s)
        --- PASS: TestFeatures/connector_multitenancy_api/Gru_tries_to_sql_inject_namespaces_listing (5.03s)
        --- PASS: TestFeatures/connector_multitenancy_api/Use_Admin_API_to_create_namespaces_for_end_users (9.56s)
        --- PASS: TestFeatures/connector_multitenancy_api/Use_Admin_API_to_create_namespace_for_organization_13640231. (9.56s)
        --- PASS: TestFeatures/connector_multitenancy_api/Create_namespaces_in_cluster_for_organization_13640230 (9.65s)
        --- PASS: TestFeatures/connector_multitenancy_api/Create_eval_namespace (13.70s)
        --- PASS: TestFeatures/connector_multitenancy_api/Use_Admin_API_to_create_namespaces_for_end_users#02 (30.85s)
        --- PASS: TestFeatures/connector_multitenancy_api/Use_Admin_API_to_create_namespaces_for_end_users#01 (30.85s)
        --- PASS: TestFeatures/connector_multitenancy_api/Create_eval_namespace#02 (34.99s)
        --- PASS: TestFeatures/connector_multitenancy_api/Create_eval_namespace#01 (34.99s)
        --- PASS: TestFeatures/connector_multitenancy_api/Cannot_create_eval_namespace_with_no_ready_eval_clusters (38.33s)
    --- PASS: TestFeatures/connector_api (58.43s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_sql_inject_connector_type_listing (26.52s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_create_a_connector_with_an_invalid_configuration_spec (26.58s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_sink_connector_types (26.60s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_create_a_connector_with_an_invalid_namespace_id (26.62s)
        --- PASS: TestFeatures/connector_api/check_that_the_connector_types_have_been_reconciled (26.63s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_connector_types_on_beta_channel,_ordered_by_version (26.63s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_connector_types_using_name_and_ilike (26.63s)
        --- PASS: TestFeatures/connector_api/Gary_uses_paging_to_list_connector_types (26.64s)
        --- PASS: TestFeatures/connector_api/Tommy_can_delete_a_connector_that_was_not_deployed,_but_is_waiting_to_be_deleted (27.75s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_sql_inject_connector_listing (8.48s)
        --- PASS: TestFeatures/connector_api/Gary_can_inspect_errors_codes (8.59s)
        --- PASS: TestFeatures/connector_api/Gary_can_discover_the_API_endpoints (8.51s)
        --- PASS: TestFeatures/connector_api/Gary_lists_all_connector_types (8.50s)
        --- PASS: TestFeatures/connector_api/Jim_creates_a_connector_but_later_that_connector_type_is_removed_from_the_system.__He_should (48.34s)
        --- PASS: TestFeatures/connector_api/Gary_creates_lists_and_deletes_a_connector_verifying_that_Evil_Bob_can't_access_Garys_Connectors (31.83s)

Because we have proper subtests, we can also specify what test to run, as example, by adding -run "TestFeatures/connector_api"to the test command, we will have the following report:

15 scenarios (15 passed)
232 steps (232 passed)
26.444465916s

Randomized with seed: 1660300031088685000
--- PASS: TestFeatures (0.00s)
    --- PASS: TestFeatures/connector_api (26.45s)
        --- PASS: TestFeatures/connector_api/Gary_can_discover_the_API_endpoints (5.01s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_sql_inject_connector_listing (5.04s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_sql_inject_connector_type_listing (5.06s)
        --- PASS: TestFeatures/connector_api/Gary_lists_all_connector_types (0.08s)
        --- PASS: TestFeatures/connector_api/Gary_uses_paging_to_list_connector_types (5.13s)
        --- PASS: TestFeatures/connector_api/check_that_the_connector_types_have_been_reconciled (5.13s)
        --- PASS: TestFeatures/connector_api/Gary_can_inspect_errors_codes (0.01s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_connector_types_on_beta_channel,_ordered_by_version (5.14s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_create_a_connector_with_an_invalid_namespace_id (0.09s)
        --- PASS: TestFeatures/connector_api/Gary_tries_to_create_a_connector_with_an_invalid_configuration_spec (0.06s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_sink_connector_types (0.12s)
        --- PASS: TestFeatures/connector_api/Gary_searches_for_connector_types_using_name_and_ilike (5.16s)
        --- PASS: TestFeatures/connector_api/Tommy_can_delete_a_connector_that_was_not_deployed,_but_is_waiting_to_be_deleted (6.26s)
        --- PASS: TestFeatures/connector_api/Jim_creates_a_connector_but_later_that_connector_type_is_removed_from_the_system.__He_should (16.30s)
        --- PASS: TestFeatures/connector_api/Gary_creates_lists_and_deletes_a_connector_verifying_that_Evil_Bob_can't_access_Garys_Connectors (26.44s)

This works also when running in GoLand, as example, with the following set-up:

image

Only the Gary_can_discover_the_API_endpointsstep, part of the connector-api feature, will be executed

Notes:

  1. as each feature is executed in a dedicated test suite, we may have a chance to remove some explicit locking
  2. debugging individual feature/steps in GoLand is still to be figured out
  3. running with make test/integration/connectors works as well but has a slightly different output

Checklist (Definition of Done)

  • All acceptance criteria specified in JIRA have been completed
  • Unit and integration tests added that prove the fix is effective or the feature works (tested against emulated and non-emulated OCM environment)
  • Documentation added for the feature
  • CI and all relevant tests are passing
  • Code Review completed
  • Verified independently by reviewer
  • All PR comments are resolved either by addressing them or creating follow up tasks
  • Required metrics/dashboards/alerts have been added (or PR created).
  • Required Standard Operating Procedure (SOP) is added.
  • JIRA has been created for changes required on the client side

@codecov
Copy link

codecov bot commented Aug 12, 2022

Codecov Report

Merging #1244 (5fb9222) into main (82a4f42) will decrease coverage by 0.04%.
The diff coverage is 25.00%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1244      +/-   ##
==========================================
- Coverage   83.32%   83.28%   -0.05%     
==========================================
  Files         135      135              
  Lines       11957    11963       +6     
==========================================
  Hits         9963     9963              
- Misses       1633     1639       +6     
  Partials      361      361              
Flag Coverage Δ
unittests 83.28% <25.00%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/db/context.go 0.00% <0.00%> (ø)
pkg/shared/util.go 71.42% <0.00%> (-28.58%) ⬇️
pkg/shared/config.go 57.89% <33.33%> (-1.57%) ⬇️
pkg/client/redhatsso/client.go 89.86% <100.00%> (ø)
pkg/services/sso/keycloak_service.go 79.08% <100.00%> (ø)

@machi1990 machi1990 merged commit 6dbcbff into bf2fc6cc711aee1a0c2a:main Aug 22, 2022
@lburgazzoli lburgazzoli deleted the godog-subtests branch February 9, 2023 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants