Skip to content

Commit

Permalink
A couple more post-Replatforming doc updates.
Browse files Browse the repository at this point in the history
Also use short references in links to Git commits. (Makes editing
a little easier.)
  • Loading branch information
sengi committed Jun 29, 2023
1 parent a163d29 commit b608afb
Show file tree
Hide file tree
Showing 2 changed files with 30 additions and 24 deletions.
18 changes: 12 additions & 6 deletions docs/tagging.md
Expand Up @@ -4,17 +4,23 @@

## `@app-<app_name>`

The Smokey job in Jenkins [has an option to filter scenarios by application](https://github.com/alphagov/govuk-puppet/blob/7f4b1684471daf09cff72c1372db88b1ed3fd1dc/modules/govuk_jenkins/templates/jobs/smokey.yaml.erb#L34), so that we can avoid running the entire test suite when only some scenarios are relevant to a change. To associate tests with an app, you need to tag the relevant scenarios (or entire feature) with `@app-<app_name>`.
The Smokey job in Jenkins [has an option to filter scenarios by application](https://github.com/alphagov/govuk-puppet/blob/7f4b168/modules/govuk_jenkins/templates/jobs/smokey.yaml.erb#L34), so that we can avoid running the entire test suite when only some scenarios are relevant to a change. To associate tests with an app, you need to tag the relevant scenarios (or entire feature) with `@app-<app_name>`.

**Important: make sure `<app_name>` matches against [the list of apps for the Deploy_App job](https://github.com/alphagov/govuk-puppet/blob/7f4b1684471daf09cff72c1372db88b1ed3fd1dc/hieradata_aws/common.yaml#L145)**. This is so the filtering will work with [the Continuous Deployment pipeline](https://github.com/alphagov/smokey/pull/675).
> **Ensure `<app_name>` matches the name of the app's Argo CD application
> resource under `govukApplications` in
> [`values-<environment>.yaml`](https://github.com/alphagov/govuk-helm-charts/blob/cda3239/charts/app-config/values-integration.yaml#L45).**
> If these don't match then the scenario will not block promotion of a release
> of that app to staging/production.
## `@not<environment>`

All features should run in **all** environments. Sometimes this isn't possible e.g.
All features should run in **all** environments. Sometimes this isn't possible, for example where:

- [The actual functionality only exists in Production](https://github.com/alphagov/smokey/blob/83fa09aa2aacec3053d58c52a2cb3af1ca27ba2b/features/mirror.feature). Tag the feature or scenario with `@notintegration` and `@notstaging` so it doesn't run in these environments i.e. its only runs in Production.

- [The functionality is being replatformed](https://github.com/alphagov/smokey/blob/5caf9635f4c4601df69c39075f5438fc3d3f3df0/features/info_frontend.feature#L1). Tag the feature or scenario with `@replatforming` or `@notreplatforming` depending on whether you expect it to work in the new platform.
- [The actual functionality only exists in
Production](https://github.com/alphagov/smokey/blob/83fa09a/features/mirror.feature).
Tag the feature or scenario with `@notintegration` and `@notstaging` so it
doesn't run in these environments i.e. its only runs in Production.
- The test creates or modifies content which can't yet be done in production.

## `@pending`

Expand Down
36 changes: 18 additions & 18 deletions docs/writing-tests.md
Expand Up @@ -6,28 +6,28 @@ This repo contains tests that check critical areas of GOV.UK are working as expe

- **Covers vital site-wide configuration**. The test checks that the foundations of GOV.UK exist, with some indication they work as expected:

- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/mirror.feature#L11) - covers fallback mirrors exist
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/gov_uk_redirect.feature#L13) - covers Fastly CDN configuration
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/gov_uk.feature#L9) - covers an edge case of limited value
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/gov_uk.feature#L27) - covers a negative or "unhappy" outcome
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/mirror.feature#L11) - covers fallback mirrors exist
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/gov_uk_redirect.feature#L13) - covers Fastly CDN configuration
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522/features/gov_uk.feature#L9) - covers an edge case of limited value
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522/features/gov_uk.feature#L27) - covers a negative or "unhappy" outcome

- **Targets app-to-app data transfer**. The test checks that data can flow between multiple apps to provide an end-to-end feature of GOV.UK:

- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/service_manual.feature#L7) - basic test that an app is accessible via Router
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/service_manual.feature#L7) - basic test that an app is accessible via Router
- Implicitly checks the app can talk to the Content Store
- Each app can have one (and only one) of these tests
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/collections.feature#L53) - covers data transfer with Email Alert API
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/router.feature#L8) - doesn’t check content data was transferred
- [Bad example](https://github.com/alphagov/smokey/blob/0d7134e3e99a8c0bd9b4ada13d6ce298bbf03f44/features/benchmarking.feature#L7) - performance testing is out-of-scope
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/collections.feature#L53) - covers data transfer with Email Alert API
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522/features/router.feature#L8) - doesn’t check content data was transferred
- [Bad example](https://github.com/alphagov/smokey/blob/0d7134e/features/benchmarking.feature#L7) - performance testing is out-of-scope

- **Second critical functional check**. A very similar test exists at a lower level but the functionality is so absolutely critical we test it again:

- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/frontend.feature#L18) - undeniably a critical page for GOV.UK ([lower level test exists](https://github.com/alphagov/frontend/blob/a40ba988c58dd882d841a6f1d82d5b894939fdc5/test/integration/help_test.rb#L18))
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/frontend.feature#L59) - one of the most critical features of GOV.UK ([lower level test exists](https://github.com/alphagov/router/blob/96d5912cec4c8052c2f9f71d257f2add2787b65d/integration_tests/redirect_test.go?L23#L12))
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/collections.feature#L29) - checks an arbitrary, non-critical page ([lower level test exists](https://github.com/alphagov/collections/blob/cc94525a05b2651fee621b588e530aca8e1f77e4/features/viewing_browse.feature))
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/finder_frontend.feature#L45) - missing a lower level test in the app itself ([example test to copy](https://github.com/alphagov/finder-frontend/blob/d70a01903e2813719e7d5adbf79fbcb1d49335fb/spec/controllers/email_alert_subscriptions_controller_spec.rb#L363))
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/frontend.feature#L18) - undeniably a critical page for GOV.UK ([lower level test exists](https://github.com/alphagov/frontend/blob/a40ba98/test/integration/help_test.rb#L18))
- [Good example](https://github.com/alphagov/smokey/blob/aaa0522/features/frontend.feature#L59) - one of the most critical features of GOV.UK ([lower level test exists](https://github.com/alphagov/router/blob/96d5912/integration_tests/redirect_test.go?L23#L12))
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522/features/collections.feature#L29) - checks an arbitrary, non-critical page ([lower level test exists](https://github.com/alphagov/collections/blob/cc94525/features/viewing_browse.feature))
- [Bad example](https://github.com/alphagov/smokey/blob/aaa0522/features/finder_frontend.feature#L45) - missing a lower level test in the app itself ([example test to copy](https://github.com/alphagov/finder-frontend/blob/d70a019/spec/controllers/email_alert_subscriptions_controller_spec.rb#L363))

Do not duplicate tests. For example, ["Email signup from an organisation home page"](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/collections.feature#L47) duplicates ["Email signup from a taxon page"](https://github.com/alphagov/smokey/blob/aaa0522666d49db116b939276f0e0969357e3ed4/features/collections.feature#L53): both cover data transfer with Email Alert API.
Do not duplicate tests. For example, ["Email signup from an organisation home page"](https://github.com/alphagov/smokey/blob/aaa0522/features/collections.feature#L47) duplicates ["Email signup from a taxon page"](https://github.com/alphagov/smokey/blob/aaa0522/features/collections.feature#L53): both cover data transfer with Email Alert API.

Do not add tests that can be implemented in other ways. We want this to be the tip of the [testing pyramid](https://martinfowler.com/bliki/TestPyramid.html), not a [testing ice cream cone](http://saeedgatson.com/the-software-testing-ice-cream-cone/). Instead of adding a test here, you can:

Expand All @@ -45,17 +45,17 @@ Scenarios should be organised by the "component" they are testing:

- **`apps/<app>.feature`** should contain tests that target a feature of \<app\> e.g.

- "[Email signup from foreign travel advice](https://github.com/alphagov/smokey/blob/a529bae614f28ab5073203a7a7d667c77c65c19d/features/foreign_travel_advice.feature#L15)" is targeting Government Frontend, so should be in `apps/government_frontend.feature`.
- "[Email signup from foreign travel advice](https://github.com/alphagov/smokey/blob/a529bae/features/foreign_travel_advice.feature#L15)" is targeting Government Frontend, so should be in `apps/government_frontend.feature`.

- "[Can log in to collections-publisher](https://github.com/alphagov/smokey/blob/a529bae614f28ab5073203a7a7d667c77c65c19d/features/publishing_tools.feature#L4)" is targeting Collections Publisher, so should be in `apps/collections_publisher.feature`.
- "[Can log in to collections-publisher](https://github.com/alphagov/smokey/blob/a529bae/features/publishing_tools.feature#L4)" is targeting Collections Publisher, so should be in `apps/collections_publisher.feature`.

- **`<layer>.feature`** should contain tests that target an infrastructure layer e.g.

- "[Check redirect from bare domain to www.gov.uk is working for HTTP](https://github.com/alphagov/smokey/blob/a529bae614f28ab5073203a7a7d667c77c65c19d/features/gov_uk_redirect.feature#L3)" is [targeting site-wide behaviour of our CDN](https://github.com/alphagov/govuk-cdn-config/blob/8ca5780ec513c0c17f64304c26bbd3de18c5138d/vcl_templates/tldredirect.vcl.erb#L46), so should be in `cdn.feature`.
- "[Check redirect from bare domain to www.gov.uk is working for HTTP](https://github.com/alphagov/smokey/blob/a529bae/features/gov_uk_redirect.feature#L3)" is [targeting site-wide behaviour of our CDN](https://github.com/alphagov/govuk-cdn-config/blob/8ca5780/vcl_templates/tldredirect.vcl.erb#L46), so should be in `cdn.feature`.

- "[Check visiting a draft page requires a signon session](https://github.com/alphagov/smokey/blob/a529bae614f28ab5073203a7a7d667c77c65c19d/features/draft_environment.feature#L11)" is targeting site-wide behaviour of our (draft) origin, so should be in `origin.feature`.
- "[Check visiting a draft page requires a signon session](https://github.com/alphagov/smokey/blob/a529bae/features/draft_environment.feature#L11)" is targeting site-wide behaviour of our (draft) origin, so should be in `origin.feature`.

Try to use existing step definitions e.g. [`When I visit "/some/path"`](https://github.com/alphagov/smokey/blob/b656e65ef6853d65de3e24a7e884911054c8bf7d/features/step_definitions/smokey_steps.rb#L70-L72). If a more specific step is required, name the `_steps.rb` file after the feature file i.e. `myfeature_steps.rb`.
Try to use existing step definitions e.g. [`When I visit "/some/path"`](https://github.com/alphagov/smokey/blob/b656e65/features/step_definitions/smokey_steps.rb#L70-L72). If a more specific step is required, name the `_steps.rb` file after the feature file i.e. `myfeature_steps.rb`.

### Tests that makes changes to GOV.UK

Expand Down

0 comments on commit b608afb

Please sign in to comment.