Skip to content

Code quality guidelines testing and linting

Jump to bottom
Corey Pyle edited this page Jul 6, 2023 · 9 revisions

Overview: We expect integration tests for single operation examples and scenario examples, and we expect unit tests for cross-service examples. All submitted code is automatically linted, and the assigned reviewer is responsible for confirming sufficient test coverage.

This document explains the following:

Terminology in this document

  • Mocking - Synthesizes an input to test that the example runs properly. Most languages have mocking frameworks.
  • Unit test - Tests individual modules of an application in isolation (without any interaction with dependencies). Unit tests are a basic, affordable way to confirm that the code is operating correctly. Unit tests require mocking, which removes the dependencies that your system has and replaces them with an implementation that you control. (For example, your code might have external dependencies like system calls or database access.)
  • Integration test - Tests modules of code to verify how systems interact with each other. Integration tests can confirm that endpoints are live and are being used correctly. Integration tests are more expensive and more thorough than unit tests. For integration tests, only test what you can control or edit. Mock anything that you can't control.
  • Linting - Automatically checks your source code for programmatic and stylistic errors. For information about recommended linting tools for your language, see Linters run on check-in.
  • Unit testing framework - A software tool that helps you write and run unit tests. For information about the recommended unit testing framework for your language, see Recommended unit test frameworks.

HOW we test code examples

Single operation examples

Generally, we don't recommend testing single-operation examples. Exceptions apply if you're processing some data, or if the single operation example is rare. For example, if the single operation is covered in an MVP scenario, the integration test for the scenario covers it. One exception is that we recommend having at least one single-operation integration test, using a unit test framework, to demonstrate test creation for developers.

Single operation API calls that aren't covered by scenarios should still have basic tests to prove that they work. These examples tend to fall into one of the following categories: trivially basic calls and obscure or complex calls. For complex calls, the example is useful for developers even if we haven’t found a scenario to use it in. For basic calls, creating the examples takes minimal effort, so there's little reason not to include them.

Note With single-operation example integration tests, make sure that any calls that create resources also have matching teardown calls to accompany the tests.

Scenario examples

We recommend using integration tests that test the relationship between AWS services in your example. Use mocking for whatever you can't edit or control.

Integration tests rely on external code and test that any relationships to dependencies are set up correctly. Because of this, integration tests should be written and run with caution, because they might incur costs. This is especially true if an integration test fails and resources are kept running. The running resources might incur charges. Integration tests confirm that your code properly integrates with specific dependencies. It's usually beneficial to test as few dependencies at a time as possible. This reduces the scope of each test and reduces the chance that one dependency will give a false negative result because of the failure of a separate process.

Example 1 This PHP S3 test example runs the scenario as if it were a user. Because the scenario was designed with testing in mind and it runs automatically (as long as credentials are set up correctly), this is the simplest way to create a scenario test.

Example 2 This Python S3 test example is similar to the preceding example. However, it also simulates user input. This way, you can have a more interactive scenario while keeping the automation concise.

Example 3 This Rust S3 test example runs against a service library instead of running the scenario example raw. All of the actions run in the scenario are tested independently here. This gives more granularity and insight into where failures occur. Instead of getting an error that a scenario failed to run, specific actions from the service library will fail or succeed. This reduces the cognitive load when investigating failures. This type of integration test is suitable for use on larger segments of code that have well-defined actions.

Cross-service examples

It's important to perform unit testing of elements of cross-service examples to verify that your business logic is correct. This sort of test isolates your code from the rest of the world, creating a bubble in which to run your tested code.

The following are examples of places where unit tests are important:

  • A function that counts the number of words in a report.
  • A function that finds words that contain the given letters (like a Scrabble searcher).
  • Text formatting.
  • Parsing the EXIF data out of a photo (Note: Although most people use an external library, it’s still important to verify that you’re accessing the EXIF data correctly after you have it.)

Most of these examples fall into the category of "sorting and searching data."

A less obvious form of this is parsing a structure. In the wordfreq example in gov2, there is a small helper that handles the kinds of messages that an Amazon Simple Queue Service (Amazon SQS) queue might have.

Example 1 This Go unit test example tests this Go example, which helps fill in several structures. Each structure is a possible form of the kind of Amazon SQS message that could come to the app. The unit test confirms that, for given known good samples of messages, the messages are identified and broken up appropriately.

WHY we test code examples

  • To demonstrate that our code works and can be safely included in documentation and recommended to developers.
  • To reduce instances of publishing code that doesn’t work.
  • To make sure that we notice any examples that might stop working.
  • To demonstrate to developers how they can test examples that use AWS services.

WHERE we test code examples

  • We add tests to our code examples before we publish them. There are different testing requirements for each category of examples (single operation, scenario, and cross-service).
  • On GitHub PR check-in, we run language-specific linters against all code to confirm that it compiles and passes against newly submitted code.
  • (Stretch goal) On GitHub PR check-in, we run all unit tests for all languages.
  • (Stretch goal) We schedule and run all tests, including integration tests, on a regular cadence. (For example, daily or weekly. Cadence to be determined.)

WHOSE code examples are tested

Code without tests is not merged to this repo. However, if you cannot provide tests, the assigned reviewer will work with you to resolve the issue. At a minimum, all submitted code should be linted.

WHEN we test code examples

Currently, the assigned reviewer is responsible for checking that all code has tested sufficiently and that it runs as expected.

Eventually, our goal is to have all unit tests, new integration tests, and linters on new code run whenever we create a new PR into main. This will help us confirm that the code adheres to our standards. In addition, we have another goal of having all tests run on a weekly basis to make sure that the code is up to date with underlying dependencies.

Linters run on check-in

Automatically checks your source code for programmatic and stylistic errors. The following lint tools are run against code submitted to the repo.

Language Lint tool License used
C++ clang-tidy
C#(.Net) dotnet-format / clang-format
Go golangci-lint
Java checkstyle
JavaScript esLint
Kotlin ktlint
PHP PHP built-in linter / PHP CodeSniffer
Python pylint
Ruby RSpec
Rust cargo clippy/ cargo fmt

Recommended unit test frameworks

Language Unit test tool License used
C++ CTest (included with CMake)
C# xUnit, Moq Apache License 2.0
DotNet
Go go test
Java JUnit5
JavaScript Jest
Kotlin JUnit5
PHP PHPUnit
PowerShell Pester
Python pytest MIT License
Ruby RSpec
Rust cargo test
Clone this wiki locally