From 9929df60f7fa7071abc49f6e83bac6b86b40ecae Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 21:23:10 +0100 Subject: [PATCH 01/12] Update terminology in visual tests part --- recipes/automated-testing.md | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index d4da7ef..6e21b9e 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -241,24 +241,37 @@ instance, but the actual measurements should be performed live) ### Visual Tests -The type of test where test runner navigates to browser page, takes screenshot -and then compares the future screenshots with the reference screenshot. +The type of test where test runner navigates to browser page, takes snapshot and +then compares the snapshots with the reference snapshot. -These types of tests will cover a lot of ground with the least effort and can -easily indicate a change in the app. The downside is that they're not very precise -and the engineer needs to spend some time to determine the cause of the error. +Visual tests allow you to quickly cover large portions of the application, +ensuring that changes in the UI are detected without writing complex test cases. +The downside is that they're requiring engineers to invest time in identifying +the root cause of errors. #### When to use -- When we want to cover broad range of features. -- When we want to increase test coverage with least effort. - When we want to make sure there are no changes in the UI. +- When we want to increase test coverage with least effort. +- During the early stages of a project, while e2e tests are still under +development. #### When **not** to use - To test a specific feature or business logic. - To test a specific user scenario. #### Best practices -- Have deterministic seeds so the UI always renders the same output. +- Ensure the UI consistently renders the same output by controlling randomness +(e.g., setting seeds for random data or controlling API responses). - Add as many pages as possible but keep the tests simple. +- Consider running visual tests at the component level to isolate and detect +issues earlier. +- Define acceptable thresholds for minor visual differences (e.g., pixel +tolerance) to reduce noise while detecting significant regressions. #### Antipatterns +- Avoid creating overly complicated visual tests that try to simulate user +behavior. These are better suited for e2e testing. +- Visual tests should complement, not replace other types of tests like e2e +tests. Over-relying on them can leave functional gaps in coverage. +- Blindly updating snapshots without investigating failures undermines the +purpose of visual testing and risks missing real issues. \ No newline at end of file From 539cfeab51650cc9bfa19a4279ce2fba62a30826 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 21:45:20 +0100 Subject: [PATCH 02/12] Update terminology in performance tests part --- recipes/automated-testing.md | 48 ++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 6e21b9e..1f0db5d 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -206,38 +206,50 @@ cause the team to ignore or bypass the tests and these should be dealt with imme ### Performance Tests -These types of tests will reproduce a typical user scenario and then simulate a -group of concurrent users and then measure the server's response time and overall -performance. - -They are typically used to stress test the infrastructure and measure the throughput -of the application. They can expose bottlenecks and identify endpoints that need +Performance tests replicate typical user scenarios and then scale up to simulate +concurrent users. They measure key performance metrics such as response time, +throughput, error rate, and resource utilization. These tests help uncover +bottlenecks and identify specific endpoints or processes that require optimization. -Performance tests are supposed to be run on actual production environment since -they test the performance of code **and** infrastructure. Keep in mind actual -users when running performance tests. Best approach is to spin up a production -clone and run tests against that environment. +Performance tests are supposed to be run on a production-like environment since +they test the performance of code **and** infrastructure. It's essential to +consider real user behavior when designing and running these tests. The best +practice is to create a clone of the production environment for testing +purposes, avoiding potential disruption to actual users. #### When to use -- To stress test infrastructure. -- To measure how increased traffic affects load speeds and overall app performance. +- To stress test application's infrastructure. +- To evaluate the app’s behavior and performance under increasing traffic. +- To identify and address bottlenecks or resource limitations in the +application. +- To ensure the application can handle anticipated peak traffic or usage +patterns. #### When **not** to use -- To test if the application works according to specs. +- To verify functional requirements or application features. - To test a specific user scenario. #### Best practices -- These tests should mimic actual human user in terms of click frequency and page -navigation. -- There should be multiple tests that test different paths in the system, not a -single performance test. +- Ensure the tests mimic actual user behavior, including realistic click +frequency, page navigation patterns, and input actions. +- Include diverse scenarios that represent different user journeys across the +system, not just a single performance test. +- Use a clone of the production environment to ensure the infrastructure matches +real-world conditions, including hardware, network, and database configurations. +- Schedule performance tests periodically or before major releases to catch +regressions early. +- Record and analyze test outcomes to understand trends over time, identify weak +points, and track improvements. +- Performance testing should not be a one-time task; it should be an ongoing +process integrated into the development lifecycle. #### Antipatterns - Running these tests locally or on an environment that doesn't match production in terms of infrastructure performance. (tests should be developed on a local instance, but the actual measurements should be performed live) - +- Ensure the test data mirrors real-world conditions, including varying user +inputs and dataset sizes. ### Visual Tests From 93ee9c39ed2cc3228ea18bf0a667982763c9414d Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 21:57:11 +0100 Subject: [PATCH 03/12] Update terminology in e2e tests part --- recipes/automated-testing.md | 39 +++++++++++++++++++++++++----------- 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 1f0db5d..82ccfd2 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -174,35 +174,50 @@ performance tests (K6). #### Antipatterns - ### E2E Tests -These tests are executed within a browser environment (Playwright, Selenium, etc.). -The purpose of these tests is to make sure that interacting with the application UI -produces the expected result. +These tests are executed within a browser environment (Playwright, Selenium, +etc.). The purpose of these tests is to make sure that interacting with the +application UI produces the expected result, verifying the application’s +functionality from a user’s perspective. Usually, these tests will cover a large portion of the codebase with least -amount of code. -Because of that, they can be the first tests to be added to existing project that -has no tests or has low test coverage. +amount of code. Because of that, they can be the first tests to be added to +existing project that has no tests or has low test coverage. -These tests should not cover all of the use cases because they are the slowest to -run. If we need to test edge cases, we should try to implement those at a lower -level (integration or unit tests). +These tests should not cover all of the use cases because they are the slowest +to run. If we need to test edge cases, we should try to implement those at a +lower level (integration or unit tests). #### When to use -- Test user interaction with the application UI. +- To validate user interactions and critical workflows in the application UI. +- For testing full system integration across multiple services or components. #### When **not** to use - For data validation. #### Best practices +- Focus on the most important user workflows rather than attempting exhaustive +coverage. +- Each test should be able to run independently, with the environment reset to a +known state before every test. - Performance is key in these tests. We want to run tests as often as possible and good performance will allow that. - Flaky tests should be immediately disabled and refactored. Flaky tests will -cause the team to ignore or bypass the tests and these should be dealt with immediately. +cause the team to ignore or bypass the tests and these should be dealt with +immediately. +- Ensure consistent data states to avoid test failures due to variability in +backend systems or environments. +- Run tests in parallel and isolate them from external dependencies to improve +speed and reliability. +- Automate E2E tests in your CI/CD pipeline to catch regressions early in the +deployment process. #### Antipatterns +- Avoid trying to cover all use cases or edge cases in E2E tests; these are +better suited for unit or integration tests. +- Don’t bypass or disable flaky tests without fixing them, as they undermine the +test suite’s reliability. ### Performance Tests From 90a1baa7210bbad6fee9ab7bfd9e2f8eb9e61047 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 22:06:33 +0100 Subject: [PATCH 04/12] Cleanup --- recipes/automated-testing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 82ccfd2..99ea2ab 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -279,7 +279,7 @@ the root cause of errors. #### When to use - When we want to make sure there are no changes in the UI. - When we want to increase test coverage with least effort. -- During the early stages of a project, while e2e tests are still under +- During the early stages of a project, while E2E tests are still under development. #### When **not** to use @@ -297,8 +297,8 @@ tolerance) to reduce noise while detecting significant regressions. #### Antipatterns - Avoid creating overly complicated visual tests that try to simulate user -behavior. These are better suited for e2e testing. -- Visual tests should complement, not replace other types of tests like e2e +behavior. These are better suited for E2E testing. +- Visual tests should complement, not replace other types of tests like E2E tests. Over-relying on them can leave functional gaps in coverage. - Blindly updating snapshots without investigating failures undermines the purpose of visual testing and risks missing real issues. \ No newline at end of file From 3c8a30c1d5ee4ee8c099fea9c5691151541ca9f4 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 22:12:21 +0100 Subject: [PATCH 05/12] Cleanup --- recipes/automated-testing.md | 120 +++++++++++++++++++++++++---------- 1 file changed, 86 insertions(+), 34 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 99ea2ab..bf7463a 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -1,28 +1,47 @@ # Automated Testing ## Glossary -**Confidence** - describes a degree to which passing tests guarantee that the app is working. -**Determinism** - describes how easy it is to determine where the problem is based on the failing test. -**Use Case** - a potential scenario in which a system receives external input and responds to it. It defines the interactions between a role (user or another system) and a system to achieve a goal. -**Combinatiorial Explosion** - the fast growth in the number of combinations that need to be tested when multiple business rules are involved. +**Confidence** - describes a degree to which passing tests guarantee that the +app is working. +**Determinism** - describes how easy it is to determine where the problem is +based on the failing test. +**Use Case** - a potential scenario in which a system receives external input +and responds to it. It defines the interactions between a role (user or another +system) and a system to achieve a goal. +**Combinatiorial Explosion** - the fast growth in the number of combinations +that need to be tested when multiple business rules are involved. ## Testing best practices ### Quality over quantity Don't focus on achieving a specific code coverage percentage. -While code coverage can help us identify uncovered parts of the codebase, it doesn't guarantee high confidence. +While code coverage can help us identify uncovered parts of the codebase, it +doesn't guarantee high confidence. -Instead, focus on identifying important paths of the application, especially from user's perspective. -User can be a developer using a shared function, a user interacting with the UI, or a client using server app's JSON API. -Write tests to cover those paths in a way that gives confidence that each path, and each separate part of the path works as expected. +Instead, focus on identifying important paths of the application, especially +from user's perspective. +User can be a developer using a shared function, a user interacting with the UI, +or a client using server app's JSON API. +Write tests to cover those paths in a way that gives confidence that each path, +and each separate part of the path works as expected. --- -Flaky tests that produce inconsistent results ruin confidence in the test suite, mask real issues, and are the source of frustration. The refactoring process to address the flakiness is crucial and should be a priority. -To adequately deal with flaky tests it is important to know how to identify, fix, and prevent them: -- Common characteristics of flaky tests include inconsistency, false positives and negatives, and sensitivity to dependency, timing, ordering, and environment. -- Typical causes of the stated characteristics are concurrency, timing/ordering problems, external dependencies, non-deterministic assertions, test environment instability, and poorly written test logic. -- Detecting flaky tests can be achieved by rerunning, running tests in parallel, executing in different environments, and analyzing test results. -- To fix and prevent further occurrences of flaky tests the following steps can be taken, isolate tests, employ setup and cleanup routines, handle concurrency, configure a stable test environment, improve error handling, simplify testing logic, and proactively deal with typical causes of the flaky tests. +Flaky tests that produce inconsistent results ruin confidence in the test suite, +mask real issues, and are the source of frustration. The refactoring process to +address the flakiness is crucial and should be a priority. +To adequately deal with flaky tests it is important to know how to identify, +fix, and prevent them: +- Common characteristics of flaky tests include inconsistency, false positives +and negatives, and sensitivity to dependency, timing, ordering, and environment. +- Typical causes of the stated characteristics are concurrency, timing/ordering +problems, external dependencies, non-deterministic assertions, test environment +instability, and poorly written test logic. +- Detecting flaky tests can be achieved by rerunning, running tests in parallel, +executing in different environments, and analyzing test results. +- To fix and prevent further occurrences of flaky tests the following steps can +be taken, isolate tests, employ setup and cleanup routines, handle concurrency, +configure a stable test environment, improve error handling, simplify testing +logic, and proactively deal with typical causes of the flaky tests. --- @@ -74,19 +93,25 @@ It is convenient to test a specific function at the lowest level so if the logic changes, we can make minimal changes to the test suite and/or mocked data. #### When to use -- Test a unit that implements the business logic, that's isolated from side effects such as database interaction or HTTP request processing +- Test a unit that implements the business logic, that's isolated from side +effects such as database interaction or HTTP request processing - Test function or class method with multiple input-output permutations #### When **not** to use -- To test unit that integrates different application layers, such as persistence layer (database) or HTTP layer (see "Integration Tests") or performs disk I/O or communicates with external system +- To test unit that integrates different application layers, such as persistence +layer (database) or HTTP layer (see "Integration Tests") or performs disk I/O or +communicates with external system #### Best practices - Unit tests should execute fast (<50ms) - Use mocks and stubs through dependency injection (method or constructor injection) #### Antipatterns -- Mocking infrastructure parts such as database I/O - instead, revert the control by using the `AppService`, `Command` or `Query` to integrate unit implementing business logic with the infrastructure layer of the application -- Monkey-patching dependencies used by the unit - instead, pass the dependencies through the constructor or method, so that you can pass the mocks or stubs in the test +- Mocking infrastructure parts such as database I/O - instead, revert the +control by using the `AppService`, `Command` or `Query` to integrate unit +implementing business logic with the infrastructure layer of the application +- Monkey-patching dependencies used by the unit - instead, pass the dependencies +through the constructor or method, so that you can pass the mocks or stubs in the test ### Integration Tests @@ -95,20 +120,36 @@ With these tests, we test how multiple components of the system behave together. #### Infrastructure -Running the tests on test infrastructure should be preferred to mocking, unlike in unit tests. Ideally, a full application instance would be run, to mimic real application behavior as close as possible. -This usually includes running the application connected to a test database, inserting fake data into it during the test setup, and doing assertions on the current state of the database. This also means integration test code should have full access to the test infrastructure for querying. +Running the tests on test infrastructure should be preferred to mocking, unlike +in unit tests. Ideally, a full application instance would be run, to mimic real +application behavior as close as possible. +This usually includes running the application connected to a test database, +inserting fake data into it during the test setup, and doing assertions on the +current state of the database. This also means integration test code should have +full access to the test infrastructure for querying. > [!NOTE] -> Regardless of whether using raw queries or the ORM, simple queries should be used to avoid introducing business logic within tests. +> Regardless of whether using raw queries or the ORM, simple queries should be +> used to avoid introducing business logic within tests. -However, mocking can still be used when needed, for example when expecting side-effects that call third party services. +However, mocking can still be used when needed, for example when expecting +side-effects that call third party services. #### Entry points -Integration test entry points can vary depending on the application use cases. These include services, controllers, or the API. These are not set in stone and should be taken into account when making a decision. For example: -- A use case that can be invoked through multiple different protocols can be tested separately from them, to avoid duplication. A tradeoff in this case is the need to write some basic tests for each of the protocols. -- A use case that will always be invokeable through a single protocol might benefit enough from only being tested using that protocol. E.g. a HTTP API route test might eliminate the need for a lower level, controller/service level test. This would also enable testing the auth layer integration within these tests, which might not have been possible otherwise depending on the technology used. - -Multiple approaches can be used within the same application depending on the requirements, to provide sufficient coverage. +Integration test entry points can vary depending on the application use cases. +These include services, controllers, or the API. These are not set in stone and +should be taken into account when making a decision. For example: +- A use case that can be invoked through multiple different protocols can be +tested separately from them, to avoid duplication. A tradeoff in this case is +the need to write some basic tests for each of the protocols. +- A use case that will always be invokeable through a single protocol might +benefit enough from only being tested using that protocol. E.g. a HTTP API route +test might eliminate the need for a lower level, controller/service level test. +This would also enable testing the auth layer integration within these tests, +which might not have been possible otherwise depending on the technology used. + +Multiple approaches can be used within the same application depending on the +requirements, to provide sufficient coverage. #### Testing surface @@ -132,11 +173,16 @@ time decoupling logic testing and endpoint testing. - To verify the API endpoint performs authentication and authorization. - To verify user permissions for that endpoint. - To verify that invalid input is correctly handled. -- To verify the basic business logic is handled correctly, both in expected success and failure cases. -- To verify infrastructure related side-effects, e.g. database changes or calls to third party services. +- To verify the basic business logic is handled correctly, both in expected +success and failure cases. +- To verify infrastructure related side-effects, e.g. database changes or calls +to third party services. #### When **not** to use -- For extensive testing of business logic permutations beyond fundamental scenarios. Integration tests contain more overhead to write compared to unit tests and can easily lead to a combinatorial explosion. Instead, unit tests should be used for thorough coverage of these permutations. +- For extensive testing of business logic permutations beyond fundamental +scenarios. Integration tests contain more overhead to write compared to unit +tests and can easily lead to a combinatorial explosion. Instead, unit tests +should be used for thorough coverage of these permutations. - For testing third party services. We should assume they work as expected. #### Best practices @@ -157,10 +203,16 @@ returns the expected data. That means we write tests for the publically available endpoints. > [!NOTE] -> As mentioned in the Integration Tests section, API can be the entry point to the integration tests, meaning API tests are a subtype of integration tests. However, when we talk about API tests here, we are specifically referring to the public API contract tests, which don't have access to the internals of the application. - -In the cases where API routes are covered extensively with integration tests, API tests might not be needed, leaving more time for QA to focus on E2E tests. -However, in more complex architectures (e.g. integration tested microservices behind an API gateway), API tests can be very useful. +> As mentioned in the Integration Tests section, API can be the entry point to +> the integration tests, meaning API tests are a subtype of integration tests. +> However, when we talk about API tests here, we are specifically referring to +> the public API contract tests, which don't have access to the internals of +> the application. + +In the cases where API routes are covered extensively with integration tests, +API tests might not be needed, leaving more time for QA to focus on E2E tests. +However, in more complex architectures (e.g. integration tested microservices +behind an API gateway), API tests can be very useful. #### When to use - To make sure the API signature is valid. From e5f8dfbbb0f005d5a2b6caed8ae625a714493c62 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 22:17:50 +0100 Subject: [PATCH 06/12] Rephrase E2E tests section --- recipes/automated-testing.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index bf7463a..50d7f37 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -228,10 +228,10 @@ performance tests (K6). ### E2E Tests -These tests are executed within a browser environment (Playwright, Selenium, -etc.). The purpose of these tests is to make sure that interacting with the -application UI produces the expected result, verifying the application’s -functionality from a user’s perspective. +E2E tests are executed in a browser environment using tools like Playwright, +Selenium, or similar frameworks. The purpose of these tests is to make sure that +interacting with the application UI produces the expected result, verifying the +application’s functionality from a user’s perspective. Usually, these tests will cover a large portion of the codebase with least amount of code. Because of that, they can be the first tests to be added to From db76fd3e7b904addf553becfef2d0bfa138c55fd Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 8 Jan 2025 23:10:25 +0100 Subject: [PATCH 07/12] Revert changes on new lines --- recipes/automated-testing.md | 120 ++++++++++------------------------- 1 file changed, 34 insertions(+), 86 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 50d7f37..e858011 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -1,47 +1,28 @@ # Automated Testing ## Glossary -**Confidence** - describes a degree to which passing tests guarantee that the -app is working. -**Determinism** - describes how easy it is to determine where the problem is -based on the failing test. -**Use Case** - a potential scenario in which a system receives external input -and responds to it. It defines the interactions between a role (user or another -system) and a system to achieve a goal. -**Combinatiorial Explosion** - the fast growth in the number of combinations -that need to be tested when multiple business rules are involved. +**Confidence** - describes a degree to which passing tests guarantee that the app is working. +**Determinism** - describes how easy it is to determine where the problem is based on the failing test. +**Use Case** - a potential scenario in which a system receives external input and responds to it. It defines the interactions between a role (user or another system) and a system to achieve a goal. +**Combinatiorial Explosion** - the fast growth in the number of combinations that need to be tested when multiple business rules are involved. ## Testing best practices ### Quality over quantity Don't focus on achieving a specific code coverage percentage. -While code coverage can help us identify uncovered parts of the codebase, it -doesn't guarantee high confidence. +While code coverage can help us identify uncovered parts of the codebase, it doesn't guarantee high confidence. -Instead, focus on identifying important paths of the application, especially -from user's perspective. -User can be a developer using a shared function, a user interacting with the UI, -or a client using server app's JSON API. -Write tests to cover those paths in a way that gives confidence that each path, -and each separate part of the path works as expected. +Instead, focus on identifying important paths of the application, especially from user's perspective. +User can be a developer using a shared function, a user interacting with the UI, or a client using server app's JSON API. +Write tests to cover those paths in a way that gives confidence that each path, and each separate part of the path works as expected. --- -Flaky tests that produce inconsistent results ruin confidence in the test suite, -mask real issues, and are the source of frustration. The refactoring process to -address the flakiness is crucial and should be a priority. -To adequately deal with flaky tests it is important to know how to identify, -fix, and prevent them: -- Common characteristics of flaky tests include inconsistency, false positives -and negatives, and sensitivity to dependency, timing, ordering, and environment. -- Typical causes of the stated characteristics are concurrency, timing/ordering -problems, external dependencies, non-deterministic assertions, test environment -instability, and poorly written test logic. -- Detecting flaky tests can be achieved by rerunning, running tests in parallel, -executing in different environments, and analyzing test results. -- To fix and prevent further occurrences of flaky tests the following steps can -be taken, isolate tests, employ setup and cleanup routines, handle concurrency, -configure a stable test environment, improve error handling, simplify testing -logic, and proactively deal with typical causes of the flaky tests. +Flaky tests that produce inconsistent results ruin confidence in the test suite, mask real issues, and are the source of frustration. The refactoring process to address the flakiness is crucial and should be a priority. +To adequately deal with flaky tests it is important to know how to identify, fix, and prevent them: +- Common characteristics of flaky tests include inconsistency, false positives and negatives, and sensitivity to dependency, timing, ordering, and environment. +- Typical causes of the stated characteristics are concurrency, timing/ordering problems, external dependencies, non-deterministic assertions, test environment instability, and poorly written test logic. +- Detecting flaky tests can be achieved by rerunning, running tests in parallel, executing in different environments, and analyzing test results. +- To fix and prevent further occurrences of flaky tests the following steps can be taken, isolate tests, employ setup and cleanup routines, handle concurrency, configure a stable test environment, improve error handling, simplify testing logic, and proactively deal with typical causes of the flaky tests. --- @@ -93,25 +74,19 @@ It is convenient to test a specific function at the lowest level so if the logic changes, we can make minimal changes to the test suite and/or mocked data. #### When to use -- Test a unit that implements the business logic, that's isolated from side -effects such as database interaction or HTTP request processing +- Test a unit that implements the business logic, that's isolated from side effects such as database interaction or HTTP request processing - Test function or class method with multiple input-output permutations #### When **not** to use -- To test unit that integrates different application layers, such as persistence -layer (database) or HTTP layer (see "Integration Tests") or performs disk I/O or -communicates with external system +- To test unit that integrates different application layers, such as persistence layer (database) or HTTP layer (see "Integration Tests") or performs disk I/O or communicates with external system #### Best practices - Unit tests should execute fast (<50ms) - Use mocks and stubs through dependency injection (method or constructor injection) #### Antipatterns -- Mocking infrastructure parts such as database I/O - instead, revert the -control by using the `AppService`, `Command` or `Query` to integrate unit -implementing business logic with the infrastructure layer of the application -- Monkey-patching dependencies used by the unit - instead, pass the dependencies -through the constructor or method, so that you can pass the mocks or stubs in the test +- Mocking infrastructure parts such as database I/O - instead, revert the control by using the `AppService`, `Command` or `Query` to integrate unit implementing business logic with the infrastructure layer of the application +- Monkey-patching dependencies used by the unit - instead, pass the dependencies through the constructor or method, so that you can pass the mocks or stubs in the test ### Integration Tests @@ -120,36 +95,20 @@ With these tests, we test how multiple components of the system behave together. #### Infrastructure -Running the tests on test infrastructure should be preferred to mocking, unlike -in unit tests. Ideally, a full application instance would be run, to mimic real -application behavior as close as possible. -This usually includes running the application connected to a test database, -inserting fake data into it during the test setup, and doing assertions on the -current state of the database. This also means integration test code should have -full access to the test infrastructure for querying. +Running the tests on test infrastructure should be preferred to mocking, unlike in unit tests. Ideally, a full application instance would be run, to mimic real application behavior as close as possible. +This usually includes running the application connected to a test database, inserting fake data into it during the test setup, and doing assertions on the current state of the database. This also means integration test code should have full access to the test infrastructure for querying. > [!NOTE] -> Regardless of whether using raw queries or the ORM, simple queries should be -> used to avoid introducing business logic within tests. +> Regardless of whether using raw queries or the ORM, simple queries should be used to avoid introducing business logic within tests. -However, mocking can still be used when needed, for example when expecting -side-effects that call third party services. +However, mocking can still be used when needed, for example when expecting side-effects that call third party services. #### Entry points -Integration test entry points can vary depending on the application use cases. -These include services, controllers, or the API. These are not set in stone and -should be taken into account when making a decision. For example: -- A use case that can be invoked through multiple different protocols can be -tested separately from them, to avoid duplication. A tradeoff in this case is -the need to write some basic tests for each of the protocols. -- A use case that will always be invokeable through a single protocol might -benefit enough from only being tested using that protocol. E.g. a HTTP API route -test might eliminate the need for a lower level, controller/service level test. -This would also enable testing the auth layer integration within these tests, -which might not have been possible otherwise depending on the technology used. - -Multiple approaches can be used within the same application depending on the -requirements, to provide sufficient coverage. +Integration test entry points can vary depending on the application use cases. These include services, controllers, or the API. These are not set in stone and should be taken into account when making a decision. For example: +- A use case that can be invoked through multiple different protocols can be tested separately from them, to avoid duplication. A tradeoff in this case is the need to write some basic tests for each of the protocols. +- A use case that will always be invokeable through a single protocol might benefit enough from only being tested using that protocol. E.g. a HTTP API route test might eliminate the need for a lower level, controller/service level test. This would also enable testing the auth layer integration within these tests, which might not have been possible otherwise depending on the technology used. + +Multiple approaches can be used within the same application depending on the requirements, to provide sufficient coverage. #### Testing surface @@ -173,16 +132,11 @@ time decoupling logic testing and endpoint testing. - To verify the API endpoint performs authentication and authorization. - To verify user permissions for that endpoint. - To verify that invalid input is correctly handled. -- To verify the basic business logic is handled correctly, both in expected -success and failure cases. -- To verify infrastructure related side-effects, e.g. database changes or calls -to third party services. +- To verify the basic business logic is handled correctly, both in expected success and failure cases. +- To verify infrastructure related side-effects, e.g. database changes or calls to third party services. #### When **not** to use -- For extensive testing of business logic permutations beyond fundamental -scenarios. Integration tests contain more overhead to write compared to unit -tests and can easily lead to a combinatorial explosion. Instead, unit tests -should be used for thorough coverage of these permutations. +- For extensive testing of business logic permutations beyond fundamental scenarios. Integration tests contain more overhead to write compared to unit tests and can easily lead to a combinatorial explosion. Instead, unit tests should be used for thorough coverage of these permutations. - For testing third party services. We should assume they work as expected. #### Best practices @@ -203,16 +157,10 @@ returns the expected data. That means we write tests for the publically available endpoints. > [!NOTE] -> As mentioned in the Integration Tests section, API can be the entry point to -> the integration tests, meaning API tests are a subtype of integration tests. -> However, when we talk about API tests here, we are specifically referring to -> the public API contract tests, which don't have access to the internals of -> the application. - -In the cases where API routes are covered extensively with integration tests, -API tests might not be needed, leaving more time for QA to focus on E2E tests. -However, in more complex architectures (e.g. integration tested microservices -behind an API gateway), API tests can be very useful. +> As mentioned in the Integration Tests section, API can be the entry point to the integration tests, meaning API tests are a subtype of integration tests. However, when we talk about API tests here, we are specifically referring to the public API contract tests, which don't have access to the internals of the application. + +In the cases where API routes are covered extensively with integration tests, API tests might not be needed, leaving more time for QA to focus on E2E tests. +However, in more complex architectures (e.g. integration tested microservices behind an API gateway), API tests can be very useful. #### When to use - To make sure the API signature is valid. From b2ddc3a4625313e73c60a51a1e268e95c1807895 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Thu, 9 Jan 2025 10:17:22 +0100 Subject: [PATCH 08/12] Resolve comments --- recipes/automated-testing.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index e858011..00bec21 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -216,8 +216,6 @@ deployment process. #### Antipatterns - Avoid trying to cover all use cases or edge cases in E2E tests; these are better suited for unit or integration tests. -- Don’t bypass or disable flaky tests without fixing them, as they undermine the -test suite’s reliability. ### Performance Tests @@ -269,7 +267,7 @@ inputs and dataset sizes. ### Visual Tests The type of test where test runner navigates to browser page, takes snapshot and -then compares the snapshots with the reference snapshot. +then compares the snapshot with the reference snapshot. Visual tests allow you to quickly cover large portions of the application, ensuring that changes in the UI are detected without writing complex test cases. @@ -287,8 +285,9 @@ development. - To test a specific user scenario. #### Best practices -- Ensure the UI consistently renders the same output by controlling randomness -(e.g., setting seeds for random data or controlling API responses). +- Ensure the UI consistently renders the same output by eliminating randomness +(e.g., by always using same seeds data or controlling API responses to always +return same values). - Add as many pages as possible but keep the tests simple. - Consider running visual tests at the component level to isolate and detect issues earlier. From fad7026b8e6cfd733d399cc5e1ccbd6b7706c465 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Thu, 9 Jan 2025 10:22:50 +0100 Subject: [PATCH 09/12] Add best practice point for performance tests --- recipes/automated-testing.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 00bec21..6c91fd2 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -244,6 +244,9 @@ patterns. - To test a specific user scenario. #### Best practices +- Establish clear goals. Are you testing scalability, stability, or +responsiveness? Without these objectives, tests risk being unfocused, resulting +in meaningless data. - Ensure the tests mimic actual user behavior, including realistic click frequency, page navigation patterns, and input actions. - Include diverse scenarios that represent different user journeys across the From 4e5da3033bba0a6d94ee5bfe70d82c8f7ba671fc Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Fri, 10 Jan 2025 12:49:07 +0100 Subject: [PATCH 10/12] Refine performance testing guidelines --- recipes/automated-testing.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index 6c91fd2..ff52d70 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -247,10 +247,8 @@ patterns. - Establish clear goals. Are you testing scalability, stability, or responsiveness? Without these objectives, tests risk being unfocused, resulting in meaningless data. -- Ensure the tests mimic actual user behavior, including realistic click -frequency, page navigation patterns, and input actions. - Include diverse scenarios that represent different user journeys across the -system, not just a single performance test. +system, not just a single performance test/scenario. - Use a clone of the production environment to ensure the infrastructure matches real-world conditions, including hardware, network, and database configurations. - Schedule performance tests periodically or before major releases to catch @@ -262,10 +260,13 @@ process integrated into the development lifecycle. #### Antipatterns - Running these tests locally or on an environment that doesn't match production -in terms of infrastructure performance. (tests should be developed on a local -instance, but the actual measurements should be performed live) -- Ensure the test data mirrors real-world conditions, including varying user -inputs and dataset sizes. +in terms of infrastructure performance. Tests should be developed on a local +instance, but the actual measurements should be performed live. +- Ignoring data variability, ensure the test data mirrors real-world conditions, +including varying user inputs and dataset sizes. +- Ignoring randomness in user behavior, ensure the tests mimic actual user +behavior, including realistic click frequency, page navigation patterns, and +input actions. ### Visual Tests From 1849fe7bc9beebda6896473d01312b6791d1ce4b Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Fri, 10 Jan 2025 15:06:57 +0100 Subject: [PATCH 11/12] Remove redundant points from automated testing guidelines --- recipes/automated-testing.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index ff52d70..c0ac686 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -280,9 +280,6 @@ the root cause of errors. #### When to use - When we want to make sure there are no changes in the UI. -- When we want to increase test coverage with least effort. -- During the early stages of a project, while E2E tests are still under -development. #### When **not** to use - To test a specific feature or business logic. From 2be0f32119f3f473a29cb1c204ea4fe71d32a820 Mon Sep 17 00:00:00 2001 From: Marko Lovric Date: Wed, 22 Jan 2025 15:53:40 +0100 Subject: [PATCH 12/12] Update When to use and Best practices in E2E tests section --- recipes/automated-testing.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/recipes/automated-testing.md b/recipes/automated-testing.md index c0ac686..6916ec6 100644 --- a/recipes/automated-testing.md +++ b/recipes/automated-testing.md @@ -191,12 +191,15 @@ lower level (integration or unit tests). #### When to use - To validate user interactions and critical workflows in the application UI. -- For testing full system integration across multiple services or components. +- For testing specific user flows. +- For making sure that critical application features are working as expected. +- For better coverage of the most common user pathways. #### When **not** to use - For data validation. #### Best practices +- Tests should be atomic and simple, all complicated tests should be thrown out. - Focus on the most important user workflows rather than attempting exhaustive coverage. - Each test should be able to run independently, with the environment reset to a