SAT: measure unit test coverage (#15443)

airbyte-integrations/bases/source-acceptance-test/.coveragerc

@@ -0,0 +1,8 @@
+[report]
+# show lines missing coverage
+show_missing = true
+# coverage 64% measured on 62303a85def89450d2e46573a3d96cd326f2e921 (2022-08-09)
+# This number should probably never be adjusted down, only up i.e: we should only ever increase our test coverage
+fail_under = 64
+skip_covered = true
+skip_empty = true

airbyte-integrations/bases/source-acceptance-test/CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog
 
+## 0.1.61
+Add unit tests coverage computation [#15443](https://github.com/airbytehq/airbyte/pull/15443/).
+
 ## 0.1.60
 Backward compatibility tests: validate fake previous config against current connector specification. [#15367](https://github.com/airbytehq/airbyte/pull/15367)
 

airbyte-integrations/bases/source-acceptance-test/Dockerfile

@@ -33,7 +33,7 @@ COPY pytest.ini setup.py ./
 COPY source_acceptance_test ./source_acceptance_test
 RUN pip install .
 
-LABEL io.airbyte.version=0.1.60
+LABEL io.airbyte.version=0.1.61
 LABEL io.airbyte.name=airbyte/source-acceptance-test
 
 ENTRYPOINT ["python", "-m", "pytest", "-p", "source_acceptance_test.plugin", "-r", "fEsx"]

airbyte-integrations/bases/source-acceptance-test/README.md

@@ -1,66 +1,59 @@
 # Source Acceptance Tests
-This package uses pytest to discover, configure and execute the tests.
-It implemented as a pytest plugin.
+This package gathers multiple test suites to assess the sanity of any Airbyte **source** connector. 
+It is shipped as a [pytest](https://docs.pytest.org/en/7.1.x/) plugin and relies on pytest to discover, configure and execute tests.
+Test-specific documentation can be found [here](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference/)).
 
-It adds new configuration option `--acceptance-test-config` - path to configuration file (by default is current folder). 
-Configuration stored in YaML format and validated by pydantic.
+## Running the acceptance tests on a source connector:
+1. `cd` into your connector project (e.g. `airbyte-integrations/connectors/source-pokeapi`) 
+2. Edit `acceptance-test-config.yml` according to your need. Please refer to our [Source Acceptance Test Reference](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference/) if you need details about the available options.
+3. Build the connector docker image ( e.g.: `docker build . -t airbyte/source-pokeapi:dev`)
+4. Use one of the following ways to run tests (**from your connector project directory**)
 
-Example configuration can be found in `sample_files/` folder:
-```yaml
-connector_image: <your_image>
-tests:
-  spec:
-    - spec_path: "<connector_folder>/spec.json"
-  connection:
-    - config_path: "secrets/config.json"
-      status: "succeed"
-    - config_path: "sample_files/invalid_config.json"
-      status: "exception"
-  discovery:
-    - config_path: "secrets/config.json"
-  basic_read:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "sample_files/configured_catalog.json"
-      empty_streams: []
-  incremental:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "sample_files/configured_catalog.json"
-      future_state_path: "sample_files/abnormal_state.json"
-      cursor_paths:
-        subscription_changes: ["timestamp"]
-        email_events: ["timestamp"]
-  full_refresh:
-    - config_path: "secrets/config.json"
-      configured_catalog_path: "sample_files/configured_catalog.json"
-```
-Required steps to test connector are the following:
-* Build docker image for connector
-* Create `acceptance-test-config.yml` file with test settings, Note: all paths in this files are relative to its location
-* Use one of the following ways to run tests:
-
-## Running
-Using python
+**Using python**
 ```bash
-cd ../../base/source-acceptance-test
-python -m pytest source_acceptance_test/tests --acceptance-test-config=<path_to_your_connector> -vvv
+python -m pytest integration_tests -p integration_tests.acceptance
 ```
 _Note: this will assume that docker image for connector is already built_
 
-Using Gradle
+**Using Gradle**
 ```bash
 ./gradlew :airbyte-integrations:connectors:source-<name>:sourceAcceptanceTest
 ```
 _Note: this way will also build docker image for the connector_
 
-Using Bash
+**Using Bash**
 ```bash
-./source-acceptance-test.sh -vv
+./acceptance-test-docker.sh
 ```
-_Note: you can append any arguments to this command, they will be forwarded to pytest
-
-
-## Developing Locally
-
-To run the tests within this dir:
-* Ensure you have `venv` set up with `python3 -m venv .venv` & source it `source ./.venv/bin/activate`
-* Run tests with `python -m pytest -s unit_tests`
+_Note: this will use the latest docker image for source-acceptance-test and will also build docker image for the connector_
+
+## When does SAT run?
+* When running local acceptance tests on connector:
+  * When running `sourceAcceptanceTest` `gradle` task
+  * When running or `./acceptance-test-docker.sh` in a connector project
+* When running `/test` command on a GitHub pull request.
+* When running `/publish` command on a GitHub pull request.
+* When running ` integration-test` GitHub action that is creating the [connector builds summary](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/builds.md).
+
+## Developing on the SAT 
+You may want to iterate on the SAT project: adding new tests, fixing a bug etc.
+These iterations are more conveniently achieved by remaining in the current directory.
+
+1. Create a `virtualenv`: `python -m venv .venv`
+2. Activate the `virtualenv`: `source ./.venv/bin/activate`
+3. Install requirements: `pip install -e .`
+4. Run the unit tests on SAT: `python -m pytest unit_tests` (add the `--pdb` option if you want to enable the debugger on test failure)
+5. Make the changes you want:
+    * Global pytest fixtures are defined in `./source_acceptance_test/conftest.py`
+    * Existing test modules are defined in `./source_acceptance_test/tests`
+    * `acceptance-test-config.yaml` structure is defined in `./source_acceptance_test/config.py`
+6. Unit test your changes by adding tests to `./unit_tests`
+7. Run the unit tests on SAT again: `python -m pytest unit_tests`, make sure the coverage did not decrease.
+8. Manually test the changes you made by running SAT on a specific connector. e.g. `python -m pytest -p source_acceptance_test.plugin --acceptance-test-config=../../connectors/source-pokeapi`
+9. Make sure you updated `docs/connector-development/testing-connectors/source-acceptance-tests-reference.md` according to your changes
+10. Bump the SAT version in `airbyte-integrations/bases/source-acceptance-test/Dockerfile`
+11. Update the project changelog `airbyte-integrations/bases/source-acceptance-test/CHANGELOG.md`
+12. Open a PR on our GitHub repository
+13. Run the unit test on the CI by running `/test connector=bases/source-acceptance-test` in a GitHub comment
+14. Publish the new SAT version if your PR is approved by running `/publish connector=bases/source-acceptance-test auto-bump-version=false` in a GitHub comment
+15. Merge your PR

airbyte-integrations/bases/source-acceptance-test/pytest.ini

@@ -1,6 +1,6 @@
 [pytest]
 
-addopts = -r fEsx --capture=no -vv --log-level=INFO --color=yes --force-sugar
+addopts = -r fEsx --capture=no -vv --log-level=INFO --color=yes --force-sugar --cov=source_acceptance_test --no-cov-on-fail
 testpaths =
     source_acceptance_test/tests
 

airbyte-integrations/bases/source-acceptance-test/setup.py

@@ -23,6 +23,7 @@
     "deepdiff~=5.8.0",
     "requests-mock~=1.9.3",
     "pytest-mock~=3.6.1",
+    "pytest-cov~=3.0.0",
     "hypothesis~=6.54.1",
     "hypothesis-jsonschema~=0.20.1",  # TODO alafanechere upgrade to latest when jsonschema lib is upgraded to >= 4.0.0 in airbyte-cdk and SAT
 ]