This package gathers multiple test suites to assess the sanity of any Airbyte connector. It is shipped as a pytest plugin and relies on pytest to discover, configure and execute tests. Test-specific documentation can be found here.
The acceptance tests are configured via the acceptance-test-config.yml
YAML file, which is passed to the plugin via the --acceptance-test-config
option.
Note there are MANY ways to do this at this time, but we are working on consolidating them.
Which method you choose to use depends on the context you are in.
Pre-requisites:
- Setting up a Service Account for Google Secrets Manager (GSM) access. See here
- Ensuring that you have the
GCP_GSM_CREDENTIALS
environment variable set to the contents of your GSM service account key file. - Poetry installed
- Pipx installed
_Note: Install instructions for airbyte-ci are here _
This runs connector acceptance and other tests that run in our CI
airbyte-ci connectors --name=<connector-name> test
Pre-requisites:
To learn how to set up ci_credentials
and your GSM Service account see here
# Hook up your GSM service account
export GCP_GSM_CREDENTIALS=`cat <path-to-gsm-service-account-key-file>`
# Install the credentials tool
pipx install airbyte-ci/connectors/ci_credentials/ --force --editable
Retrieve a connectors sandbox secrets
# From the root of the airbyte repo
# Writes the secrets to airbyte-integrations/connectors/source-faker/secrets
VERSION=dev ci_credentials connectors/source-faker write-to-storage
Run install dependencies
# Navigate to our CAT test directory
cd airbyte-integrations/bases/connector-acceptance-test/
# Install dependencies
poetry install
Run the tests
# Run tests against your connector
poetry run pytest -p connector_acceptance_test.plugin --acceptance-test-config=../../connectors/source-faker --pdb
cd
into your connector project (e.g.airbyte-integrations/connectors/source-pokeapi
)- Edit
acceptance-test-config.yml
according to your need. Please refer to our Connector Acceptance Test Reference if you need details about the available options. - Build the connector docker image ( e.g.:
airbyte-ci connectors --name=source-pokeapi build
) - Use one of the following ways to run tests (from your connector project directory)
You may want to iterate on the acceptance test project itself: adding new tests, fixing a bug etc. These iterations are more conveniently achieved by remaining in the current directory.
- Install dependencies via
poetry install
- Run the unit tests on the acceptance tests themselves:
poetry run pytest unit_tests
(add the--pdb
option if you want to enable the debugger on test failure) - To run specific unit test(s), add
-k
to the above command, e.g.poetry run python -m pytest unit_tests -k 'test_property_can_store_secret'
. You can use wildcards*
here as well. - Make the changes you want:
- Global pytest fixtures are defined in
./connector_acceptance_test/conftest.py
- Existing test modules are defined in
./connector_acceptance_test/tests
acceptance-test-config.yaml
structure is defined in./connector_acceptance_test/config.py
- Global pytest fixtures are defined in
- Unit test your changes by adding tests to
./unit_tests
- Run the unit tests on the acceptance tests again:
poetry run pytest unit_tests
, make sure the coverage did not decrease. You can bypass slow tests by using theslow
marker:poetry run pytest unit_tests -m "not slow"
. - Manually test the changes you made by running acceptance tests on a specific connector:
- First build the connector to ensure your local image is up-to-date:
airbyte-ci connectors --name=source-pokeapi build
- Then run the acceptance tests on the connector:
poetry run pytest -p connector_acceptance_test.plugin --acceptance-test-config=../../connectors/source-pokeapi
- First build the connector to ensure your local image is up-to-date:
- Make sure you updated
docs/connector-development/testing-connectors/connector-acceptance-tests-reference.md
according to your changes - Update the project changelog
airbyte-integrations/bases/connector-acceptance-test/CHANGELOG.md
- Open a PR on our GitHub repository
- This GitHub action workflow will be triggered and run the unit tests on your branch.
- Publish the new acceptance test version if your PR is approved by running
/legacy-publish connector=bases/connector-acceptance-test run-tests=false
in a GitHub comment - Merge your PR
We introduced changes in the structure of acceptance-test-config.yml
files in version 0.2.12.
The legacy configuration format is still supported but should be deprecated soon.
To migrate a legacy configuration to the latest configuration format please run:
python -m venv .venv # If you don't have a virtualenv already
source ./.venv/bin/activate # If you're not in your virtualenv already
python connector_acceptance_test/tools/strictness_level_migration/config_migration.py ../../connectors/source-to-migrate/acceptance-test-config.yml