-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Standard tests: update template (#2937)
* update template for python sources * update templates and their docs * update docs, rename folders, add acceptance script * adding hooks for custom setup and teardown + improve plugin behaviour, enable it explicitly + add acceptance.py to run acceptance tests together with integration tests + remove helper acceptance-test-python.sh + fix gradle command for acceptance tests + update docs Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com> Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
- Loading branch information
1 parent
94d5b27
commit 88b77aa
Showing
50 changed files
with
579 additions
and
310 deletions.
There are no files selected for viewing
2 changes: 1 addition & 1 deletion
2
airbyte-integrations/bases/source-acceptance-test/.dockerignore
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
* | ||
!Dockerfile | ||
!standard_test | ||
!source_acceptance_test | ||
!setup.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
24 changes: 24 additions & 0 deletions
24
airbyte-integrations/connector-templates/source-python/acceptance-test-config.yml.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# See [Source Acceptance Tests](https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/source-acceptance-tests.md) | ||
# for more information about how to configure these tests | ||
connector_image: airbyte/{{dashCase name}} | ||
tests: | ||
spec: | ||
- spec_path: "source_{{snakeCase name}}/spec.json" | ||
connection: | ||
- config_path: "secrets/config.json" | ||
status: "succeed" | ||
- config_path: "integration_tests/invalid_config.json" | ||
status: "exception" | ||
discovery: | ||
- config_path: "secrets/config.json" | ||
basic_read: | ||
- config_path: "secrets/config.json" | ||
configured_catalog_path: "integration_tests/configured_catalog.json" | ||
validate_output_from_all_streams: yes | ||
incremental: # TODO if your connector does not implement incremental sync, remove this block | ||
- config_path: "secrets/config.json" | ||
configured_catalog_path: "integration_tests/configured_catalog.json" | ||
state_path: "integration_tests/abnormal_state.json" | ||
full_refresh: | ||
- config_path: "secrets/config.json" | ||
configured_catalog_path: "integration_tests/configured_catalog.json" |
7 changes: 7 additions & 0 deletions
7
airbyte-integrations/connector-templates/source-python/acceptance-test-docker.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
#!/usr/bin/env sh | ||
docker run --rm -i airbyte/source-acceptance-test \ | ||
-v /var/run/docker.sock:/var/run/docker.sock \ | ||
-v /tmp:/tmp \ | ||
-v ./:/test_input \ | ||
-w /test_input \ | ||
-p integration_tests.acceptance |
24 changes: 3 additions & 21 deletions
24
airbyte-integrations/connector-templates/source-python/build.gradle.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,32 +1,14 @@ | ||
plugins { | ||
id 'airbyte-python' | ||
id 'airbyte-docker' | ||
id 'airbyte-standard-source-test-file' | ||
id 'airbyte-source-acceptance-test' | ||
} | ||
|
||
airbytePython { | ||
moduleDirectory 'source_{{snakeCase name}}' | ||
moduleDirectory 'source_{{snakeCase name}}_singer' | ||
} | ||
|
||
airbyteStandardSourceTestFile { | ||
// For more information on standard source tests, see https://docs.airbyte.io/contributing-to-airbyte/building-new-connector/testing-connectors | ||
|
||
// All these input paths must live inside this connector's directory (or subdirectories) | ||
// TODO update the spec JSON file | ||
specPath = "source_{{snakeCase name}}/spec.json" | ||
|
||
// configPath points to a config file which matches the spec.json supplied above. secrets/ is gitignored by default, so place your config file | ||
// there (in case it contains any credentials) | ||
// TODO update the config file to contain actual credentials | ||
configPath = "secrets/config.json" | ||
// TODO update the sample configured_catalog JSON for use in testing | ||
// Note: If your source supports incremental syncing, then make sure that the catalog that is returned in the get_catalog method is configured | ||
// for incremental syncing (e.g. include cursor fields, etc). | ||
configuredCatalogPath = "sample_files/configured_catalog.json" | ||
} | ||
|
||
|
||
dependencies { | ||
implementation files(project(':airbyte-integrations:bases:base-standard-source-test-file').airbyteDocker.outputs) | ||
implementation files(project(':airbyte-integrations:bases:source-acceptance-test').airbyteDocker.outputs) | ||
implementation files(project(':airbyte-integrations:bases:base-python').airbyteDocker.outputs) | ||
} |
Empty file.
5 changes: 5 additions & 0 deletions
5
airbyte-integrations/connector-templates/source-python/integration_tests/abnormal_state.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"todo-stream-name": { | ||
"todo-field-name": "todo-abnormal-value" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
3 changes: 3 additions & 0 deletions
3
airbyte-integrations/connector-templates/source-python/integration_tests/invalid_config.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"todo-wrong-field": "this should be an incomplete config file, used in standard tests" | ||
} |
3 changes: 3 additions & 0 deletions
3
airbyte-integrations/connector-templates/source-python/integration_tests/sample_config.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"fix-me": "TODO" | ||
} |
5 changes: 5 additions & 0 deletions
5
airbyte-integrations/connector-templates/source-python/integration_tests/sample_state.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
{ | ||
"todo-stream-name": { | ||
"todo-field-name": "value" | ||
} | ||
} |
1 change: 1 addition & 0 deletions
1
airbyte-integrations/connector-templates/source-python/requirements.txt.hbs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,5 @@ | ||
# This file is autogenerated -- only edit if you know what you are doing. Use setup.py for declaring dependencies. | ||
-e ../../bases/airbyte-protocol | ||
-e ../../bases/base-python | ||
-e ../../bases/source-acceptance-test | ||
-e . |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.