Standard tests: update template #2937

keu · 2021-04-18T18:56:45Z

closes #2185

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Describe the solution

Pre-merge Checklist

Run integration tests
Publish Docker images

Recommended reading order

test.java
component.ts
the rest

sherifnada

`we`` should also change the README to show how to run tests, and do the same thing for the singer template

airbyte-integrations/connector-templates/source-python/sample_files/invalid_config.json

sherifnada

Looks good with some small comments! We should merge this with the docs changes so they are done atomically probably.

sherifnada · 2021-04-19T20:40:42Z

airbyte-integrations/connector-templates/source-python/README.md.hbs

@@ -86,6 +86,10 @@ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files
 1. To run additional integration tests, place your integration tests in a new directory `integration_tests` and run them with `python -m pytest -s integration_tests`.
   Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.

+### Standard Tests
+1. Update standard_test_config.yml file to disable/repeat or customize behaviour/inputs of specific tests.


Suggested change

1. Update standard_test_config.yml file to disable/repeat or customize behaviour/inputs of specific tests.

1. Update standard_test_config.yml file to configure tests. See (standard testing)[some-link] for more information.

this link should point to the actual docs once we have them.

sherifnada · 2021-04-19T20:43:23Z

airbyte-integrations/connector-templates/source-python/sample_files/abnormal_state.json

@@ -0,0 +1,5 @@
+{


I wonder if we should get rid of the sample_files directory and just integration_tests or something. I think less people will be confused by this. I am in favor of doing the latter name. We can do it now or in another PR. up to you.

sherifnada · 2021-04-19T20:44:05Z

airbyte-integrations/connector-templates/source-python/setup.py.hbs

-    package_data={"": ["*.json"]}
+    install_requires=MAIN_REQUIREMENTS,
+    package_data={"": ["*.json"]},
+    extras_require={


is it idiomatic to require installing an extra to run tests? I wasn't sure if this was the right solution earlier.

there are many envs, testing env might require quite heavy dependencies, and absolutely not needed for a connector to work. usually, it is a good thing to separate such dependencies, as well as sometimes separate dev dependencies (like tools to debug, etc)

Ok in this case we should also update the README.md testing section to say that the user should install pip install ".[tests]" before running tests

sherifnada · 2021-04-19T21:08:56Z

airbyte-integrations/connector-templates/source-singer/README.md.hbs

@@ -99,6 +99,10 @@ docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files
 1. To run additional integration tests, create a directory `integration_tests` which contain your tests and run them with `pytest integration_tests`.
   Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.

+### Standard Tests
+1. Update standard_test_config.yml file to disable/repeat or customize behaviour/inputs of specific tests.


same comment as above

…nnectors.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

airbyte-integrations/connector-templates/source-python/acceptance-test-config.yml.hbs

airbyte-integrations/connector-templates/source-python/acceptance-test.sh

airbyte-integrations/connector-templates/source-python/README.md.hbs

docs/contributing-to-airbyte/building-new-connector/testing-connectors.md

sherifnada · 2021-04-22T20:46:16Z

docs/contributing-to-airbyte/building-new-connector/testing-connectors.md

+* individual tests can be disabled in the config file
+* individual tests can have different inputs and run multiple time with different sets of inputs
+* each test has a timeout
+* each test has stores all inputs and outputs 


Suggested change

* each test has stores all inputs and outputs

* each test stores all inputs and outputs, which can be used for debugging failures

docs/contributing-to-airbyte/building-new-connector/testing-connectors.md

airbyte-integrations/connector-templates/source-python/build.gradle.hbs

airbyte-integrations/connector-templates/source-python/README.md.hbs

+ improve plugin behaviour, enable it explicitly + add acceptance.py to run acceptance tests together with integration tests + remove helper acceptance-test-python.sh + fix gradle command for acceptance tests + update docs

…-tests/update-template

auto-assign bot requested review from ChristopheDuong and jrhizor April 18, 2021 18:56

keu requested review from sherifnada, vitaliizazmic and yevhenii-ldv and removed request for jrhizor and ChristopheDuong April 18, 2021 18:57

keu added the zazmic label Apr 18, 2021

keu linked an issue Apr 18, 2021 that may be closed by this pull request

Standard Tests: Use by default in generated templates #2185

Closed

sherifnada suggested changes Apr 18, 2021

View reviewed changes

keu force-pushed the keu/standard-python-tests-gradle-cmd branch 2 times, most recently from 2243c25 to eae41d8 Compare April 19, 2021 18:06

Base automatically changed from keu/standard-python-tests-gradle-cmd to keu/standard-python-tests April 19, 2021 18:08

keu force-pushed the keu/standard-python-tests-update-template branch from d5c6249 to 8a534bc Compare April 19, 2021 18:35

sherifnada reviewed Apr 19, 2021

View reviewed changes

airbyte-integrations/connector-templates/source-python/sample_files/invalid_config.json Outdated Show resolved Hide resolved

keu requested a review from sherifnada April 19, 2021 19:53

sherifnada approved these changes Apr 19, 2021

View reviewed changes

Base automatically changed from keu/standard-python-tests to master April 20, 2021 19:26

eugene-kulak and others added 5 commits April 22, 2021 16:43

update template for python sources

9881e5e

update templates and their docs

037d4b9

update docs, rename folders, add acceptance script

9793cf5

update docs

da1d5b5

Update docs/contributing-to-airbyte/building-new-connector/testing-co…

786f883

…nnectors.md Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

keu force-pushed the keu/standard-python-tests-update-template branch from b8fb533 to 786f883 Compare April 22, 2021 13:44

update docs in templates and building-new-connector

64cf867

keu requested a review from sherifnada April 22, 2021 16:25

keu mentioned this pull request Apr 22, 2021

Standard Tests: Update docs #2182

Closed

sherifnada suggested changes Apr 22, 2021

View reviewed changes

address comments from @sherifnada

e050737

keu requested a review from sherifnada April 27, 2021 16:53

spelling

e8dd3cc

sherifnada approved these changes Apr 28, 2021

View reviewed changes

eugene-kulak added 5 commits April 28, 2021 22:41

adding hooks for custom setup and teardown

e2074fd

+ improve plugin behaviour, enable it explicitly + add acceptance.py to run acceptance tests together with integration tests + remove helper acceptance-test-python.sh + fix gradle command for acceptance tests + update docs

fix pytest_load_initial_conftests hook

6a3a8a7

disable standard tests that fails for hubspot, so CI can pass

476809c

fix headers

f442e4e

format

dc689ab

keu force-pushed the keu/standard-python-tests-update-template branch from b3fa792 to dc689ab Compare April 28, 2021 20:17

eugene-kulak added 5 commits April 28, 2021 23:21

format

4271f6b

fix source-acceptance install

40fb936

fix source-acceptance-test Dockerfile

33d222c

Merge remote-tracking branch 'origin/master' into keu/standard-python…

b611263

…-tests/update-template

license header

bbe8dd2

keu merged commit 88b77aa into master Apr 28, 2021

keu deleted the keu/standard-python-tests-update-template branch April 28, 2021 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standard tests: update template #2937

Standard tests: update template #2937

keu commented Apr 18, 2021

sherifnada left a comment •

edited

sherifnada left a comment

sherifnada Apr 19, 2021

sherifnada Apr 19, 2021

sherifnada Apr 19, 2021

sherifnada Apr 19, 2021

keu Apr 19, 2021

sherifnada Apr 19, 2021

sherifnada Apr 19, 2021

sherifnada Apr 22, 2021

	1. Update standard_test_config.yml file to disable/repeat or customize behaviour/inputs of specific tests.
	1. Update standard_test_config.yml file to configure tests. See (standard testing)[some-link] for more information.

	* each test has stores all inputs and outputs
	* each test stores all inputs and outputs, which can be used for debugging failures

Standard tests: update template #2937

Standard tests: update template #2937

Conversation

keu commented Apr 18, 2021

What

How

Pre-merge Checklist

Recommended reading order

sherifnada left a comment • edited

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada left a comment •

edited