Standard Tests: create modular standard test suite v2 scaffolding

[keu]

closes #2180

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How this actually works

I tried to keep things simple and at the same time aligned with pytest way of doing things. The latter allow us to override and/or extend behaviour by customizing fixtures in the code if needed, thuse our tests will have less surprises for people familiar with pytest.
To execute test I had two controversial approaches:

• run from standard test package and point to configuration file with connector/test settings
• run from connector's package, i.e. create a file with test and import standard tests from test package

I choice the first approach because it requires less files.

The bread and butter of this framework is fixture. Each test suite has set of inputs, multiple sets will result in parameterized test suite. The fixture called inputs represents such set of inputs, its value depends on which test suits we execute at the moment and the content of the config file.

typical config file:

connector_image: airbyte/source-hubspot:dev
tests:
  core:
    - config_path: "secrets/config.json"
      invalid_config_path: "secrets/invalid_config.json"
      spec_path: "source_hubspot/spec.json"
      configured_catalog_path: "sample_files/configured_catalog.json"
    - config_path: "secrets/config.json"
      invalid_config_path: "secrets/invalid_config.json"
      spec_path: "source_hubspot/spec.json"
      configured_catalog_path: "sample_files/configured_catalog.json"
  incremental: []
  full_refresh: []

Pre-merge Checklist

• Run integration tests
• Publish Docker images

Recommended reading order

1. test.java
2. component.ts
3. the rest

[michel-tricot]

How are we going to inject the version?

[keu]

to be honest I don't know, this way of configuring standard test suite was in original requirements,
@sherifnada please give me a hand here :)

offtopic: it is relatively easy to override tag/version from custom command-option (something like pytest --connector="airbyte/source-hubspot:0.1.0")

[michel-tricot]

When do we move to poetry? :)

[keu]

well, poetry has some issues with dependencies specified as local relative paths (python-poetry/poetry#34)
we need to spend some time to investigate all pros and cons

[michel-tricot]

Should we call it StandardTestBaseInterface ?

[keu]

Test is a verb here.
nevertheless it was renamed to TestCore to align with configuration file

[sherifnada]

Very very exciting! 🎉 a lot of my comments can be addressed in follow up issues/separate PRs. There's a few comments that should be addressed before merging, but it's probably best to start separating new features/requests into different PRs.

[sherifnada]

maybe a clearer name for this file is config_models.py? or put it in a models/configs.py?

[keu]

but, I don't see why models? this is configuration of standard_test package right?

[sherifnada]

right, it's the configuration's model. Model here just means the expected schema. This is not the actual config object. Another name is potentially spec but model is more common

[keu]

but it is an actual config object. The fact that we use pydantic to simplify it deserialization is nice feature. There is no need to have extra class to have a configuration. You can check another typical usage of Config object in pydantic docs. Maybe I'm missing something here (too pythonic way of thinking...dunno), but I being using pydantic models as configs in all my projects without any extra layers on top of it.

[michel-tricot]

I am confused about this config.

It seems every time we need to create a connection test, we need to test for both success and failure.
Why don't we have

config_path: str
success: bool

?

[keu]

updated, success -> status, where status is enum of succeed, failed, exception