-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CAT tools/acceptance_test_config_migration: Make relatively configurable per migration, add (hacky) ability to run CAT tests locally for multiple connectors #24377
Changes from all commits
f596e3f
c88305e
1f91a59
a8ef276
3e5cf1f
a7eb6f6
2545f8f
356b12a
a53c31a
c82879e
646b372
fa72e00
be92aef
96eb3fc
9fc650c
ede50bd
cea154a
a2569e4
63a273b
f729dc2
1aa9644
6535110
9de27c3
ae1c263
b538a8c
358b0f8
f42a3a9
abe4006
f61804d
7655714
b460b1c
532a35e
c498f9f
6a17ba1
92bf491
8d9cfa7
e739e12
fb06bd6
6091f69
9f865fe
a9003ed
a487f30
1827e5d
027fd23
18b9c10
20f0c96
022ce66
7876cdf
c603e9f
73cc08e
b8364d1
99d1100
4265ff8
fc3fa2b
42da5f1
d0896c5
b1d91fe
1a6157d
5858e4d
be8b8e2
ba0b757
94bbf12
46b9c5e
66446be
887f2dc
6bfeb69
7fe82b0
d05404e
7d9fb14
3aa0d33
511476b
eefc8d2
330b59c
9e7d6ce
a328ec1
dc8d20a
5ba051c
7c88cb1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,296 @@ | ||
# Tooling for automated migration of `acceptance-test-config.yml` files | ||
|
||
This directory contains scripts that can help us manage the migration of connectors' `acceptance-test-config.yml` files. | ||
|
||
## Setup | ||
Before running these scripts you need to set up a local virtual environment in the **current directory**: | ||
```bash | ||
python -m venv .venv | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One small idea / nit here. What are your thoughts on using poetry here over venv + pip? It would allow us to reduce the number of steps to install and at the same time remove the whole "whoops I forgot to activate my venv" problem
This is a tool weve been using with the Dagger and Metadata work and so far its been working well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have no preference either way since this is "Connector ops" code - happy to switch over! Especially if in the future people won't be doing much running of the scripts of their own (Ideally there's only one command to perform the testing, migrating failed configs, opening issues, etc.) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It shouldn't be too big of a lift. But very optional, up to you in terms of how much other work you have on your plate :) |
||
source .venv/bin/activate | ||
pip install -r requirements.txt | ||
brew install gh | ||
``` | ||
|
||
Then create a module to contain the information for your migration/issues etc.: | ||
``` | ||
mkdir migrations/<migration_name> | ||
touch migrations/<migraton_name>/__init__.py | ||
touch migrations/<migration_name>/config.py | ||
``` | ||
|
||
Copy a config.py file from another migration and fill in the `MODULE_NAME` variable. The other variables | ||
can be filled in when you use certain scripts. | ||
```python | ||
# | ||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
from typing import Optional, List | ||
|
||
# SET THESE BEFORE USING THE SCRIPT | ||
MODULE_NAME: str = "<migration_name>" | ||
# GITHUB_PROJECT_NAME: Optional[str] = None | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why are these commented out? but not module name? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The others are only used for certain scripts, e.g. you don't need to define them to run There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
# COMMON_ISSUE_LABELS: List[str] = ["area/connectors", "team/connectors-python", "type/enhancement", "column-selection-sources"] | ||
# ISSUE_TITLE: str = "Add undeclared columns to spec" | ||
``` | ||
|
||
## Scripts | ||
|
||
The scripts perform operations on a set of connectors. | ||
|
||
### `run_tests.py`: Run CAT tests and save results by exit code | ||
|
||
#### What it does | ||
|
||
The script will run CAT on all connectors and report the results to `migrations/<migration_name>/output` and classify the | ||
results by exit code. | ||
|
||
TODO: Replace this process with Dagger | ||
|
||
#### Before running | ||
|
||
1. The tests will run on the `latest` version of CAT by default. To run the `dev` version of CAT, and select a specific | ||
test, commit the hacky changes in [this commit](https://github.com/airbytehq/airbyte/pull/24377/commits/7d9fb1414911a512cd5d5ffafe2a384e8004fb1e). | ||
erohmensing marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
2. Give Docker a _lot_ of space to build all the connector images! | ||
|
||
3. Make sure you have the secrets downloaded from GSM for all of the connectors you want to run tests on. Please keep | ||
in mind that secrets need to be re-uploaded for connectors with single-use Oauth tokens. | ||
|
||
#### How to run | ||
|
||
Typical usage: | ||
``` | ||
python run_tests.py | ||
``` | ||
|
||
Full options: | ||
``` | ||
usage: run_tests.py [-h] --connectors [CONNECTORS ...] [--allow_alpha | --no-allow_alpha] [--allow_beta | --no-allow_beta] [--max_concurrency MAX_CONCURRENCY] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also is there any benefit to wrapping this in the click library that augustin used in QA checks? |
||
|
||
Run connector acceptance tests for a list of connectors. | ||
|
||
options: | ||
-h, --help show this help message and exit | ||
--connectors [CONNECTORS ...] | ||
A list of connectors (separated by spaces) to run a script on. (default: all connectors) | ||
--allow_alpha, --no-allow_alpha | ||
Whether to apply the change to alpha connectors, if they are included in the list of connectors. (default: False) | ||
--allow_beta, --no-allow_beta | ||
Whether to apply the change to bets connectors, if they are included in the list of connectors. (default: False) | ||
--max_concurrency MAX_CONCURRENCY | ||
The maximum number of acceptance tests that should happen at once. | ||
``` | ||
|
||
### `create_issues.py`: Create issues in bulk | ||
|
||
#### What it does: | ||
For each connector: | ||
1. Generate an issue content (title, body, labels, project), using an `issue.md.j2` template | ||
2. Find an already existing issue with the same title, if one already exists | ||
3. Create the issue and return its url if it does not exist. | ||
|
||
Issues get created with the title according to `ISSUE_TITLE`. Labels are added according to `COMMON_ISSUE_LABELS`. Issues are added to the `GITHUB_PROJECT_NAME` project, if one is provided. | ||
|
||
#### Before running | ||
|
||
1. Update your config file to define the following variables: | ||
|
||
```python | ||
# | ||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
from typing import Optional, List | ||
|
||
# SET THESE BEFORE USING THE SCRIPT | ||
MODULE_NAME: str = "<migration_name>" | ||
GITHUB_PROJECT_NAME: Optional[str] = "<project_name>" | ||
erohmensing marked this conversation as resolved.
Show resolved
Hide resolved
|
||
COMMON_ISSUE_LABELS: List[str] = ["<label_1>", "<label_2>", "..."] | ||
ISSUE_TITLE: str = "<issue_title>" | ||
``` | ||
Note that `ISSUE_TITLE` will be prepended with `Source <source name>:` in the actual created issue. | ||
|
||
2. Create a template for your issue: | ||
|
||
```bash | ||
touch migrations/<migration_name>/issue.md.j2 | ||
``` | ||
|
||
If you need to fill more variables than are currently defined in the call to `template.render()` | ||
in `create_issues.py`, edit the script to allow filling of that variable and define how it should be | ||
filled. Please keep in mind the other migrations when you do this. | ||
|
||
3. Update the following line in the script so that it points to the config file from your migration: | ||
|
||
```python | ||
## Update this line before running the script | ||
from migrations.<migration_name> import config | ||
``` | ||
|
||
#### How to run: | ||
|
||
Typical usage (dry run): | ||
``` | ||
python create_issues.py --connectors <connectors> | ||
``` | ||
|
||
Typical usage (real execution): | ||
``` | ||
python create_issues.py --connectors <connectors> --no-dry | ||
``` | ||
|
||
Full options: | ||
``` | ||
usage: create_issues.py [-h] [-d | --dry | --no-dry] --connectors [CONNECTORS ...] [--allow_beta | --no-allow_beta] [--allow_alpha | --no-allow_alpha] | ||
|
||
Create issues for a list of connectors from a template. | ||
|
||
options: | ||
-h, --help show this help message and exit | ||
-d, --dry, --no-dry Whether the action performed is a dry run. In the case of a dry run, no git actions will be pushed to the remote. (default: True) | ||
--connectors [CONNECTORS ...] | ||
A list of connectors (separated by spaces) to run a script on. (default: all connectors) | ||
--allow_beta, --no-allow_beta | ||
Whether to apply the change to bets connectors, if they are included in the list of connectors. (default: False) | ||
--allow_alpha, --no-allow_alpha | ||
Whether to apply the change to alpha connectors, if they are included in the list of connectors. (default: False) | ||
``` | ||
|
||
|
||
### `config_migration.py`: Perform migrations on `acceptance-test-config.yml` files | ||
|
||
#### What it does: | ||
For each connector: | ||
1. Load the connector's `acceptance-test-config.yml` | ||
2. Migrate the connector to the new format if this option was chosen | ||
3. Apply the given migration to the `acceptance-test-config.yml` file | ||
|
||
Note that all changes happen on the working branch. | ||
|
||
#### Before running | ||
|
||
1. Create a method in `config_migration.py` to perform your migration on a given config. For this, | ||
You can take inspiration from the existing `set_high_strictness_level` and `set_ignore_extra_columns` | ||
migration methods. | ||
|
||
2. Update the following line to point to your new migration: | ||
```python | ||
# Update this before running the script | ||
MIGRATION_TO_RUN = <migration_method> | ||
``` | ||
|
||
#### How to run: | ||
|
||
Typical usage: | ||
``` | ||
python config_migration.py --connectors <connectors> | ||
``` | ||
|
||
Full options: | ||
``` | ||
usage: config_migration.py [-h] --connectors [CONNECTORS ...] [--allow_alpha | --no-allow_alpha] [--allow_beta | --no-allow_beta] [--migrate_from_legacy | --no-migrate_from_legacy] | ||
|
||
Migrate acceptance-test-config.yml files for a list of connectors. | ||
|
||
options: | ||
-h, --help show this help message and exit | ||
--connectors [CONNECTORS ...] | ||
A list of connectors (separated by spaces) to run a script on. (default: all connectors) | ||
--allow_alpha, --no-allow_alpha | ||
Whether to apply the change to alpha connectors, if they are included in the list of connectors. (default: False) | ||
--allow_beta, --no-allow_beta | ||
Whether to apply the change to bets connectors, if they are included in the list of connectors. (default: False) | ||
--migrate_from_legacy, --no-migrate_from_legacy | ||
Whether to migrate config files from the legacy format before applying the migration. (default: False) | ||
``` | ||
|
||
|
||
### `create_prs.py`: Create a PR per connector that performs a config migration and pushes it | ||
|
||
## Create migration PRs for GA connectors (`create_prs.py`) | ||
|
||
#### What it does: | ||
For each connector: | ||
1. Create a branch and check it out | ||
2. Locally apply the migration to `acceptance_test_config.yml` by calling `config_migration.py` | ||
3. Commit and push the changes on this branch | ||
4. Open a PR for this branch | ||
5. Run a connector acceptance test on this branch by posting a `/test` comment on the PR | ||
|
||
An example of the PR it creates can be found [here](https://github.com/airbytehq/airbyte/pull/19136). | ||
|
||
PRs get created with the title according to `ISSUE_TITLE`. Labels are added according to `COMMON_ISSUE_LABELS`. PRs are added to the `GITHUB_PROJECT_NAME` project, if one is provided. | ||
|
||
#### Before running | ||
|
||
1. Update your config file to define the following variables: | ||
|
||
```python | ||
# | ||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
from typing import Optional, List | ||
|
||
# SET THESE BEFORE USING THE SCRIPT | ||
MODULE_NAME: str = "<migration_name>" | ||
GITHUB_PROJECT_NAME: Optional[str] = "<project_name>" | ||
COMMON_ISSUE_LABELS: List[str] = ["<label_1>", "<label_2>", "..."] | ||
ISSUE_TITLE: str = "<issue_title>" | ||
``` | ||
Note that `ISSUE_TITLE` will be prepended with `Source <source name>:` in the actual created issue. | ||
|
||
2. Create a template for your PR description: | ||
|
||
```bash | ||
touch migrations/<migration_name>/pr.md.j2 | ||
``` | ||
|
||
If you need to fill more variables than are currently defined in the call to `template.render()` | ||
in `create_pr.py`, edit the script to allow filling of that variable and define how it should be | ||
filled. Please keep in mind the other migrations when you do this. | ||
|
||
3. Update the following line in the `create_prs.py` so that it points to the config file from your migration: | ||
|
||
```python | ||
## Update this line before running the script | ||
from migrations.<migration_name> import config | ||
``` | ||
|
||
4. Ensure that your current git envronment is clean by (perhaps temorarily) committing changes. | ||
|
||
|
||
#### How to run: | ||
|
||
Typical usage (dry run): | ||
``` | ||
python create_prs.py --connectors <connectors> | ||
``` | ||
|
||
Typical usage (real execution): | ||
``` | ||
python create_prs.py --connectors <connectors> --no-dry | ||
``` | ||
|
||
Full options: | ||
``` | ||
usage: create_prs.py [-h] [-d | --dry | --no-dry] --connectors [CONNECTORS ...] [--allow_alpha | --no-allow_alpha] [--allow_beta | --no-allow_beta] | ||
|
||
Create PRs for a list of connectors from a template. | ||
|
||
options: | ||
-h, --help show this help message and exit | ||
-d, --dry, --no-dry Whether the action performed is a dry run. In the case of a dry run, no git actions will be pushed to the remote. (default: True) | ||
--connectors [CONNECTORS ...] | ||
A list of connectors (separated by spaces) to run a script on. (default: all connectors) | ||
--allow_alpha, --no-allow_alpha | ||
Whether to apply the change to alpha connectors, if they are included in the list of connectors. (default: False) | ||
--allow_beta, --no-allow_beta | ||
Whether to apply the change to bets connectors, if they are included in the list of connectors. (default: False) | ||
``` | ||
|
||
## Existing migrations | ||
* `strictness_level_migration`: Migrates a connector from the old format to the new format, and adds enforcement of high strictness level. | ||
* `fail_on_extra_columns`: Adds `fail_on_extra_columns: false` to connectors which fail the `Additional properties are not allowed` extra column validation. | ||
Supports adding this parameter to configs in the old and new config format. |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Original file: (diff was too large for it to clock a move vs a delete and add) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
# | ||
# Copyright (c) 2023 Airbyte, Inc., all rights reserved. | ||
# | ||
|
||
import argparse | ||
import logging | ||
from pathlib import Path | ||
from typing import Callable | ||
|
||
import definitions | ||
import utils | ||
from connector_acceptance_test.config import Config | ||
from ruamel.yaml import YAML | ||
|
||
yaml = YAML() | ||
yaml.preserve_quotes = True | ||
yaml.width = 150 | ||
|
||
parser = argparse.ArgumentParser(description="Migrate acceptance-test-config.yml files for a list of connectors.") | ||
utils.add_connectors_param(parser) | ||
utils.add_allow_alpha_param(parser) | ||
utils.add_allow_beta_param(parser) | ||
parser.add_argument( | ||
"--migrate_from_legacy", | ||
action=argparse.BooleanOptionalAction, | ||
default=False, | ||
help="Whether to migrate config files from the legacy format before applying the migration.", | ||
) | ||
|
||
|
||
def load_config(config_path: Path) -> Config: | ||
with open(config_path, "r") as file: | ||
config = yaml.load(file) | ||
return config | ||
|
||
|
||
def migrate_to_new_config_format(config: Config): | ||
if Config.is_legacy(config): | ||
return Config.migrate_legacy_to_current_config(config) | ||
else: | ||
logging.warning("The configuration is not in a legacy format.") | ||
return config | ||
|
||
|
||
def set_high_test_strictness_level(config): | ||
if Config.is_legacy(config): | ||
raise Exception("You can't set a strictness level on a legacy config. Please use the `--migrate_from_legacy` flag.") | ||
config["test_strictness_level"] = "high" | ||
for basic_read_test in config["acceptance_tests"].get("basic_read", {"tests": []})["tests"]: | ||
basic_read_test.pop("configured_catalog_path", None) | ||
return config | ||
|
||
|
||
def set_ignore_extra_columns(config): | ||
if Config.is_legacy(config): | ||
for basic_read_test in config["tests"].get("basic_read"): | ||
basic_read_test["fail_on_extra_columns"] = False | ||
else: | ||
for basic_read_test in config["acceptance_tests"].get("basic_read", {"tests": []})["tests"]: | ||
basic_read_test["fail_on_extra_columns"] = False | ||
return config | ||
|
||
|
||
def write_new_config(new_config, output_path): | ||
with open(output_path, "w") as output_file: | ||
yaml.dump(new_config, output_file) | ||
logging.info("Saved the configuration in its new format") | ||
|
||
|
||
def update_configuration(config_path, migration: Callable, migrate_from_legacy: bool): | ||
config_to_migrate = load_config(config_path) | ||
if migrate_from_legacy: | ||
config_to_migrate = migrate_to_new_config_format(config_to_migrate) | ||
new_config = migration(config_to_migrate) | ||
write_new_config(new_config, config_path) | ||
logging.info(f"The configuration was successfully updated: {config_path}") | ||
return config_path | ||
|
||
|
||
if __name__ == "__main__": | ||
args = parser.parse_args() | ||
|
||
# Update this before running the script | ||
MIGRATION_TO_RUN = set_high_test_strictness_level | ||
Comment on lines
+83
to
+84
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the future, this is something I'd like to both define and set in the config. I.e. in the config, we have a method something like
which we can use via as config.run_migration There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we turn this into an cli argument? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I think so - sort of a combo of "We should be able to import the module via a cli var" and "we should be able to define the migration in the module in a standardized way" would get us there. |
||
|
||
for definition in utils.get_valid_definitions_from_args(args): | ||
config_path = utils.acceptance_test_config_path(definitions.get_airbyte_connector_name_from_definition(definition)) | ||
update_configuration(config_path, MIGRATION_TO_RUN, args.migrate_from_legacy) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Original file:
airbyte-integrations/bases/connector-acceptance-test/tools/strictness_level_migration/README.md
(diff was too large for it to clock a move vs a delete and add)