airbyte-lib: Add testing to connectors

[flash1293]

Building on the airbyte-lib-validate-source command to validate whether a python connector works with airbyte-lib, this PR introduces a new optional test step to run this command with the current connector under test.

To test:

• Install dev airbyte-ci tools via make tools.airbyte-ci-dev.install
• Run tests for connector where they will work: airbyte-ci connectors --name=source-apify-dataset test

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Feb 5, 2024 4:09pm

[alafanechere]

This code looks good to me and I like the fact that the validation happens in the airbyte-lib package.
I'm a bit more concern by the footprint of this new check on the connector tests in CI.
Could we make the following part of the validation optional to not overlap the test with CAT parts?

    * Answers according to the Airbyte protocol when called with spec, check, discover and read.

[flash1293]

@alafanechere Maybe this is worded badly - what it means to say is that airbyte-lib can use the connector as expected, which includes getting the spec to statically validate the config the user has provided, running check to fully validate it, and running discover and read to actually fetch data and give it to the user.

The check is not doing any protocol-specific checks, it's just trying to use airbyte-lib with the connector:

airbyte/airbyte-lib/airbyte_lib/validate.py

Line 60 in f95e0c8

source = ab.get_connector(

When you say footprint, are you talking about test runtimes? Because this is relatively lightweight in this sense, it calls spec, check, discover and read only once and stops after the very first record.

The reason why I would like to actually call read in this test is that some connectors do things only if a read is actually triggered (e.g. lazily loading some modules or static files) and I want to validate that this works correctly. If airbyte-lib managed to get a record from a single stream out of the connector, there is a very good chance the other ones will work too

[flash1293]

For example for pokeapi the validation took only 13s (and other connectors shouldn't be much slower as it will stop as soon as it has its record):

[alafanechere]

The reason why I would like to actually call read in this test is that some connectors do things only if a read is actually triggered (e.g. lazily loading some modules or static files) and I want to validate that this works correctly. If airbyte-lib managed to get a record from a single stream out of the connector, there is a very good chance the other ones will work too

This makes sense to me, and I'm 🆗 with it if it's not significantly slowing down the full test suite. One concern remains: calling read or discover on a connector with short lived access token means these operation can lead to an update of the access token that has to be reflected on the GSM side. Is your validation code performing this update?

[flash1293]

@alafanechere good point, I changed this PR to only implement the basic check of "install works and spec can be fetched" for now. I will create an issue to track the more complete setup

[flash1293]

The failing test looks unrelated to the changes to me (and also passes locally) - maybe it's flaky?

[alafanechere]

Thanks for the change.
Would you mind manually testing it on a couple of connectors by dispatching this workflow?

[flash1293]

Pokeapi: https://github.com/airbytehq/airbyte/actions/runs/7786827008
Google Sheets: https://github.com/airbytehq/airbyte/actions/runs/7786834735
Facebook marketing: https://github.com/airbytehq/airbyte/actions/runs/7786843721
Shopify: https://github.com/airbytehq/airbyte/actions/runs/7786856167

[flash1293]

Second try: https://github.com/airbytehq/airbyte/actions/runs/7787418433

[flash1293]

Ran tests for a bunch of connectors (see above) and the airbyte-lib validation passed for all of them, merging and crafting a message to connector devs so they are aware