Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

airbyte-lib: Lightweight validation #34475

Merged
merged 3 commits into from
Jan 26, 2024
Merged

Conversation

flash1293
Copy link
Contributor

To enable us to publish the long tail of python connectors without running the full test suite, this PR prepares the most basic smoketest:

  • Try to install the connector in a venv
  • See whether it returns the spec correctly

This will give us enough confidence for the initial publish as it eliminates a lot of error classes that render the connector completely unusable.

There are still some scenarios that could lead to broken connectors being shipped like this (e.g. lazily loaded modules cause an error, API changes rendered the connector useless), but this risk is accepted.

Drive-by change

I noticed an issue with airbyte_messages_to_record_dicts - its types state that it returns a iterator of dicts, but it will return none if an emitted message is not a record message. I fixed this by not emitting anything in this case (this broke the validation logic if the first airbyte message returned from the connector is not a record).

Copy link

vercel bot commented Jan 24, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Jan 24, 2024 0:16am

Copy link
Collaborator

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Approved for my part.

Long-term, I'd love for these to live inside CAT, but importantly that probably requires (1) that CAT be more portable and (2) that CAT has options to run a subset of checks. Maybe in another thread, we can discuss the long-term path here. I'm not familiar enough with CAT to make any suggestions or know what is feasible there.

@flash1293
Copy link
Contributor Author

I have a PR to integrate it into airbyte-ci so it can be run in an automated way: #34044. CAT is pretty hard-wired on docker, that's why it's difficult that way.

@flash1293 flash1293 merged commit 3f2ede5 into master Jan 26, 2024
25 checks passed
@flash1293 flash1293 deleted the flash1293/lightweight-validation branch January 26, 2024 08:20
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants