Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use python package if available #15

Merged
merged 9 commits into from
Mar 30, 2024
Merged

Conversation

pnadolny13
Copy link
Contributor

@pnadolny13 pnadolny13 commented Mar 15, 2024

This is a first iteration at installing and running the pip installable packages instead of using docker when theyre available, otherwise continue using docker. The main challenge with running airbyte sources with meltano is that they require docker to be available which makes the normal deployment routes more difficult (i.e. needed docker in docker support).

I found this was the simplest approach since the alternative options require installing the source python package at meltano install time which is hard to do. I attempted to do that with a custom install script but it was hacker than this approach and would require us to determine which sources are pip installable and update the hub which is a pain.

It simply checks if the package is pip installable via the registry endpoint that airbyte provides and installs it in a new virtualenv if it is.

@pnadolny13 pnadolny13 marked this pull request as ready for review March 18, 2024 13:37
@z3z1ma
Copy link
Collaborator

z3z1ma commented Mar 30, 2024

LGTM, we can probably unify the command construction instead of the if ... pip_installable ternary with separate lists. But its insignificant enough that it doesn't matter.

@z3z1ma z3z1ma merged commit 87da181 into main Mar 30, 2024
1 of 4 checks passed
@z3z1ma z3z1ma deleted the use_python_package_if_available branch March 30, 2024 04:36
@z3z1ma
Copy link
Collaborator

z3z1ma commented Mar 30, 2024

I cleaned up the code in this commit
@pnadolny13 e1349aa#diff-ed817cab065e4c6d520120485629589b6e09dc6c0eec42adfc9d27d6f796d998

Let me know if it works for you as expected. Airbyte CDK is apparently broken on py311 (acknowledged since December 🙄)

@pnadolny13
Copy link
Contributor Author

@z3z1ma looks great! Thanks for getting it across the finish line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants