Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AirbyteLib: More robust error handling, installation improvements #34572

Merged
merged 30 commits into from
Jan 30, 2024

Conversation

aaronsteers
Copy link
Collaborator

@aaronsteers aaronsteers commented Jan 26, 2024

This does a few things:

  • Remove -e (editable) flag from install. Users can still get an editable install by using -e ./path/to/my-connector in pip_url. (This was done because -e seemed to fail when used with installations that were not from local paths.)
  • Improve the stderr output from pip when printing installation exceptions.
  • Clean up a failed installation so users can try again.
  • Move certain checks out of the constructors (or make optional), so we can still return an object for the user to use in debugging.
  • Solve some issues where running tests in parallel would bleed into each other tests and/or create race conditions.

Copy link

vercel bot commented Jan 26, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Jan 30, 2024 5:58am

Copy link
Collaborator Author

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unable to debug a new notebook I was creating for GitHub analytics. This PR addresses some of those challenges I ran into.

@aaronsteers aaronsteers marked this pull request as ready for review January 27, 2024 04:28
@octavia-squidington-iii octavia-squidington-iii removed the area/connectors Connector related issues label Jan 28, 2024
@aaronsteers aaronsteers changed the title AirbyteLib: More rebust error handling in installation AirbyteLib: More robust error handling, installation improvements Jan 28, 2024
Copy link
Contributor

@flash1293 flash1293 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for these changes, I agree with most of it, left some nit and one question.

@@ -185,7 +193,7 @@ class AirbyteConnectorError(AirbyteError):


class AirbyteConnectorNotFoundError(AirbyteConnectorError):
"""Connector not found."""
"""Connector name not found in registry."""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at the code, by my understanding AirbyteConnectorNotFoundError means the connector is not found locally, and AirbyteConnectorNotRegisteredError means the connector is not found in the registry, but this docstring indicates otherwise.

Could you clarify?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! This was a mistake. I've renamed the error to AirbyteConnectorExecutableNotFoundError so it is more explicit.

@@ -12,47 +14,81 @@
from airbyte_lib.version import get_version


__cache: dict[str, ConnectorMetadata] | None = None
_cache_lock = threading.Lock()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like this lock is never actually aquired? Am I missing something?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I had it in an earlier version when debugging a race condition, then refactored so it wasn't needed.

(Now removed.)

self,
config: dict[str, Any],
*,
validate: bool = False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the advantage of deferring the check? I thought of it being quite nice as it will tell you as early as possible if your config won't work, instead of waiting for actually invoking.

What's the workflow you had in mind here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed in DM.

@aaronsteers aaronsteers enabled auto-merge (squash) January 30, 2024 05:59
@aaronsteers aaronsteers merged commit 29dbb0c into master Jan 30, 2024
17 checks passed
@aaronsteers aaronsteers deleted the aj/airbyte-lib/install-failure-handling branch January 30, 2024 06:07
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants