Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connector Ops: Fix third party support in get_all_connectors_in_repo #31487

Merged
merged 10 commits into from
Oct 19, 2023

Conversation

bnchrch
Copy link
Contributor

@bnchrch bnchrch commented Oct 17, 2023

Reopening of #31166

Problem

Third party pathing breaks get_all_connectors_in_repo and the Connector class

related to a comment from #30456

Solution

  1. Refactor Connector to take a relative path as its init argument
  2. Update technical name to be based off that relative path
  3. Update documentation paths to be based off documentation url
  4. Update type to be based off metadata
  5. Add tests to assert that all connectors returned by get_all_connectors_in_repo have metadata and docs
  6. Fix the appstore documentation path

@vercel
Copy link

vercel bot commented Oct 17, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 19, 2023 0:17am

@bnchrch bnchrch requested a review from a team October 17, 2023 00:35
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 17, 2023
@@ -30,17 +30,17 @@ def test__get_type_and_name_from_technical_name(self, technical_name, expected_t
"connector, exists",
[
(utils.Connector("source-faker"), True),
(utils.Connector("source-notpublished"), False),
(utils.Connector("source-doesnotexist"), False),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alafanechere I think given where the code base is at, we should change this to an error

I.e. if you try to instantiate Connector but we cant find a metadata.yaml file. we throw an Error.

What are your thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in for this but I think this could break a couple of test.
I think that in some test scenario we provision a fake connector, without a metadata (or other file).
Feel free to make this change but please run the pipelines test suite to make sure nothing breaks. (I believe it will run automatically thanks to your recent change on the workflow).

Copy link
Contributor Author

@bnchrch bnchrch Oct 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do! I think ill hold it for a subsequent PR

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Light suggestion. Thanks for the improvements!
And please bump connector_ops version + update poetry.lock of pipelines**

airbyte-ci/connectors/connector_ops/connector_ops/utils.py Outdated Show resolved Hide resolved
airbyte-ci/connectors/connector_ops/connector_ops/utils.py Outdated Show resolved Hide resolved
airbyte-ci/connectors/connector_ops/connector_ops/utils.py Outdated Show resolved Hide resolved
if not self.has_airbyte_docs:
return None

return Path(f"{self.relative_documentation_path_str}.md")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise an error when has_airbyte_docs is True but path does not exists?
I would appreciate that the existence check logic move to this util instead of giving this responsability to the caller. Wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that idea!

if not self.has_airbyte_docs:
return None

return Path(f"{self.relative_documentation_path_str}.inapp.md")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise an error when has_airbyte_docs is True but path does not exists?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im unsure. A hard error would prevent the QA checks from running on a list of connectors as it would fail if just one was missing a documentation file. Is that a large concern?

@bnchrch bnchrch merged commit 6b5a8c2 into master Oct 19, 2023
20 checks passed
@bnchrch bnchrch deleted the bnchrch/fix-third-party-support branch October 19, 2023 00:49
ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants