Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable the flytekit to be pluggable #2039

Merged
merged 19 commits into from
Dec 19, 2023

Conversation

thomasjpfan
Copy link
Member

@thomasjpfan thomasjpfan commented Dec 11, 2023

Alternative to #1941

Why are the changes needed?

This PR makes pyflyte pluggable by external libraries to adjust the CLI commands and update the remote.

What changes were proposed in this pull request?

This PR adds a FlytekitPlugin to have the pluggable CLI API be in one place. For this initial PR, it gives the minimum API surface for configuring the API.

Implementation-wise, the external plugins are loaded through entry points and have the group name: flytekit.configuration.plugin.

How was this patch tested?

  • Existing unit tests were updated to into account the changes.
  • The Plugin API is tested with it's own unit test.

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Copy link

codecov bot commented Dec 11, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (74f2f53) 85.27% compared to head (c645d02) 83.87%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2039      +/-   ##
==========================================
- Coverage   85.27%   83.87%   -1.40%     
==========================================
  Files         274      309      +35     
  Lines       21360    23060    +1700     
  Branches     3481     3489       +8     
==========================================
+ Hits        18214    19342    +1128     
- Misses       2537     3095     +558     
- Partials      609      623      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cosmicBboy
cosmicBboy previously approved these changes Dec 13, 2023
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
@thomasjpfan thomasjpfan changed the title Enable the pyflyte cli to be pluggable Enable the flytekit to be pluggable Dec 13, 2023
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Copy link
Member Author

@thomasjpfan thomasjpfan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the FlytekitPlugin to flytekit.configuration because it is "configuration flytekit".

flytekit/configuration/plugin.py Outdated Show resolved Hide resolved
@thomasjpfan
Copy link
Member Author

@cosmicBboy The monodocs failure in the CI is unrelated to this PR, but looks real.

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
@thomasjpfan
Copy link
Member Author

In 9b90029 I have pytest patch the plugin variable. This way if another plugin gets loaded, the tests still run with the first-party FlytekitPlugin.

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
@thomasjpfan
Copy link
Member Author

I updated the solution to #2039, which I am happy with. This allows pytest to configure the loaded plugin on demand.

Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
flytekit/configuration/plugin.py Outdated Show resolved Hide resolved
flytekit/configuration/plugin.py Outdated Show resolved Hide resolved
flytekit/configuration/plugin.py Show resolved Hide resolved
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
pingsutw
pingsutw previously approved these changes Dec 18, 2023
Copy link
Member

@pingsutw pingsutw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, could you resolve the merge conflict?

@eapolinario eapolinario merged commit a41bd1f into flyteorg:master Dec 19, 2023
76 of 77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants