Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add progress bar support for apptainer push/pull for oras protocol #1587

Merged
merged 1 commit into from
Aug 7, 2023

Conversation

JasonYangShadow
Copy link
Member

@JasonYangShadow JasonYangShadow commented Jul 27, 2023

Description of the Pull Request (PR):

add progress bar support for apptainer push/pull for oras protocol

This fixes or addresses the following GitHub issues:

Before submitting a PR, make sure you have done the following:

Test

Apptainer Push

push

Apptainer Pull

pull

@JasonYangShadow JasonYangShadow changed the title add progress bar support for apptainer push for oras protocol [WIP]add progress bar support for apptainer push for oras protocol Jul 27, 2023
@JasonYangShadow JasonYangShadow changed the title [WIP]add progress bar support for apptainer push for oras protocol add progress bar support for apptainer push/pull for oras protocol Jul 27, 2023
@panda1100
Copy link
Contributor

panda1100 commented Jul 27, 2023

I tested push/pull and both works great! Thank you @JasonYangShadow !

@DrDaveD
Copy link
Contributor

DrDaveD commented Jul 27, 2023

I'm surprised this has a whole new source file just for the oras protocol. Aren't there progress bars on other protocols that can share code? Or is this the first one and it should be extended to others? Why should we treat the oras protocol specially?

@panda1100
Copy link
Contributor

panda1100 commented Jul 27, 2023

@DrDaveD -san,
This is not the first progress bar implementation. docker:// pull and library:// push/pull has progress bar. Each has its own implementation.
The reason we need unique implementation for oras "as well", as my understanding from discussing with @JasonYangShadow and @cclerget about this PR design, we need to access reader/writer buffer to track progress to implement progress bar, to access those buffer we need to control over the reader/writer returned by Push/Fetch method those are under oras library (those are not exposed as callable function?). oras library itself doesn't have progress bar implementation yet. so, we need to use those interfaces directly from Apptainer. This is the reason we decided to implement this way.

Please feel free to follow up or to correct me @JasonYangShadow and @cclerget .

@DrDaveD DrDaveD added this to the 1.2.3 milestone Jul 27, 2023
@DrDaveD
Copy link
Contributor

DrDaveD commented Jul 27, 2023

The reason we need unique implementation for oras "as well", as my understanding from discussing with @JasonYangShadow and @cclerget about this PR design, we need to access reader/writer buffer to track progress to implement progress bar, to access those buffer we need to control over the reader/writer returned by Push/Fetch method those are under oras library (those are not exposed as callable function?).

Are you saying the code can't be refactored to share at least the visual part of the progress bar implementation? That's not good for maintenance, because one implementation might get changed to improve something and the others forgotten.

@panda1100
Copy link
Contributor

Thank you @DrDaveD -san, I see your point. I'll discuss with @JasonYangShadow about it tomorrow morning.

@JasonYangShadow
Copy link
Member Author

I will take a look at current progressbar and mpb related implementation and see if we can reuse mpb.

@JasonYangShadow JasonYangShadow force-pushed the issue/437 branch 3 times, most recently from bfaa332 to e3ffe34 Compare August 1, 2023 08:19
@JasonYangShadow
Copy link
Member Author

@panda1100 I have rewritten the function using apptainer existing progressbar, the same we used previously. Please help verify and check if it is okay.

@panda1100
Copy link
Contributor

@JasonYangShadow Thank you Jason! I did test on my end and works good for both push and pull 👍

Copy link
Contributor

@DrDaveD DrDaveD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an improvement. However I still don't like that there's a new function called getOrasResolver() along with an old function called getResolver() with the same parameters and only for oras, in fact in oras.go. It also looks like they're mostly identical code too.

I suggest renaming getOrasResolver to getResolver and adding a flag that makes it optionally behave like the existing getResolver, then remove the current getResolver.

@JasonYangShadow
Copy link
Member Author

This is an improvement. However I still don't like that there's a new function called getOrasResolver() along with an old function called getResolver() with the same parameters and only for oras, in fact in oras.go. It also looks like they're mostly identical code too.

I suggest renaming getOrasResolver to getResolver and adding a flag that makes it optionally behave like the existing getResolver, then remove the current getResolver.

Thanks for the comments, next time I will be more serious on the duplicated code.

internal/pkg/client/oras/resolver.go Show resolved Hide resolved
internal/pkg/client/oras/oras.go Outdated Show resolved Hide resolved
Signed-off-by: jason yang <jasonyangshadow@gmail.com>
Copy link
Contributor

@DrDaveD DrDaveD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also cherry-pick to release-1.2

@DrDaveD DrDaveD merged commit 6076cef into apptainer:main Aug 7, 2023
17 checks passed
@panda1100
Copy link
Contributor

Thank you DrDave-san and Jason!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: progress indicator during apptainer push
3 participants