-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added support for progress bars #66
Conversation
Code used to run: print("Hello World") for i in range(5):
print(i) from tqdm import tqdm
import time
my_list = list(range(100))
with tqdm(total=len(my_list)) as pbar:
for x in my_list:
time.sleep(0.1)
pbar.update(1)
if x%20==0:
print(x)
print("Hello World")
print('Hello World 2') Issues:
Any suggestions on how to resolve this issue? |
Why don't you just output it as is? |
I think for this PR, let's just make the output of ploomber-engine be as close as possible to papermill. Based on the discussion, it sounds like we just need to add 1) cell separators 2) Displaying standard error 2) flushing standard output/error Once that's in, we can tackle updating the progress bar inline. |
|
@mehtamohit013 check if you can ask for review now? I've updated the permissions |
cell separators: https://user-images.githubusercontent.com/43580047/225945963-93552b85-83ef-48ae-9993-fd9acf5ba1b6.png Executing cell X ok so if stderr is displayed and we cannot flush output at the moment, I think what's missing is to fix the CI, right? |
So how are we going to show the output as it comes? I think that's essentially the main difference we have to solve it. |
The problem lies with the fact that I cannot make a parser just only for
No,
I think it is because of def patch_sys_std_out_err(display_output):
"""Path sys.{stout, sterr} to capture output"""
# keep a reference to the system ones
stdout, stderr = sys.stdout, sys.stderr
# patch them
stdout_stream = IO(default = stdout,std_type='out', display = display_output)
stderr_stream = IO(default = stderr, std_type='err')
sys.stdout, sys.stderr = stdout_stream, stderr_stream
try:
yield stdout_stream, stderr_stream
finally:
# revert
sys.stdout, sys.stderr = stdout, stderr class IO(StringIO):
def __init__(self, default,std_type,display = True, newline="\n", initial_value="" ):
super().__init__(initial_value=initial_value, newline=newline)
self.default = default
self.std_type = std_type
self.display = display
self._values = []
def write(self, s):
self._values.append(s)
if self.display:
self.default.write(s)
super().write(s)
def get_separated_values(self):
return self._values[:] |
ok, so then the only "easy" thing we can do at the moment are the cell separators right? I'd suggest opening a PR for that, we merge, make a release. and then we investigate this flushing issue |
Yeah I think you're right, I also assume it works fine: |
😂😂
I guess it's a nice to have, we should open an issue just for this since
it's probably more than a quick fix!
…On Sat, Mar 18, 2023, 11:24 Mohit Mehta ***@***.***> wrote:
Yeah I think you're right, I also assume it works fine: tqdm + papermill?
No, it doesn't 😅
Ours is messy but tqdm progress bar is updated inplace and we are
registering \r also.
The robust solution is to make a custom wrapper with tqdm
Below is the output with papermill:
[image: Screenshot_2023-03-18_11-19-45]
<https://user-images.githubusercontent.com/43580047/226114862-00b101a2-7108-4a19-a434-868a50add0e0.png>
—
Reply to this email directly, view it on GitHub
<#66 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACYPJONQ76Z2PSL5TPHTR7TW4XHTRANCNFSM6AAAAAAV7C4ZNU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
@edublancas @idomic |
I think it's a good temporary solution (might be a bit confusing for the
users)
…On Sat, Mar 18, 2023, 11:36 Mohit Mehta ***@***.***> wrote:
@edublancas <https://github.com/edublancas> @idomic
<https://github.com/idomic>
If you want a quick fix, how about this: Just some extra \n but the
progress bar is working
[image: Screenshot_2023-03-18_11-34-42]
<https://user-images.githubusercontent.com/43580047/226115554-8881b4eb-061d-46ea-be46-bc76ede3861e.png>
—
Reply to this email directly, view it on GitHub
<#66 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACYPJONHOKDS627HF4MYSXDW4XJAVANCNFSM6AAAAAAV7C4ZNU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Implemented this one |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good besides the leading \n .
It doesn't closes ploomber/ploomber#1080 since we're still using papermill there. |
Describe your changes
Added support for flushing out stdout and stderr
Issue ticket number and link
Closes #61 #1080
Checklist before requesting a review
📚 Documentation preview 📚: https://ploomber-engine--66.org.readthedocs.build/en/66/