Progress Bar

[shyamd]

Is there anyway we could build in a global progress bar for builders that also worked well with logger/input outputs and didn't require complex logic?

[dwinston]

What do you mean by "worked well with logger/input outputs"? We could use (nested) tqdm progress bars, which default to sys.stderr. @tschaume ?

[dwinston]

Noting here that this enhancement could be a step towards @jdagdelen's suggestion of built-in profiling: because tqdm reports seconds per iteration ("s/it"), one can profile Builder.process_item performance this way.

[mkhorton]

Is there a way to do this automatically? I can see if get_items is a list or a cursor it could work, but not if it's a generator, because we wouldn't know the total number of items?

[shyamd]

Yeah, I was thinking of something along the lines of:

self.total_items = 23432
for i in items:
  yield i

And it should build the appropriate tqdm progress bar.
Ideally it should detect what kind of env and use the right one as well.

[mkhorton]

What do you want as a best practice? A yield, a cursor..?

[shyamd]

It shouldn't matter which one and maggma should just work with it. If you give it no total count it should still have progress bar that increments and provides it/s. Basically this needs to be robust to bad and good user builder code. Thanks in advance, Shyam Dwaraknath Materials Research Scientist Lawrence Berkeley National Laboratory

…

On Tue, Apr 24, 2018 at 5:04 PM, Matthew Horton ***@***.***> wrote: What do you want as a best practice? A yield, a cursor..? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#21 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQDC-sdxuzY5E-ZDipxPy48XMd-gBUXUks5tr72QgaJpZM4TZanY> .

[dwinston]

Right. If there's no count, it would still be valuable to see it/s. I think the runner can try to call <obj>.count() and then len(<obj>) on what get_items() returns (wrapping in a try/except) to pass the total= kwarg to the tqdm constructor. I think it'd also be helpful for the runner to look for a Builder.total_items set in the case of yielding.

[shyamd]

Taken care of with #24