Automatic units in monitor for large quantities

[lschmelzeisen]

Would it be possible to support automatic unit conversion, similar to how tqdm implements this?

• unit : str, optional
  String that will be used to define the unit of each iteration [default: it].
• unit_scale : bool or int or float, optional
  If 1 or True, the number of iterations will be reduced/scaled automatically and a metric prefix following the International System of Units standard will be added (kilo, mega, etc.) [default: False]. If any other non-zero number, will scale total and n.

[rsalmei]

Hello @lschmelzeisen!

I'm not sure I understand it, you would like to see numbers divided by 1000, with suffixes of kilo, mega, etc?
And my alive_bar by design does not use "units", it uses "# of things/second" like "800/s", the user should know what that refers to.

So, instead of:

|███████████████████████                 | ▁▃▅ 2881/5000 [58%] in 1s (1527.1/s, eta: 1s)

Are you asking for this?

|███████████████████████                 | ▁▃▅ 2K/5K [58%] in 1s (1K/s, eta: 1s)

Aren't we losing too much information by truncating it all?
Or maybe this would be better, maybe configurable too:

|███████████████████████                 | ▁▃▅ 2.9K/5K [58%] in 1s (1.5K/s, eta: 1s)

If it really is what you're asking, I don't know if it would be that useful, as I'm not comfortable in truncating and reducing information, and also don't think it would be a popular feature for that.
I can actually see one use case, when someone is dealing with very large numbers:

|▊                                       | ▃▁▃ 102783086/5000000000 [2%] in 45s (2260220.3/s, eta: 0:36:07)

which would become:

|▊                                       | ▃▁▃ 102.8M/5.0G [2%] in 45s (2.3M/s, eta: 0:36:07)

Would by any chance be that what you're looking for?

[lschmelzeisen]

Sorry, I imaged you would be familiar with tqdm, so let me elaborate a bit.

The idea of the unit parameter would be to tell the user what each iteration consists of. So if by default the bar looks like this:

|███████████████████████                 | ▁▃▅ 2881/5000 [58%] in 1s (1527.1/s, eta: 1s)

With unit="b" it would look like this:

|███████████████████████                 | ▁▃▅ 2881b/5000b [58%] in 1s (1527.1/s, eta: 1s)

(Of course than the bar would also need to be shortened by two characters to fit into the number of available columns, but I didn't bother to do this in these examples.)

With unit_scale=True the above would then be converted to include SI-Prefixes such that a predefined (or even configurable?) number of significant digits would still be visible. So for 3 significant digits the above would become:

|███████████████████████                 | ▁▃▅ 2.88Kb/5.00Kb [58%] in 1s (1527.1/s, eta: 1s)

(Here is the tqdm implemenation of this scaling.)

I concede that this probably isn't too useful for such small numbers. However, my main use case for progress bars is iterating over large files (up to multiple GBs large). There such an automatic unit scaling behavior is very convenient.

For unit_scale, tqdm further adds the possibility to divide the iteration counters by a custom scaling factor. When displaying this of course still shows the same number of significant digits. I can't comment on how useful this addition is, because I never needed it.

One other way you could look at this would be that currently there is no way to customize the text next to the bar (I think, have only looked at the README not your code). So you could also just add a callback with a default implementation like the following to allow users to implement this themselves if desired:

def callback(cur_iteration, max_iteration, elapsed, speed, eta):
    return "{}/{} [{:.0%}] in {}s ({:.1f}/s, eta: {}s)".format(
        cur_iteration, max_iteration, cur_iteration/max_iteration, elapsed, speed, eta)

which would then be called on each bar update with parameters like this:

callback(2881, 5000, 1, 1527.1, 1)

(Could be that I'm simplifying too much here, as I didn't look at the code yet, sorry.)

[rsalmei]

Hey @lschmelzeisen, thanks for the more detailed explanation.
I do something similar to the unit scaling in my other project https://github.com/rsalmei/about-time, could implement that in here too!

It's just that my plate is too full at this moment, I'm implementing a big refactoring in alive-progress to allow me to write unit tests for the core alive_bar (very important) and a dynamic reconfiguration system, to deliver a new "quantifying mode", which will allow one to calculate the very same total the bar will use (with it already on screen, animated)!
Then it's in my queue:

• support variable width size, listening to changes in terminal size
• enable multiple simultaneous bars, for nested or multiple statuses
• proper support for python logging
• create a contrib system, to allow a simple way to share users' spinners and bars styles
• jupyter notebook support

Well, that's about it, you see that there's several cool things on the horizon, but I'll get there!... 😅

And about the text next to the bar, you sure can customize it. You can send a default one title='Fixing file', or even send what I call a "situational message" in each iteration via the bar(text='Running, please wait').

[lschmelzeisen]

Sure, it's your project and you get to decide where to spent your time. Looking forward to see how this will develop in the future!

[rsalmei]

Hey @lschmelzeisen, how are you man?

Just to let you know I've changed my plate, and I'll implement this for you after 2.0!
I thought it would indeed be cool, so I'll put this in front of all.
I'm just finishing the new readme for 2.0, and a new #19 appearance in spinners. The 2.1 should get units!

[jhonatan-lopes]

Hi @rsalmei, I'm also very interested in this feature (this is the only thing keeping me from migrating some of my projects from tqdm to alive-progress). Let me know if you would like some help. I would be very glad to contribute.

[rsalmei]

Hey @jhonatan-lopes!

I'm working on it! I do not have much free time, but I've started this feature 👍

[rsalmei]

Wow, it's already working in about-time!!
It's very late here, gotta go, but I wouldn't stop until I'd seen it working!!
rsalmei/about-time#6

[rsalmei]

Hey @lschmelzeisen and @jhonatan-lopes, question:
Do you think it really does make sense to support these two options separately?

I'm thinking... If I sent a "unit" like b, perhaps it wouldn't make much sense to show numbers like 1850000b instead of 1.85Mb, and vice-versa, if I wanted 1.85M, it would be nicer to send the unit of that. In the worst case, one could send an empty string as the unit.

In short, I think I could implement both the "unit" and "scaling with SI-prefixes" using just one configuration: unit.
WDYT?

[jhonatan-lopes]

Hi @rsalmei, I see what you mean, but consider the following scenarios:

1. I am counting something (say S units) that I don't care much about the specific number, just a rough idea would be fine. Since it goes into thousands/millions, I am fine with an auto-scaling. Therefore, I use unit=S and wish that it auto-scales 1850000S to 1.85MS. In this case, alive-bar must default to auto-scaling and a 1000 unit scale parameter, i.e. a kilo-unit means 1000 of that unit.

2. I want to configure the bar to display bytes downloaded. This scenario is quite similar to the one above, except that unit_scale=1024. You could code the unit parameter to scale to 1024 when unit=B or unit=b, but I think that it loses generality.

3. I want to configure the bar to display an iteration counter that goes up to thousands. Maybe in this case I want to actually see at which iteration I am at a given moment with precision, so unit=it (for iterations) but unit_scale=1 meaning that I don't want it to scale with SI prefixes. In this case I actually want to see 1850000 it.

With all that said, I do think that the two configurations are needed, both unit and unit_scale, especially since the user can wish to scale with esoteric bases/factors other than 10 and 2.

It could be that the default is that when unit is given and unit_scale is not, it assumes base 10 and auto-scaling with SI-prefixes.

What do you think?

[rsalmei]

Thanks, @jhonatan-lopes.

I think it is almost that... Your first and second scenarios are actually the same, where we do have auto-scaling when the unit is given (regardless of the divisor). The third is the one I do not think would be very needed: with the unit given but disabling auto-scaling...

But OK, I can make them separate, and give alive-progress more flexibility.
It just would be ugly if I wanted to support the _end variants, to enable a different rendition in the receipt. Thus, at least for the moment, I'll code the receipt to display as the running bar.

👍

[rsalmei]

Hey, look at that!
It's working!

Next week I should release it. 😉

[rsalmei]

For the configuration, we have:

• unit: any text, of any length.

I've also created a new factory that accepts some fixed options with aliases, so:

• scale: one of (None, '2', '2_iec', '10', False, True)
  -> False is the same as None, True is 10

And the cherry on top, I'm implementing a help system for configuration errors:

with alive_bar(..., scale=2) as bar:
    ...
---------------------------------------------------------------------------
...
ValueError: Invalid config value: scale=2
Expected one of: (None, '2', '2_iec', '10', False, True)

[rsalmei]

It's done, it will be included in the next release!

[Jasonzyt]

is there a pre-release to use this feature?:eyes:
thx

[rsalmei]

Hey @Jasonzyt, not yet, I couldn't even commit it yet!
It's still all on my laptop here, but it is completely ready.
I'll try to finalize it asap 👍

[rsalmei]

Hey folks, it is committed, and I'm about to release it...
Thanks for waiting this long.