Ability to more easily track bandwidth usage

[Joshfindit]

I've successfully deployed an MVP, but I've run in to unexpectedly high usage on the bucket.

I see that the debug logs do list usage in the form of read[7] 32768 bytes from 16384, but what we'd really like is something that could be used without debug logging.

Ideally stats where we could see:

• What file was accessed
• What the datetime was
• How much of the data was read at that time

[Joshfindit]

(It may be worth noting that we're using B2, and Backblaze currently does not offer any sort of usage reporting. You have to contact support to get the total for individual days as opposed to the monthly total shown in the UI)

[ncclementi]

@Joshfindit There have been recently some efforts on getting more information display in the form of plots about network bandwidth, check dask/distributed#5129 .

I am afraid that this feature request is a bit broad and needs more information, if you are interested in describing in more detail what is what you would like to see, please let us know in this thread. I would leave the issue open for now, with a needs info tag.

[Joshfindit]

@ncclementi I checked the thread and that looks awesome already.

I apologize as I’m too far out from the project I was referencing and don’t have access to the notes about the specifics.

Generally my questions around these types of requests are about one of the following questions:

• How can we be sure that everything is running as expected?
• There was an unexpectedly high usage bill. What specific process was at fault, and what was it accessing/doing to cause that?

Feel free to close this request if those don’t lead to meaningful upgrades for Dask.

[ncclementi]

Thank you for your reply @Joshfindit, I think those are valid questions and they would probably be helpful for other people. I wonder if @jrbourbeau has any insight/updates on this.

[ian-r-rose]

The situation has improved in the last year around visibility into worker network usage. I don't think there are many specific ideas here for how to further improve observability in a cloud context, so I'm going to close this. But if you have more thoughts about specific features we could build into dask/distributed, feel free to comment here or in a new issue.