You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The MonitorStage currently has several flaws that reduce its usefulness:
It has its own buffered queue meaning messages are not tracked as they are processed, but when back pressure forces them to be processed. This delays and alters the measured results
It requires a separate python thread to periodically update the screen with the latest results
The throughput numbers cannot be queried or displayed anywhere besides the console
It does not scale well. If multiple monitors are added, they all redraw and refresh at separate times.
Describe the solution you'd like
Ideally, the MonitorStage would:
Immediately track the throughput of the previous stage without any buffering. This would require changing the type of node that is used to a node that does not have it's own progress engine.
Be implemented in C++ to avoid grabbing the GIL when monitoring throughput
Utilize a common library such as Prometheus to allow extracting the measured values or displaying outside of the console
Synchronize updates between multiple instances to avoid scaling issues.
The text was updated successfully, but these errors were encountered:
This PR converts `MonitorStage` to be a node component so it will track progress using the upstream node's progress engine. This will make the output of the monitor stage more accurate since results will not be buffered before they are tracked.
Closes#86
Authors:
- Michael Demoret (https://github.com/mdemoret-nv)
- David Gardner (https://github.com/dagardner-nv)
Approvers:
- David Gardner (https://github.com/dagardner-nv)
URL: #805
Is your feature request related to a problem? Please describe.
The
MonitorStage
currently has several flaws that reduce its usefulness:Describe the solution you'd like
Ideally, the
MonitorStage
would:The text was updated successfully, but these errors were encountered: