Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Improve Monitor Stage #86

Closed
mdemoret-nv opened this issue May 5, 2022 · 1 comment · Fixed by #805
Closed

[FEA] Improve Monitor Stage #86

mdemoret-nv opened this issue May 5, 2022 · 1 comment · Fixed by #805
Assignees
Labels
feature request New feature or request

Comments

@mdemoret-nv
Copy link
Contributor

Is your feature request related to a problem? Please describe.
The MonitorStage currently has several flaws that reduce its usefulness:

  1. It has its own buffered queue meaning messages are not tracked as they are processed, but when back pressure forces them to be processed. This delays and alters the measured results
  2. It requires a separate python thread to periodically update the screen with the latest results
  3. The throughput numbers cannot be queried or displayed anywhere besides the console
  4. It does not scale well. If multiple monitors are added, they all redraw and refresh at separate times.

Describe the solution you'd like
Ideally, the MonitorStage would:

  1. Immediately track the throughput of the previous stage without any buffering. This would require changing the type of node that is used to a node that does not have it's own progress engine.
  2. Be implemented in C++ to avoid grabbing the GIL when monitoring throughput
  3. Utilize a common library such as Prometheus to allow extracting the measured values or displaying outside of the console
  4. Synchronize updates between multiple instances to avoid scaling issues.
@mdemoret-nv mdemoret-nv added the feature request New feature or request label May 5, 2022
@mdemoret-nv mdemoret-nv self-assigned this May 5, 2022
@mdemoret-nv mdemoret-nv added this to Issue - Needs Prioritizing in v22.06 Release via automation May 5, 2022
@mdemoret-nv mdemoret-nv moved this from Issue - Needs Prioritizing to Issue - P1 in v22.06 Release May 5, 2022
@mdemoret-nv mdemoret-nv removed this from Issue - P1 in v22.06 Release Jun 13, 2022
@mdemoret-nv mdemoret-nv added this to Issue - Needs Prioritizing in v22.08 Release via automation Jun 13, 2022
@mdemoret-nv mdemoret-nv moved this from Issue - Needs Prioritizing to Issue - P1 in v22.08 Release Jul 5, 2022
@mdemoret-nv
Copy link
Contributor Author

Moving to 23.11 since this relies on non-linear pipelines in SRF.

@rapids-bot rapids-bot bot closed this as completed in #805 Mar 29, 2023
rapids-bot bot pushed a commit that referenced this issue Mar 29, 2023
This PR converts `MonitorStage` to be a node component so it will track progress using the upstream node's progress engine. This will make the output of the monitor stage more accurate since results will not be buffered before they are tracked.

Closes #86

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #805
v22.08 Release automation moved this from Issue - P1 to Done Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants