Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds open telemetry metrics for tasks #3821

Closed
Tracked by #4604
bmbouter opened this issue May 12, 2023 · 0 comments · Fixed by #5234
Closed
Tracked by #4604

Adds open telemetry metrics for tasks #3821

bmbouter opened this issue May 12, 2023 · 0 comments · Fixed by #5234
Assignees
@bmbouter
Labels
Feature services

Comments

@bmbouter
Copy link
Member

bmbouter commented May 12, 2023 •
edited by decko

Background / Need

It would be good to know more about the tasking system to answer the following question:

  • I want to know if I have the right number of workers for a given queue in a specific time.

Proposal

Report measurements from the tasking system, like:

  • Number of unblocked waiting tasks waiting more than 5 seconds
  • the longest waiting task time

Also, find the best place to trigger those measurements to avoid unneeded queries or any avoidable overload.
@bmbouter bmbouter self-assigned this May 12, 2023
bmbouter added a commit to bmbouter/pulpcore that referenced this issue May 12, 2023
@bmbouter
Adds basic worker metrics
4dc03a3 
The following metrics are added:
* `task_count` - The cumulative number of tasks a worker has executed
  since it was started.

closes pulp#3821
@bmbouter bmbouter mentioned this issue May 12, 2023
Closed
@bmbouter bmbouter changed the title Adds worker metrics tracking Adds open telemetry metrics for Worker tasks Oct 20, 2023
@bmbouter bmbouter changed the title Adds open telemetry metrics for Worker tasks Adds open telemetry metrics for workers Oct 20, 2023
@bmbouter bmbouter changed the title Adds open telemetry metrics for workers Adds open telemetry metrics for tasks Oct 20, 2023
@bmbouter bmbouter mentioned this issue Oct 20, 2023
Open
4 tasks
@git-hyagi git-hyagi mentioned this issue Jan 11, 2024
Open
@bmbouter bmbouter mentioned this issue Feb 16, 2024
Closed
decko added a commit to decko/pulpcore that referenced this issue Apr 8, 2024
@decko
Add initial code. Tests still need to be written.
0909c11 
Close pulp#3821
@decko decko mentioned this issue Apr 8, 2024
Merged
decko added a commit to decko/pulpcore that referenced this issue Apr 8, 2024
@decko
Add initial code. Tests still need to be written.
cfcaffc 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 10, 2024
@decko
A slight change on how to generate metrics.
fc8200d 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 19, 2024
@decko
Refactor the previous development. Watch out! Work in progress.
24f5a91 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 22, 2024
@decko
Change the way we trigger the metric.
f47836c 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 24, 2024
@decko
Add a new query and some updates.
a8ad19f 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 24, 2024
@decko
Update template_config.yml
433a32c 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 24, 2024
@decko
Linting issues
eeefa7f 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 24, 2024
@decko
Linting issues
0f45777 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 24, 2024
@decko
Linting issues
d51fdb5 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 25, 2024
@decko
Change trigger to heartbeat.
2c12daf 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 25, 2024
@decko
Remove an unused line.
1c86c00 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 25, 2024
@decko
Remove an unused idea.
0919a62 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue Apr 26, 2024
@decko
Change the way we trigger the metric heartbeat.
be8d645 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 2, 2024
@decko
Change some comments.
0a08e81 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the test to fill up the workers with long running tasks.
45ca91a 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
a7eba48 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
24ea78b 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
9ee7609 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
2259df4 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
d7adb2a 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
e936e19 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
70598d6 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
addade5 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 3, 2024
@decko
Change the way we setup the test, and also add some comments.
1c84817 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Cancel the tasks after running the asserts.
9aae6b2 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Cancel the tasks after running the asserts.
29537ce 
Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Remove the NOTIFY and LISTEN for metric emiter. No sense on using the…
82d79e4 
…m now.

Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Remove the NOTIFY and LISTEN for metric emiter. No sense on using the…
5023476 
…m now.

Close pulp#3821
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Adds two new metrics for the Task system.
45c4693 
Close pulp#3821
Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko
Adds two new metrics for the Task system.
12b5b71 
Close pulp#3821
Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 6, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
5e989df 
Close pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 7, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
32cce28 
Close pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 7, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
2be246a 
Close pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 8, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
89acc63 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 8, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
d8ec695 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 13, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
0f43213 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 21, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
6e4b181 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 21, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
1817e92 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 22, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
fb8274c 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 22, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
f0ad30e 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
decko added a commit to decko/pulpcore that referenced this issue May 22, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
48ed4e5 
Closes pulp#3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
mdellweg added a commit that referenced this issue May 23, 2024
@decko @mdellweg
@lubosmj @ggainey @ipanova
Adds two new metrics for the Task system.
914c3e3 
Closes #3821

Co-authored-by: Matthias Dellweg <2500@gmx.de>
Co-authored-by: Ľuboš Mjachky <lmjachky@redhat.com>
Co-authored-by: Grant Gainey <ggainey@users.noreply.github.com>
Co-authored-by: Ina Panova <ipanova@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bmbouter bmbouter

Labels
Feature services
Projects
RH Pulp Kanban board
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Worker metrics. decko/pulpcore
Adds basic worker metrics bmbouter/pulpcore
3 participants
@bmbouter @ipanova @ggainey