Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachprod: prometheus file_sd_config has missing labels #124319

Closed
nameisbhaskar opened this issue May 17, 2024 · 1 comment · Fixed by #124598
Closed

roachprod: prometheus file_sd_config has missing labels #124319

nameisbhaskar opened this issue May 17, 2024 · 1 comment · Fixed by #124598
Assignees
@nameisbhaskar
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

Comments

@nameisbhaskar
Copy link
Contributor

nameisbhaskar commented May 17, 2024 •
edited
Loading

Previously when the gce_sd_configs was used, few labels were read from the gce metadata using the config below:

  • source_labels: [__meta_gce_private_ip]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: ${1}:9100
    action: replace
  • source_labels: [__meta_gce_instance_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}-sys
    action: replace
  • source_labels: [__meta_gce_instance_name]
    separator: ;
    regex: (.*)
    target_label: host
    replacement: ${1}
    action: replace
  • source_labels: [__meta_gce_label_cluster]
    separator: ;
    regex: (.*)
    target_label: cluster
    replacement: ${1}
    action: replace
  • source_labels: [__meta_gce_tags]
    separator: ;
    regex: (.*)
    target_label: instance_tags
    replacement: ${1}
    action: replace
  • source_labels: [__meta_gce_label_test_name]
    separator: ;
    regex: (.*)
    target_label: test_name
    replacement: ${1}
    action: replace
  • source_labels: [__meta_gce_label_test_run_id]
    separator: ;
    regex: (.*)
    target_label: test_run_id
    replacement: ${1}
    action: replace

Now, as we are using file_sd_configs, these configs has to be fed to prometheus in a different way.

The approach can be to add the labels directly to the file using the pormhelpers

Jira issue: CRDB-38823
@nameisbhaskar nameisbhaskar added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label May 17, 2024
@nameisbhaskar nameisbhaskar self-assigned this May 17, 2024
@nameisbhaskar nameisbhaskar mentioned this issue May 20, 2024
Merged
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 20, 2024
@nameisbhaskar
roachprod: add custom tags for prometheus config
0fc302a 
few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 21, 2024
@nameisbhaskar
roachprod: add custom tags for prometheus config
9e30f6f 
few of the labels are missing in prometheus with the change
from "gce_sd_configs" to "file_sd_configs". This is because
these were extracted from the VM metadata which is not
available now. So, the same values are set from roachprod now.

Informs: cockroachdb#124319
Epic: none
craig bot pushed a commit that referenced this issue May 21, 2024
@nameisbhaskar
Merge #124409
d146ecf 
124409: roachprod: add custom tags for prometheus config r=srosenberg a=nameisbhaskar

few of the labels are missing in prometheus with the change from "gce_sd_configs" to "file_sd_configs". This is because these were extracted from the VM metadata which is not available now. So, the same values are set from roachprod now.

Informs: #124319
Epic: none

Co-authored-by: Bhaskarjyoti Bora <bhaskar.bora@cockroachlabs.com>
@nameisbhaskar nameisbhaskar mentioned this issue May 21, 2024
Closed
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 21, 2024
@nameisbhaskar
roachprod: changes in promhelper client for modify labels
87646e9 
When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 22, 2024
@nameisbhaskar
roachprod: changes in promhelper client for modify labels
2413158 
When a test is run, a new label has to be added to prometheus.
To do this, we need to first get the current config, add the
new label and update the config. This change does that. Post
this change, the label update will be integrated.

The change also needs a change in httputil client GET function
to support headers being passed.

Fixes: cockroachdb#124319
Epic: none
@nameisbhaskar
Copy link
Contributor Author

The issue is fixed with the previous commit as we set the test tags at start and we always restart the clusters before running a new test.

nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 23, 2024
@nameisbhaskar
roachprod: condition to support cockroach-ephemeral
a93aff0 
only the clusters in cockroach-ephemeral should be selected
for scrapping. So, adding a condition for the same.

Fixes: cockroachdb#124319
Epic: none
@nameisbhaskar nameisbhaskar mentioned this issue May 23, 2024
Merged
nameisbhaskar added a commit to nameisbhaskar/cockroach that referenced this issue May 23, 2024
@nameisbhaskar
roachprod: condition to support cockroach-ephemeral
d6728f3 
only the clusters in cockroach-ephemeral should be selected
for scrapping. So, adding a condition for the same.

Fixes: cockroachdb#124319
Epic: none
craig bot pushed a commit that referenced this issue May 28, 2024
@nameisbhaskar
Merge #124598
46b6fab 
124598: roachprod: condition to support cockroach-ephemeral r=herkolategan a=nameisbhaskar

only the clusters in cockroach-ephemeral should be selected for scrapping. So, adding a condition for the same.

Fixes: #124319
Epic: none

Co-authored-by: Bhaskarjyoti Bora <bhaskar.bora@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nameisbhaskar nameisbhaskar

Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.
Projects
None yet
Milestone
No milestone
1 participant
@nameisbhaskar