Bug 2059470: Unable to connect external Grafana with Openshift Monitoring #1549

arajkumar · 2022-01-28T11:49:05Z

Executed the following API calls as described in https://grafana.com/docs/grafana/latest/datasources/prometheus/#query-variable.

$ curl -X GET -kG 'https://localhost:9092/api/v1/query?query=up&namespace=ns1'  -H "Authorization: Bearer $TOKEN"
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"up","endpoint":"web","instance":"10.128.2.14:8080","job":"prometheus-example-app","namespace":"ns1","pod":"prometheus-example-app-d748cfb54-fss5h","prometheus":"openshift-user-workload-monitoring/user-workload","service":"prometheus-example-app"},"value":[1643640318.046,"1"]},{"metric":{"__name__":"up","endpoint":"web","instance":"10.129.2.14:8080","job":"prometheus-example-app","namespace":"ns1","pod":"prometheus-example-app-d748cfb54-z2xh6","prometheus":"openshift-user-workload-monitoring/user-workload","service":"prometheus-example-app"},"value":[1643640318.046,"1"]}]}}


$ curl -X GET -kG 'https://localhost:9092/api/v1/labels?namespace=ns1'  -H "Authorization: Bearer $TOKEN"
{"status":"success","data":["__name__","cluster_ip","condition","configmap","container","container_id","cpu","created_by_kind","created_by_name","deployment","device","endpoint","host_ip","host_network","id","image","image_id","instance","interface","job","label_app","label_kubernetes_io_metadata_name","label_pod_template_hash","metrics_path","name","namespace","network_name","node","owner_is_controller","owner_kind","owner_name","phase","pod","pod_ip","prometheus","prometheus_replica","reason","replicaset","secret","service","status","type","uid","ulimit","version","workload","workload_type"]}

$ curl -X GET -kG 'https://localhost:9092/api/v1/label/__name__/values?namespace=ns1'  -H "Authorization: Bearer $TOKEN"
{"status":"success","data":["container_cpu_system_seconds_total","container_cpu_usage_seconds_total","container_cpu_user_seconds_total","container_file_descriptors","container_fs_usage_bytes","container_last_seen","container_memory_cache","container_memory_failcnt","container_memory_mapped_file","container_memory_max_usage_bytes","container_memory_rss","container_memory_swap","container_memory_usage_bytes","container_memory_working_set_bytes","container_network_receive_bytes_total","container_network_receive_errors_total","container_network_receive_packets_dropped_total","container_network_receive_packets_total","container_network_transmit_bytes_total","container_network_transmit_errors_total","container_network_transmit_packets_dropped_total","container_network_transmit_packets_total","container_processes","container_sockets","container_spec_cpu_period","container_spec_cpu_shares","container_spec_memory_limit_bytes","container_spec_memory_reservation_limit_bytes","container_spec_memory_swap_limit_bytes","container_start_time_seconds","container_threads","container_threads_max","container_ulimits_soft","kube_configmap_info","kube_deployment_labels","kube_deployment_metadata_generation","kube_deployment_spec_paused","kube_deployment_spec_replicas","kube_deployment_spec_strategy_rollingupdate_max_surge","kube_deployment_spec_strategy_rollingupdate_max_unavailable","kube_deployment_status_condition","kube_deployment_status_observed_generation","kube_deployment_status_replicas","kube_deployment_status_replicas_available","kube_deployment_status_replicas_unavailable","kube_deployment_status_replicas_updated","kube_endpoint_address_available","kube_endpoint_address_not_ready","kube_endpoint_info","kube_endpoint_labels","kube_namespace_labels","kube_namespace_status_phase","kube_pod_container_info","kube_pod_container_state_started","kube_pod_container_status_ready","kube_pod_container_status_restarts_total","kube_pod_container_status_waiting","kube_pod_info","kube_pod_labels","kube_pod_owner","kube_pod_start_time","kube_pod_status_phase","kube_pod_status_ready","kube_pod_status_reason","kube_replicaset_labels","kube_replicaset_owner","kube_replicaset_spec_replicas","kube_replicaset_status_fully_labeled_replicas","kube_replicaset_status_ready_replicas","kube_replicaset_status_replicas","kube_running_pod_ready","kube_secret_info","kube_secret_type","kube_service_info","kube_service_labels","kube_service_spec_type","kubelet_container_log_filesystem_used_bytes","namespace:container_cpu_usage:sum","namespace:container_memory_usage_bytes:sum","namespace_workload_pod:kube_pod_owner:relabel","node_namespace_pod:kube_pod_info:","node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate","node_namespace_pod_container:container_memory_cache","node_namespace_pod_container:container_memory_rss","node_namespace_pod_container:container_memory_swap","node_namespace_pod_container:container_memory_working_set_bytes","pod:container_cpu_usage:sum","pod:container_fs_usage_bytes:sum","pod_network_name_info","scrape_duration_seconds","scrape_samples_post_metric_relabeling","scrape_samples_scraped","scrape_series_added","up","version"]}

$ curl -X GET -kG 'https://localhost:9092/api/v1/series?namespace=ns1'  -H "Authorization: Bearer $TOKEN"
{"status":"success","data":[{"__name__":"container_cpu_system_seconds_total","container":"POD","endpoint":"https-metrics","id":"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9d7c8cb_4320_4937_aef8_c8c94a3d5653.slice/crio-74283bda373d3c53c64ab94e52b674841e0da1d7c1bf5f7f77be0f252d77296b.scope","instance":"10.0.128.4:10250","job":"kubelet","metrics_path":"/metrics/cadvisor","name":"k8s_POD_prometheus-example-app-d748cfb54-z2xh6_ns1_a9d7c8cb-4320-4937-aef8-c8c94a3d5653_0","namespace":"ns1","node":"ci-ln-zjbdptk-72292-fg6kj-worker-b-2rkrp","pod":"prometheus-example-app-d748cfb54-z2xh6","prometheus":"openshift-monitoring/k8s","service":"kubelet"},{"__name__":"container_cpu_system_seconds_total","container":"POD","endpoint":"https-metrics"
...

I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.

openshift-ci · 2022-01-28T11:49:06Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2022-01-28T11:49:06Z

@arajkumar: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Back port changes to support external grafana

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-01-31T14:45:55Z

@arajkumar: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Back port changes to support external grafana

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

arajkumar · 2022-02-10T17:01:43Z

Tested the changes with grafana-operator(with grafana v7.5.11) and all prometheus template query variable function works!

jan--f · 2022-02-11T13:09:31Z

This lgtm as a whole. Any reason to keep this as a draft?

I guess these commits were cherry-picked from master? I'd strongly encourage the use of gits -x option in future situations like this.

arajkumar · 2022-02-15T06:36:16Z

This lgtm as a whole. Any reason to keep this as a draft?

I was actually testing the changes.

jan--f · 2022-02-25T14:51:51Z

/lgtm
We will need a BZ for this.

openshift-ci · 2022-02-25T14:54:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: arajkumar, jan--f

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [arajkumar,jan--f]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-02-25T14:59:20Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T15:11:22Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T15:23:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T15:59:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T16:11:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T16:59:19Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T19:35:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T19:47:19Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-02-25T21:11:20Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-03-09T05:28:37Z

@arajkumar: This pull request references Bugzilla bug 2059470, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.z) matches configured target release for branch (4.9.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2059468 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2059468 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
bug has dependents

Requesting review from QA contact:
/cc @juzhao

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

arajkumar · 2022-03-09T05:28:59Z

/retest

arajkumar · 2022-03-09T05:32:38Z

/skip

simonpasquier · 2022-03-09T12:49:29Z

/hold cancel

https://bugzilla.redhat.com/show_bug.cgi?id=2059468 has been verified

openshift-bot · 2022-03-09T13:43:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-09T14:31:40Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

simonpasquier · 2022-03-09T15:04:56Z

/skip

juzhao · 2022-03-10T07:05:11Z

/api/v1/series is supported by the namespace scoped Thanos endpoint 9092 port
/label cherry-pick-approved

juzhao · 2022-03-10T07:05:27Z

/label qe-approved

openshift-bot · 2022-03-10T10:49:29Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T11:41:28Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T13:51:28Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T15:09:28Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T16:53:29Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T17:19:31Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T18:44:29Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T20:32:30Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-10T22:56:30Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-03-10T23:26:36Z

@arajkumar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-single-node	`cfe9f1f`	link	false	`/test e2e-aws-single-node`
ci/prow/versions	`cfe9f1f`	link	false	`/test versions`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2022-03-11T03:33:22Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-03-11T04:33:21Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-03-11T05:46:33Z

@arajkumar: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#1549

Bugzilla bug 2059470 has been moved to the MODIFIED state.

In response to this:

Bug 2059470: Unable to connect external Grafana with Openshift Monitoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

raptorsun and others added 4 commits January 28, 2022 12:37

expose /api/v1/labels end point for Thanos query.

c06e74b

e2e test for prometheus endpoint /api/v1/labels

9ced1be

jsonnet,test: enable /api/v1/series on Thanos tenancy port

c7cfecb

assets: regenerate

cfe9f1f

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 28, 2022

openshift-ci bot requested review from fpetkovski and jan--f January 28, 2022 11:49

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 28, 2022

arajkumar marked this pull request as ready for review February 15, 2022 06:36

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 15, 2022

openshift-ci bot requested a review from slashpai February 15, 2022 06:37

openshift-ci bot assigned jan--f Feb 25, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 25, 2022

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Mar 9, 2022

openshift-ci bot requested review from juzhao and removed request for fpetkovski March 9, 2022 05:28

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 9, 2022

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Mar 10, 2022

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Mar 10, 2022

openshift-merge-robot merged commit f387b22 into openshift:release-4.9 Mar 11, 2022

Bug 2059470: Unable to connect external Grafana with Openshift Monitoring #1549

Bug 2059470: Unable to connect external Grafana with Openshift Monitoring #1549

Conversation

arajkumar commented Jan 28, 2022 • edited

openshift-ci bot commented Jan 28, 2022

openshift-ci bot commented Jan 28, 2022

openshift-ci bot commented Jan 31, 2022

arajkumar commented Feb 10, 2022

jan--f commented Feb 11, 2022

arajkumar commented Feb 15, 2022

jan--f commented Feb 25, 2022

openshift-ci bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-bot commented Feb 25, 2022

openshift-ci bot commented Mar 9, 2022

arajkumar commented Mar 9, 2022

arajkumar commented Mar 9, 2022

simonpasquier commented Mar 9, 2022

openshift-bot commented Mar 9, 2022

openshift-bot commented Mar 9, 2022

simonpasquier commented Mar 9, 2022

juzhao commented Mar 10, 2022

juzhao commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-bot commented Mar 10, 2022

openshift-ci bot commented Mar 10, 2022 • edited

openshift-bot commented Mar 11, 2022

openshift-bot commented Mar 11, 2022

openshift-ci bot commented Mar 11, 2022

arajkumar commented Jan 28, 2022 •

edited

openshift-ci bot commented Mar 10, 2022 •

edited