Bug 1811834: Sync jsonnet dependencies #722

paulfantom · 2020-03-25T09:23:44Z

Multiple things in one PR (sorry):

Porting Add DELETECOLLECTION to long-running apiserver verbs kubernetes-monitoring/kubernetes-mixin#375 to fix BZ 1811834
Securing /metrics endpoint for prometheus-operators, JIRA MON-990
updating labels on node-exporter and adding mechanism to delete DaemonSet when necessary

Some other changes include ones described in prometheus-operator/kube-prometheus#470

Note: jsonnetfile.lock.json was updated in stages to check what is changing, but in the end it came down to running jb update on all.

/cc @openshift/openshift-team-monitoring

openshift-ci-robot · 2020-03-25T09:23:58Z

@paulfantom: This pull request references Bugzilla bug 1811834, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1811834: Sync jsonnet dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

paulfantom · 2020-03-25T12:16:14Z

/hold
until prometheus/node_exporter#1649 is merged and propagated

lilic · 2020-03-25T15:11:45Z

Its been merged it seems.

paulfantom · 2020-03-26T09:18:48Z

/unhold

prometheus/node_exporter#1649 is merged and propagated here.

simonpasquier · 2020-03-26T10:17:53Z

assets/prometheus-operator/service-monitor.yaml

@@ -9,8 +9,19 @@ metadata:
  namespace: openshift-monitoring
 spec:
  endpoints:
-  - honorLabels: true
-    port: http
+  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token


why do we have 2 endpoint entries?

Good catch! Fixing.

paulfantom · 2020-03-26T13:07:25Z

/test generate

pkg/client/client.go

paulfantom · 2020-03-26T14:08:34Z

/retest

lilic · 2020-03-26T16:51:03Z

/retest

should be back now

paulfantom · 2020-03-27T07:52:06Z

/retest

… certificates on /metrics endpoints

paulfantom · 2020-03-27T08:46:37Z

Rebased to solve merge conflicts

lilic

Couple of questions, otherwise looks good

lilic · 2020-03-27T09:21:50Z

assets/prometheus-k8s/rules.yaml

+          is configured on this host.
+        summary: Clock not synchronising.
+      expr: |
+        min_over_time(node_timex_sync_status[5m]) == 0


We don't need the job label selector here, like we had for ClockSkewDetected?

That metric comes only from node_exporter which is deployed as DaemonSet, so I don't see why we would need it.

lilic · 2020-03-27T09:28:55Z

assets/prometheus-operator-user-workload/cluster-role.yaml

@@ -87,3 +87,15 @@ rules:
  - get
  - list
  - watch
+- apiGroups:


Do we know why this was added for prometheus-operator in kube-prom, can't seem to find it in the prometheus-operator repo itself? https://github.com/coreos/prometheus-operator/blob/master/example/rbac/prometheus-operator/prometheus-operator-cluster-role.yaml should this be updated there?

Without those kube-rbac-proxy didn't work and metrics couldn't be scraped.

okay that explains that, thanks!

lilic · 2020-03-27T09:29:52Z

pkg/client/client.go

 			}
 		}
-		return errors.Wrap(err, "updating deployment object failed")
+		return errors.Wrap(err, "updating Deployment object failed")


Follow up task to us, maybe tech-debt, unrelated to this PR: we should include the name of these objects in the logs. WDYT?

good idea, could you create JIRA?

Done! https://issues.redhat.com/browse/MON-1048

lilic · 2020-03-27T10:22:29Z

aws fails with 🤔:

fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:231]: possibly some services didn't register ServiceMonitors to allow metrics collection
Unexpected error:
<*errors.errorString | 0xc00028d1d0>: {
s: "timed out waiting for the condition",
}
timed out waiting for the condition
occurred

paulfantom · 2020-03-27T11:13:04Z

argh, yes. We are checking if prometheus-operator is available over http and not https -.-

Source: https://github.com/openshift/origin/blob/master/test/extended/prometheus/prometheus.go#L222

paulfantom · 2020-03-29T09:26:24Z

/retest

lilic

/lgtm

/hold

In case simon wants to review as well.

simonpasquier · 2020-03-30T08:36:34Z

/lgtm

/hold cancel

openshift-ci-robot · 2020-03-30T08:36:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lilic, paulfantom, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lilic,paulfantom,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-03-30T10:02:49Z

@paulfantom: All pull requests linked via external trackers have merged: openshift/cluster-monitoring-operator#722. Bugzilla bug 1811834 has been moved to the MODIFIED state.

In response to this:

Bug 1811834: Sync jsonnet dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sferich888 · 2020-06-16T13:45:33Z

/cherry-pick release-4.4
/cherry-pick release-4.3
/cherry-pick release-3.11

openshift-cherrypick-robot · 2020-06-16T13:45:39Z

@sferich888: #722 failed to apply on top of branch "release-4.4":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	jsonnet/jsonnetfile.lock.json
Falling back to patching base and 3-way merge...
Auto-merging jsonnet/jsonnetfile.lock.json
CONFLICT (content): Merge conflict in jsonnet/jsonnetfile.lock.json
Patch failed at 0001 jsonnet: update dependencies

In response to this:

/cherry-pick release-4.4
/cherry-pick release-4.3
/cherry-pick release-3.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lilic · 2020-06-16T14:11:57Z

@sferich888 that won't work as the lock file needs to be generated for each release branch, what are you trying to do backport? :)

openshift-ci-robot requested a review from a team March 25, 2020 09:23

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Mar 25, 2020

paulfantom force-pushed the sync_deps branch 4 times, most recently from e0ea276 to 9923716 Compare March 25, 2020 11:48

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 25, 2020

paulfantom force-pushed the sync_deps branch from 9923716 to 79162a5 Compare March 26, 2020 09:14

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 26, 2020

simonpasquier reviewed Mar 26, 2020

View reviewed changes

paulfantom force-pushed the sync_deps branch 2 times, most recently from 63c1fef to 78836ef Compare March 26, 2020 12:35

simonpasquier reviewed Mar 26, 2020

View reviewed changes

pkg/client/client.go Outdated Show resolved Hide resolved

paulfantom changed the title ~~Bug 1811834: Sync jsonnet dependencies~~ WIP: Bug 1811834: Sync jsonnet dependencies Mar 26, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2020

paulfantom force-pushed the sync_deps branch 2 times, most recently from e276c10 to ad4ef92 Compare March 26, 2020 13:55

paulfantom changed the title ~~WIP: Bug 1811834: Sync jsonnet dependencies~~ Bug 1811834: Sync jsonnet dependencies Mar 27, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 27, 2020

paulfantom added 2 commits March 27, 2020 09:42

jsonnet: update dependencies

23fde35

jsonnet,pkg/manifests: update prometheus-operators to use trusted TLS…

9817e6d

… certificates on /metrics endpoints

paulfantom added 3 commits March 27, 2020 09:45

assets,manifests,pkg/manifests: regenerate

4130a3d

pkg/client/client.go: If update of DaemonSet fails recreate DaemonSet

9b3c2b4

pkg/client/client.go: Uppercase Deployment object name

7134248

paulfantom force-pushed the sync_deps branch from ad4ef92 to 7134248 Compare March 27, 2020 08:46

lilic reviewed Mar 27, 2020

View reviewed changes

paulfantom mentioned this pull request Mar 27, 2020

test/extended/prometheus: do not check if prom-op is up openshift/origin#24780

Merged

lilic reviewed Mar 30, 2020

View reviewed changes

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 30, 2020

openshift-ci-robot assigned lilic Mar 30, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 30, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 30, 2020

openshift-ci-robot assigned simonpasquier Mar 30, 2020

openshift-merge-robot merged commit e61d233 into openshift:master Mar 30, 2020

paulfantom mentioned this pull request Mar 30, 2020

Check if prometheus-operator is reporting metrics openshift/origin#24795

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1811834: Sync jsonnet dependencies #722

Bug 1811834: Sync jsonnet dependencies #722

paulfantom commented Mar 25, 2020

openshift-ci-robot commented Mar 25, 2020

paulfantom commented Mar 25, 2020

lilic commented Mar 25, 2020

paulfantom commented Mar 26, 2020

simonpasquier Mar 26, 2020

paulfantom Mar 26, 2020

paulfantom commented Mar 26, 2020

paulfantom commented Mar 26, 2020

lilic commented Mar 26, 2020

paulfantom commented Mar 27, 2020

paulfantom commented Mar 27, 2020

lilic left a comment

lilic Mar 27, 2020

paulfantom Mar 27, 2020

lilic Mar 27, 2020

paulfantom Mar 27, 2020 •

edited

lilic Mar 27, 2020

lilic Mar 27, 2020

lilic Mar 27, 2020

paulfantom Mar 27, 2020

lilic Mar 30, 2020

lilic commented Mar 27, 2020

paulfantom commented Mar 27, 2020

paulfantom commented Mar 29, 2020

lilic left a comment

simonpasquier commented Mar 30, 2020

openshift-ci-robot commented Mar 30, 2020

openshift-ci-robot commented Mar 30, 2020

sferich888 commented Jun 16, 2020

openshift-cherrypick-robot commented Jun 16, 2020

lilic commented Jun 16, 2020

Bug 1811834: Sync jsonnet dependencies #722

Bug 1811834: Sync jsonnet dependencies #722

Conversation

paulfantom commented Mar 25, 2020

openshift-ci-robot commented Mar 25, 2020

paulfantom commented Mar 25, 2020

lilic commented Mar 25, 2020

paulfantom commented Mar 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulfantom commented Mar 26, 2020

paulfantom commented Mar 26, 2020

lilic commented Mar 26, 2020

paulfantom commented Mar 27, 2020

paulfantom commented Mar 27, 2020

lilic left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulfantom Mar 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilic commented Mar 27, 2020

paulfantom commented Mar 27, 2020

paulfantom commented Mar 29, 2020

lilic left a comment

Choose a reason for hiding this comment

simonpasquier commented Mar 30, 2020

openshift-ci-robot commented Mar 30, 2020

openshift-ci-robot commented Mar 30, 2020

sferich888 commented Jun 16, 2020

openshift-cherrypick-robot commented Jun 16, 2020

lilic commented Jun 16, 2020

paulfantom Mar 27, 2020 •

edited