Bug 1849244: [release-4.5]Trace ovn processing in cluster #160

pecameron · 2020-06-19T23:16:27Z

Backport #156 and #159 to release-4.5

PR 159 - Extract trace of ovn bringup

Scan through the generated trace for key lines in ovn bringup and report them.

SDN-1028 - Trace: generate trace of networking bring up
Trace: generate trace of networking bring up

PR 156 - Generate trace of ovn operation

This processes log files for namespace openshift-ovn-kubernetes.
It also captures the master and worker journals for kublet and crio.
It:

formats time stamps into a common format
adds a prefix to each log line consisting of:
time stamp
node name
pod name
container name

Extracts the log and previous log for each continer

applies the prefix to each line
combines all resulting files and
sorts the results into the trace
deletes work files.

SDN-975 - Logging: add must-gather networking-gather-trace script
https://issues.redhat.com/browse/SDN-975

Signed-off-by: Phil Cameron pcameron@redhat.com

Backport openshift#156 to release-4.5 -------------- This processes log files for namespace openshift-ovn-kubernetes. It also captures the master and worker journals for kublet and crio. It: - formats time stamps into a common format - adds a prefix to each log line consisting of: - time stamp - node name - pod name - container name Extracts the log and previous log for each continer - applies the prefix to each line - combines all resulting files and - sorts the results into the trace - deletes work files. SDN-975 - Logging: add must-gather networking-gather-trace script https://issues.redhat.com/browse/SDN-975 Signed-off-by: Phil Cameron <pcameron@redhat.com>

openshift-ci-robot · 2020-06-19T23:16:28Z

@pecameron: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Trace ovn processing in cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-19T23:16:58Z

@pecameron: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

[release-4.5]Trace ovn processing in cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-06-19T23:37:07Z

@pecameron: This pull request references Bugzilla bug 1849244, which is invalid:

expected dependent Bugzilla bug 1849243 to target a release in 4.6.0, 4.6.z, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1849244: [release-4.5]Trace ovn processing in cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-06-19T23:37:46Z

/bugzilla refresh

openshift-ci-robot · 2020-06-19T23:37:48Z

@pecameron: This pull request references Bugzilla bug 1849244, which is invalid:

expected dependent Bugzilla bug 1849243 to target a release in 4.6.0, 4.6.z, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-06-19T23:41:26Z

/bugzilla refresh

openshift-ci-robot · 2020-06-19T23:41:32Z

@pecameron: This pull request references Bugzilla bug 1849244, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.0) matches configured target release for branch (4.5.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1849243 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Bugzilla bug 1849243 targets the "4.6.0" release, which is one of the valid target releases: 4.6.0, 4.6.z
bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-06-19T23:44:04Z

@dcbw PTAL
need this and (TDB) backport 4.4 to get openshift/release#8654 to work. CI is a 4.4 cluster.

dcbw · 2020-06-24T18:31:26Z

/approve
/lgtm
/retest
(looks like an AWS setup issue, not due to this PR)

pecameron · 2020-07-07T12:15:55Z

@dcbw what does it take to move this forward?

pecameron · 2020-07-08T12:36:10Z

/assign @sferich888

sferich888 · 2020-07-08T12:58:59Z

/lgtm

openshift-bot · 2020-07-08T13:02:26Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-07-08T13:15:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

pecameron · 2020-07-09T17:18:49Z

@sferich888 any idea when this will merge?

sferich888 · 2020-07-09T18:18:36Z

When tests pass.

Scan through the generated trace for key lines in ovn bringup and report them. SDN-1028 - Trace: generate trace of networking bring up Trace: generate trace of networking bring up Signed-off-by: Phil Cameron <pcameron@redhat.com>

openshift-ci-robot · 2020-07-10T12:29:26Z

@pecameron: This pull request references Bugzilla bug 1849244, which is invalid:

expected dependent Bugzilla bug 1849243 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is CLOSED (WORKSFORME) instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1849244: [release-4.5]Trace ovn processing in cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-07-10T12:38:08Z

@dcbw PTAL
@sferich888 PTAL
I added PR159 which generates and additional report on network bringup

pecameron · 2020-07-10T14:21:50Z

/bugzilla refresh

openshift-ci-robot · 2020-07-10T14:21:53Z

@pecameron: This pull request references Bugzilla bug 1849244, which is invalid:

expected dependent Bugzilla bug 1849243 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is CLOSED (WORKSFORME) instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-07-10T14:24:41Z

/bugzilla refresh

openshift-ci-robot · 2020-07-10T14:24:45Z

@pecameron: This pull request references Bugzilla bug 1849244, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.5.z) matches configured target release for branch (4.5.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1849243 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
dependent Bugzilla bug 1849243 targets the "4.6.0" release, which is one of the valid target releases: 4.6.0, 4.6.z
bug has dependents

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-07-20T17:25:57Z

@sferich888 @dcbw do you know who can cherry-pick approve this? It adds trace support for 4.5 clusters including the ci cluster. This permits a trace to be generated as part of ci artifacts. This change is a new script added to gather_network_ovn_trace which is not referenced by anything currently in 4.5. There is another PR in release to that is blocked waiting for this. This is a no risk change.

dcbw · 2020-07-30T18:00:02Z

/lgtm

openshift-ci-robot · 2020-07-30T18:00:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, pecameron, sferich888

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~collection-scripts/OWNERS~~ [sferich888]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sdodson · 2020-07-31T18:13:32Z

do you know who can cherry-pick approve this?

That's always the patch manager, that process never varies.

sdodson · 2020-08-01T17:09:38Z

/retest

openshift-ci-robot · 2020-08-01T18:41:36Z

@pecameron: All pull requests linked via external trackers have merged: openshift/must-gather#160. Bugzilla bug 1849244 has been moved to the MODIFIED state.

In response to this:

Bug 1849244: [release-4.5]Trace ovn processing in cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from davemulford and RickJWagner June 19, 2020 23:16

pecameron changed the title ~~Trace ovn processing in cluster~~ [release-4.5]Trace ovn processing in cluster Jun 19, 2020

pecameron changed the title ~~[release-4.5]Trace ovn processing in cluster~~ Bug 1849244: [release-4.5]Trace ovn processing in cluster Jun 19, 2020

openshift-ci-robot added the bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. label Jun 19, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Jun 19, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jun 19, 2020

openshift-ci-robot assigned dcbw Jun 24, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 24, 2020

openshift-ci-robot assigned sferich888 Jul 8, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 8, 2020

Extract trace of ovn bringup

b15eea9

Scan through the generated trace for key lines in ovn bringup and report them. SDN-1028 - Trace: generate trace of networking bring up Trace: generate trace of networking bring up Signed-off-by: Phil Cameron <pcameron@redhat.com>

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 10, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2020

sdodson added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jul 31, 2020

openshift-merge-robot merged commit 4e97dec into openshift:release-4.5 Aug 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 1849244: [release-4.5]Trace ovn processing in cluster #160

Bug 1849244: [release-4.5]Trace ovn processing in cluster #160

pecameron commented Jun 19, 2020 •

edited

openshift-ci-robot commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

dcbw commented Jun 24, 2020

pecameron commented Jul 7, 2020

pecameron commented Jul 8, 2020

sferich888 commented Jul 8, 2020

openshift-bot commented Jul 8, 2020

openshift-bot commented Jul 8, 2020

pecameron commented Jul 9, 2020

sferich888 commented Jul 9, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 10, 2020

pecameron commented Jul 10, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 10, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 20, 2020

dcbw commented Jul 30, 2020

openshift-ci-robot commented Jul 30, 2020

sdodson commented Jul 31, 2020

sdodson commented Aug 1, 2020

openshift-ci-robot commented Aug 1, 2020

Bug 1849244: [release-4.5]Trace ovn processing in cluster #160

Bug 1849244: [release-4.5]Trace ovn processing in cluster #160

Conversation

pecameron commented Jun 19, 2020 • edited

openshift-ci-robot commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

openshift-ci-robot commented Jun 19, 2020

pecameron commented Jun 19, 2020

dcbw commented Jun 24, 2020

pecameron commented Jul 7, 2020

pecameron commented Jul 8, 2020

sferich888 commented Jul 8, 2020

openshift-bot commented Jul 8, 2020

openshift-bot commented Jul 8, 2020

pecameron commented Jul 9, 2020

sferich888 commented Jul 9, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 10, 2020

pecameron commented Jul 10, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 10, 2020

openshift-ci-robot commented Jul 10, 2020

pecameron commented Jul 20, 2020

dcbw commented Jul 30, 2020

openshift-ci-robot commented Jul 30, 2020

sdodson commented Jul 31, 2020

sdodson commented Aug 1, 2020

openshift-ci-robot commented Aug 1, 2020

pecameron commented Jun 19, 2020 •

edited