Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1970129: Add env variable OVS_SYS_LOG_LEVEL for ovn nodes to setup ovs syslog level #1142

Merged
merged 2 commits into from Jul 15, 2021

Conversation

npinaeva
Copy link
Member

@npinaeva npinaeva commented Jun 28, 2021

Reverted: #1163

Since OVS is run on host, OVS logs are collected from systemd journal. Default value for ovs syslog level is err, set it to info to collect info level logs with must-gather.

How to test:
Run ovs-appctl vlog/list on a node, assure syslog level is set to requested level.
Run must-gather, check host_service_logs/<node_name>/ovs-vswitchs_service.log file. Previously it only contained ERR messages like this:

Jun 28 07:35:19.248737 ci-ln-hxbr65t-f76d1-zw5kl-master-0 ovs-vswitchd[1234]: ovs|00040|stream_ssl|ERR|SSL_use_PrivateKey_file: error:20074002:BIO routines:file_ctrl:system lib
Jun 28 07:35:19.248780 ci-ln-hxbr65t-f76d1-zw5kl-master-0 ovs-vswitchd[1234]: ovs|00041|stream_ssl|ERR|failed to load client certificates from /ovn-ca/ca-bundle.crt: error:140AD002:SSL routines:SSL_CTX_use_certificate_file:system lib

After setting default level to INFO it should have INFO messages like this:

Jun 28 08:54:14.080482 ci-ln-hxbr65t-f76d1-zw5kl-master-0 ovs-vswitchd[1234]: ovs|00507|connmgr|INFO|br-ex<->unix#1028: 2 flow_mods in the last 0 s (2 adds)
Jun 28 08:54:21.020662 ci-ln-hxbr65t-f76d1-zw5kl-master-0 ovs-vswitchd[1234]: ovs|00508|connmgr|INFO|br-int<->unix#1020: 5123 flow_mods 10 s ago (5122 adds, 1 deletes)

You can change default OVS_LOG_LEVEL value with ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: env-overrides
  namespace: openshift-ovn-kubernetes
  annotations:
data:
  # to set the node processes on a single node to verbose
  # replace this with the node's name (from oc get nodes)
  ip-10-0-135-96.us-east-2.compute.internal: |
    OVN_KUBE_LOG_LEVEL=5
    OVN_LOG_LEVEL=dbg
    OVS_LOG_LEVEL=dbg
  # to adjust master log levels, use _master
  _master: |
    OVN_KUBE_LOG_LEVEL=5
    OVN_LOG_LEVEL=dbg

@openshift-ci openshift-ci bot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Jun 28, 2021
@openshift-ci openshift-ci bot requested review from abhat and danwinship June 28, 2021 10:52
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 28, 2021

@npinaeva: This pull request references Bugzilla bug 1970129, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (anusaxen@redhat.com), skipping review request.

In response to this:

Bug 1970129: Add env variable OVS_LOG_LEVEL for ovn nodes to setup ovs syslog level

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Jun 28, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 29, 2021

@npinaeva: This pull request references Bugzilla bug 1970129, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (anusaxen@redhat.com), skipping review request.

In response to this:

Bug 1970129: Add env variable OVS_LOG_LEVEL for ovn nodes to setup ovs syslog level

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@npinaeva
Copy link
Member Author

npinaeva commented Jul 7, 2021

/retest

@tssurya
Copy link
Contributor

tssurya commented Jul 7, 2021

/lgtm

Simple fix to change the ovs log level.
Based on the comment here: openshift/must-gather#243 (comment) from @trozet seems like the right way to go.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 7, 2021
@tssurya
Copy link
Contributor

tssurya commented Jul 7, 2021

CI failures seem unrelated

README.md Outdated
@@ -229,6 +229,7 @@ data:
ip-10-0-135-96.us-east-2.compute.internal: |
OVN_KUBE_LOG_LEVEL=5
OVN_LOG_LEVEL=dbg
OVS_LOG_LEVEL=dbg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we call this OVS_SYS_LOG_LEVEL? That makes it more explicit, since you could set console and file logging too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, explicit is always better

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 8, 2021
@npinaeva npinaeva changed the title Bug 1970129: Add env variable OVS_LOG_LEVEL for ovn nodes to setup ovs syslog level Bug 1970129: Add env variable OVS_SYS_LOG_LEVEL for ovn nodes to setup ovs syslog level Jul 14, 2021
@abhat
Copy link
Contributor

abhat commented Jul 14, 2021

/lgtm
/retest

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 14, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 14, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, npinaeva, tssurya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 14, 2021
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

3 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 15, 2021

@npinaeva: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-openstack-ovn 3d44831 link /test e2e-openstack-ovn
ci/prow/e2e-azure-ovn 3d44831 link /test e2e-azure-ovn
ci/prow/e2e-ovn-ipsec-step-registry 3d44831 link /test e2e-ovn-ipsec-step-registry
ci/prow/e2e-metal-ipi-ovn-ipv6-ipsec 3d44831 link /test e2e-metal-ipi-ovn-ipv6-ipsec
ci/prow/e2e-gcp-ovn-upgrade 3d44831 link /test e2e-gcp-ovn-upgrade
ci/prow/e2e-vsphere-windows 3d44831 link /test e2e-vsphere-windows

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@abhat
Copy link
Contributor

abhat commented Jul 15, 2021

/override ci/prow/e2e-gcp-ovn

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 15, 2021

@abhat: Overrode contexts on behalf of abhat: ci/prow/e2e-gcp-ovn

In response to this:

/override ci/prow/e2e-gcp-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-merge-robot openshift-merge-robot merged commit 3c03534 into openshift:master Jul 15, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 15, 2021

@npinaeva: All pull requests linked via external trackers have merged:

Bugzilla bug 1970129 has been moved to the MODIFIED state.

In response to this:

Bug 1970129: Add env variable OVS_SYS_LOG_LEVEL for ovn nodes to setup ovs syslog level

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@npinaeva npinaeva deleted the change-ovs-log-level branch July 20, 2021 08:44
npinaeva added a commit to npinaeva/cluster-network-operator that referenced this pull request Jul 23, 2021
Move setting ovs log level to machine-config-operator
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 23, 2021

@npinaeva: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

These pull request must merge or be unlinked from the Bugzilla bug in order for it to move to the next state. Once unlinked, request a bug refresh with /bugzilla refresh.

Bugzilla bug 1970129 has not been moved to the MODIFIED state.

In response to this:

Bug 1970129: Add env variable OVS_SYS_LOG_LEVEL for ovn nodes to setup ovs syslog level

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants