Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1874638: Fixes missing nbctl daemon logging #772

Merged

Conversation

trozet
Copy link
Contributor

@trozet trozet commented Sep 1, 2020

Signed-off-by: Tim Rozet trozet@redhat.com

@trozet
Copy link
Contributor Author

trozet commented Sep 1, 2020

@astoycos FYI

@trozet trozet changed the title Fixes missing nbctl daemon logging Bug 1874638: Fixes missing nbctl daemon logging Sep 1, 2020
@trozet
Copy link
Contributor Author

trozet commented Sep 1, 2020

/bugzilla refresh

@astoycos
Copy link
Contributor

astoycos commented Sep 1, 2020

Thanks @trozet this should help in debugging future issues between ovnkube-master and nbdb

@astoycos
Copy link
Contributor

astoycos commented Sep 1, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2020

# include nbctl daemon logging, allow for ovn-nbctl to create the log file
sleep 1
tail -f /run/ovn/ovn-nbctl.log &
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add --retry to the tail args and then it will work even if "sleep 1" isn't long enough.

(You could theoretically get rid of the "sleep 1" entirely, but then tail would log a pointless warning, so it's best to have both.)

@openshift-ci-robot openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2020
@danwinship
Copy link
Contributor

Jinx! So the problem I was saying is, the way it is now, you'll get:

tail: cannot open '/run/ovn/ovn-nbctl.log' for reading: No such file or directory
tail: '/run/ovn/ovn-nbctl.log' has appeared;  following new file

ugh, and actually you get

tail: warning: --retry only effective for the initial open

too. Way to be useless, tail...

@trozet
Copy link
Contributor Author

trozet commented Sep 2, 2020

hmm not seeing the nbctl logs

Signed-off-by: Tim Rozet <trozet@redhat.com>
@trozet
Copy link
Contributor Author

trozet commented Sep 24, 2020

/retest

@trozet
Copy link
Contributor Author

trozet commented Sep 24, 2020

I see a bunch of ovn-nbctl logging now in the ovn-kube logs.

@astoycos
Copy link
Contributor

astoycos commented Oct 5, 2020

/retest

2 similar comments
@astoycos
Copy link
Contributor

astoycos commented Oct 6, 2020

/retest

@astoycos
Copy link
Contributor

astoycos commented Oct 9, 2020

/retest

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 9, 2020

@trozet: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-vsphere cfff586 link /test e2e-vsphere
ci/prow/e2e-gcp-upgrade cfff586 link /test e2e-gcp-upgrade
ci/prow/e2e-openstack 8aa3078 link /test e2e-openstack
ci/prow/e2e-vsphere-ovn 8aa3078 link /test e2e-vsphere-ovn

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@astoycos
Copy link
Contributor

/retest

--db "{{.OVN_NB_DB_LIST}}" --log-file=/run/ovn/ovn-nbctl.log)

# include nbctl daemon logging, allow for ovn-nbctl to create the log file
tail -f --retry /run/ovn/ovn-nbctl.log &
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to tail it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we can see exactly in the ovnkube-master logs when the commands happen. Really helpful in debugging. I suspect as we migrate more and more to the golang bindings they will eventually go away, but it's a good way to track things as well that need to get migrated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I should have provided more context. I meant, yes, its useful to tail it, but that data is already in the log file on disk. But I suppose the real value add is the interleaving of those commands with the master logs.

Copy link
Contributor

@rbbratta rbbratta Nov 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-f --retry is --follow=descriptor --retry whereas the usual -F is --follow=name --retry

       -f, --follow[={name|descriptor}]
              output appended data as the file grows;

              an absent option argument means 'descriptor'

We usually want --follow=name, is there a reason we are only following descriptor.

Also, do we want --pid=$(cat /var/run/ovn/ovn-nbctl.pid) as in the other tail commands to stop the tail once the daemon stops?

@abhat
Copy link
Contributor

abhat commented Nov 3, 2020

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 3, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhat, astoycos, trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2020
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

6 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@astoycos
Copy link
Contributor

astoycos commented Nov 3, 2020

@abhat It's weird the merge bot is still not recognizing ci/prow/e2e-openstack as not required

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

2 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@abhat
Copy link
Contributor

abhat commented Nov 3, 2020

/override e2e-openstack

@openshift-ci-robot
Copy link
Contributor

@abhat: /override requires a failed status context to operate on.
The following unknown contexts were given:

  • e2e-openstack

Only the following contexts were expected:

  • ci/prow/e2e-agnostic-upgrade
  • ci/prow/e2e-aws-ovn-windows
  • ci/prow/e2e-aws-ovn-windows-custom-vxlan
  • ci/prow/e2e-aws-sdn-multi
  • ci/prow/e2e-aws-sdn-single
  • ci/prow/e2e-azure
  • ci/prow/e2e-azure-ovn
  • ci/prow/e2e-gcp
  • ci/prow/e2e-gcp-ovn
  • ci/prow/e2e-gcp-ovn-upgrade
  • ci/prow/e2e-metal-ipi
  • ci/prow/e2e-metal-ipi-ovn-ipv6
  • ci/prow/e2e-openstack
  • ci/prow/e2e-openstack-ovn
  • ci/prow/e2e-operator-with-custom-vxlan-port
  • ci/prow/e2e-ovn-hybrid-step-registry
  • ci/prow/e2e-ovn-step-registry
  • ci/prow/e2e-upgrade
  • ci/prow/e2e-vsphere-ovn
  • ci/prow/e2e-windows-hybrid-network
  • ci/prow/images
  • ci/prow/unit
  • ci/prow/verify
  • tide

In response to this:

/override e2e-openstack

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@abhat
Copy link
Contributor

abhat commented Nov 3, 2020

/override ci/prow/e2e-openstack

@openshift-ci-robot
Copy link
Contributor

@abhat: Overrode contexts on behalf of abhat: ci/prow/e2e-openstack

In response to this:

/override ci/prow/e2e-openstack

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

10 similar comments
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit ad49849 into openshift:master Nov 4, 2020
@openshift-ci-robot
Copy link
Contributor

@trozet: All pull requests linked via external trackers have merged:

Bugzilla bug 1874638 has been moved to the MODIFIED state.

In response to this:

Bug 1874638: Fixes missing nbctl daemon logging

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants