Bug 1798191: Bump to ovn2.12 #445

pecameron · 2020-01-15T15:32:25Z

New paths for components

SDN-643

Signed-off-by: Phil Cameron pcameron@redhat.com

pecameron · 2020-01-15T15:33:42Z

/test e2e-aws-ovn

pecameron · 2020-01-15T18:21:02Z

/test e2e-aws-ovn

pecameron · 2020-01-20T22:45:21Z

/test e2e-aws-ovn

pecameron · 2020-01-21T21:01:57Z

/test e2e-aws-ovn

pecameron · 2020-01-21T21:16:04Z

/test e2e-aws-ovn
/retest

pecameron · 2020-01-21T22:02:31Z

/test e2e-aws-ovn

pecameron · 2020-01-21T22:04:46Z

@alexanderConstantinescu @rcarrillocruz @dcbw PTAL
This only has effect when ovn-kube image has ovn2.12
I need to get this in to get ovn-kubernetes PR72 to pass ci
This worked in a local test on AWS.

danwinship · 2020-01-22T09:32:21Z

bindata/network/ovn-kubernetes/ovnkube-master.yaml

@@ -70,7 +70,7 @@ spec:

          exec ovn-northd \
            --no-chdir "-vconsole:${OVN_LOG_LEVEL}" -vfile:off \
-            --pidfile=/var/run/openvswitch/ovn-northd.pid \
+            --pidfile=ovn-northd.pid \


According to the man pages (at least for 2.11) this will make it put the pidfile in .

However, do we actually need to be writing the pid files at all? It looks like the only pid file we ever read is the ovn-nbctl daemon's, and the others are "write-only" and we could just remove the --pidfile arguments...

Changed. Dropped unneeded pid files.

danwinship · 2020-01-22T09:33:48Z

bindata/network/ovn-kubernetes/ovnkube-master.yaml

@@ -330,7 +336,7 @@ spec:
          fi

          # start nbctl daemon for caching
-          export OVN_NB_DAEMON=$(ovn-nbctl --pidfile=/run/openvswitch/ovnk-nbctl.pid \
+          export OVN_NB_DAEMON=$(ovn-nbctl --pidfile=ovnk-nbctl.pid \


the container's preStop hook still does kill $(cat /run/openvswitch/ovnk-nbctl.pid)

danwinship · 2020-01-22T09:35:26Z

bindata/network/ovn-kubernetes/ovnkube-master.yaml

@@ -394,7 +402,11 @@ spec:
        hostPath:
          path: /var/lib/ovn/data
      - name: run-openvswitch
-        emptyDir: {}
+        hostPath:
+          path: /var/run/openvswitch


This makes it so that the container's /run/openvswitch would be preserved between runs of different containers, which I don't think we want...

So, if I am not mistaken (and I think I would like to "call a friend" on this one: @dcbw ). Those locations contain the OVN DB. Which we do not want to preserve between runs because:

I am pretty sure we can not upgrade and will crash when going from non-HA -> HA and vice versa (if we would ever, not sure we would though as non-HA is only in version 4.1/4.2 - tech-preview?)

Our source of truth is k8s and we sync directly with it on startup anyways, and thus re-write the DB.

So, yes, I don't think we want this.

Why did you add this though @pecameron ? Is it required for 2.12?

The unix socket needs to be everywhere it is referenced. This fixed the problem.

This actually passes tests. I will back out the change.

pecameron · 2020-01-22T21:44:35Z

/test e2e-aws-ovn

pecameron · 2020-01-23T16:11:44Z

/test e2e-aws-ovn

pecameron · 2020-01-23T18:10:40Z

No nodes, no pods....
/test e2e-aws-ovn

pecameron · 2020-01-23T18:47:48Z

"Error: ResourceLimitExceeded: Task limit exceeded"
/test e2e-aws-ovn

pecameron · 2020-01-23T21:56:15Z

tcp 10.0.140.52:6443: connect: connection refused
/test e2e-aws-ovn

pecameron · 2020-01-24T18:25:10Z

Cant't reach api server
/test e2e-aws-ovn

pecameron · 2020-01-27T18:40:00Z

/test e2e-aws-ovn

pecameron · 2020-01-31T21:13:53Z

/retest
/test e2e-aws-ovn

pecameron · 2020-02-03T14:50:19Z

/retest
/test e2e-aws-ovn

pecameron · 2020-02-03T19:42:27Z

level=error msg="Cluster operator openshift-etcd Degraded is True with ConfigObservationDegradedError: ConfigObservationDegraded: error looking up self: lookup _etcd-server-ssl._tcp.ci-op-nd4331wx-583c6.origin-ci-int-aws.dev.rhcloud.com on 172.30.0.10:53: read udp 10.129.0.21:43654->172.30.0.10:53: i/o timeout\nConfigObservationDegraded: error looking up self: lookup _etcd-server-ssl._tcp.ci-op-nd4331wx-583c6.origin-ci-int-aws.dev.rhcloud.com on 172.30.0.10:53: read udp 10.129.0.21:55834->172.30.0.10:53: i/o timeout"

/retest
/test e2e-aws-ovn

pecameron · 2020-02-03T22:00:44Z

Different set of failures this time. Trying again.
/retest
/test e2e-aws-ovn

pecameron · 2020-02-04T14:30:27Z

/retest
/test e2e-aws-ovn

pecameron · 2020-02-04T14:43:01Z

e2e-aws-ovn
error: Couldn't find resource for "crd-publish-openapi-test-foo.example.com/v1,
"alertname":"KubeAPILatencyHigh",
E kube-apiserver failed contacting the API: Get https://api.ci-op-h891bjwp-583c6.origin-ci-int-aws.dev.rhcloud.com:6443
The container could not be located when the pod was terminated
error deleting EBS volume "vol-0c95094660511ec2a" since volume is currently attached to "i-04c55ee4d769dbb2d"

pecameron · 2020-02-04T17:07:13Z

@dcbw 445 passed tests. @alexanderConstantinescu @danwinship @rcarrillocruz Can we get this in? PTAL. I need it to test PR72 ovn2.12 image.

New paths for components pidfile changes. SDN-643 Signed-off-by: Phil Cameron <pcameron@redhat.com>

pecameron · 2020-02-04T20:10:19Z

/test e2e-aws-ovn

pecameron · 2020-02-04T20:31:01Z

Back to a flaky CI....
release "release-latest" failed: the pod ci-op-kztttv4m/release-latest failed after 35s
/test e2e-aws-ovn

dcbw · 2020-02-04T20:51:15Z

/test e2e-gcp-ovn

dcbw · 2020-02-04T21:06:14Z

level=info msg="Credentials loaded from the \"default\" profile in file \"/etc/openshift-installer/.awscred\""
level=fatal msg="failed to fetch Terraform Variables: failed to fetch dependency of \"Terraform Variables\": failed to fetch dependency of \"Bootstrap Ignition Config\": failed to fetch dependency of \"Common Manifests\": failed to generate asset \"DNS Config\": getting public zone for \"origin-ci-int-aws.dev.rhcloud.com\": listing hosted zones: Throttling: Rate exceeded\n\tstatus code: 400, request id: 8de95699-cd1b-4fb2-b67b-d297e4222f61"
2020/02/04 20:51:09 Container setup in pod e2e-aws-ovn failed, exit code 1, reason Error

/test e2e-aws-ovn

dcbw · 2020-02-04T21:16:50Z

/lgtm
/approve

openshift-ci-robot · 2020-02-04T21:21:34Z

@pecameron: This pull request references Bugzilla bug 1798191, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug 1798191: Bump to ovn2.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dcbw · 2020-02-04T21:41:48Z

/approve

openshift-ci-robot · 2020-02-04T21:57:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, pecameron

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dcbw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-02-04T22:12:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-04T22:36:57Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-02-05T00:06:07Z

@pecameron: All pull requests linked via external trackers have merged. Bugzilla bug 1798191 has been moved to the MODIFIED state.

In response to this:

Bug 1798191: Bump to ovn2.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-05T00:09:48Z

@pecameron: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-ovn	`2711b32`	link	`/test e2e-gcp-ovn`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 15, 2020

openshift-ci-robot requested review from alexanderConstantinescu and rcarrillocruz January 15, 2020 15:33

pecameron force-pushed the sdn643a branch from 9728eb8 to 4d29e13 Compare January 20, 2020 22:43

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 20, 2020

pecameron force-pushed the sdn643a branch from 4d29e13 to 4e0f79e Compare January 21, 2020 22:01

pecameron changed the title ~~[WIP] bump to ovn2.12~~ Bump to ovn2.12 Jan 21, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2020

danwinship reviewed Jan 22, 2020

View reviewed changes

pecameron force-pushed the sdn643a branch from 4e0f79e to 0ee40a2 Compare January 22, 2020 21:43

pecameron changed the title ~~Bump to ovn2.12~~ [WIP] (need to test with image that has ovn2.12) Bump to ovn2.12 Jan 23, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2020

pecameron force-pushed the sdn643a branch from 0ee40a2 to a421917 Compare January 23, 2020 16:11

pecameron force-pushed the sdn643a branch from a421917 to 70e694e Compare January 27, 2020 18:39

pecameron changed the title ~~Bump to ovn2.12~~ Bug 1798191 Bump to ovn2.12 Feb 4, 2020

pecameron force-pushed the sdn643a branch from 47975fa to 6d36234 Compare February 4, 2020 18:58

Bug 1798191 Bump to ovn2.12

2711b32

New paths for components pidfile changes. SDN-643 Signed-off-by: Phil Cameron <pcameron@redhat.com>

pecameron force-pushed the sdn643a branch from 6d36234 to 2711b32 Compare February 4, 2020 19:22

openshift-ci-robot assigned dcbw Feb 4, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 4, 2020

pecameron changed the title ~~Bug 1798191 Bump to ovn2.12~~ Bug 1798191: Bump to ovn2.12 Feb 4, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Feb 4, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 4, 2020

openshift-merge-robot merged commit 9dc3987 into openshift:master Feb 5, 2020

danwinship mentioned this pull request Feb 18, 2020

Bug 1806009: Backport ovn-kubernetes-related changes to 4.3 #483

Merged

Bug 1798191: Bump to ovn2.12 #445

Bug 1798191: Bump to ovn2.12 #445

Conversation

pecameron commented Jan 15, 2020

pecameron commented Jan 15, 2020

pecameron commented Jan 15, 2020

pecameron commented Jan 20, 2020

pecameron commented Jan 21, 2020

pecameron commented Jan 21, 2020

pecameron commented Jan 21, 2020

pecameron commented Jan 21, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pecameron commented Jan 22, 2020

pecameron commented Jan 23, 2020

pecameron commented Jan 23, 2020

pecameron commented Jan 23, 2020

pecameron commented Jan 23, 2020

pecameron commented Jan 24, 2020

pecameron commented Jan 27, 2020

pecameron commented Jan 31, 2020

pecameron commented Feb 3, 2020

pecameron commented Feb 3, 2020

pecameron commented Feb 3, 2020

pecameron commented Feb 4, 2020

pecameron commented Feb 4, 2020

pecameron commented Feb 4, 2020

pecameron commented Feb 4, 2020

pecameron commented Feb 4, 2020

dcbw commented Feb 4, 2020

dcbw commented Feb 4, 2020

dcbw commented Feb 4, 2020

openshift-ci-robot commented Feb 4, 2020

dcbw commented Feb 4, 2020

openshift-ci-robot commented Feb 4, 2020

openshift-bot commented Feb 4, 2020

openshift-bot commented Feb 4, 2020

openshift-ci-robot commented Feb 5, 2020

openshift-ci-robot commented Feb 5, 2020

pecameron commented Jan 21, 2020 •

edited