Fix Kind support (Linux hosts only) #137

antoninbas · 2019-11-25T23:05:29Z

The following changes were required:

Disable TX HW checksum offload in containers. This is done in the
Antrea CNI server when setting-up Pod networking, using an ioctl
ethtool system call.
Disable TX HW checksum offload in the Linux host for the veth
interface of each Kind Node. This must be done by invoking an
additional script (hack/kind_linux.sh) after creating the Kind
cluster.
Create a secondary br-phy bridge on each Node, as required by OVS
userspace tunneling.
Use a new version of start_ovs (start_ovs_netdev) which modifies the
ovs-ctl script in-place to avoid loading the kernel module.

Refer to #14 for the rationale for all the above bullet points.

A new test "provider" was added to the e2e test framework so that all
the e2e tests can be run on Kind clusters. As part of this, some
changes to the framework had to be performed. For example it is
impractical to run SSH commands on Kind Nodes - as they do not have an
SSH server - so instead we use "docker exec".

Fixes #14
Fixes #13

antrea-bot · 2019-11-25T23:05:31Z

Thanks for your PR.
Unit tests and code linters are run automatically every time the PR is updated.
E2e tests can only be triggered by a member of the vmware-tanzu organization. Regular contributors to the project should join the org.

The following commands are available:

/test-e2e: to trigger e2e tests. This command can only be run by members of the vmware-tanzu organization
/skip-e2e: to skip e2e tests. This command can only be run by members of the vmware-tanzu organization

antoninbas · 2019-11-25T23:06:01Z

pkg/agent/cniserver/pod_configuration.go

@@ -66,6 +66,11 @@ func setupInterface(podName string, podNamespace string, ifname string, netns ns
 		containerIface.Sandbox = netns.Path()
 		hostIface.Name = hostVeth.Name
 		hostIface.Mac = hostVeth.HardwareAddr.String()
+		// TODO: should be conditional on the OVS datapath type
+		// Waiting for https://github.com/vmware-tanzu/antrea/pull/125 to be merged


waiting for #125 to be merged and I will update this PR

antoninbas · 2019-11-25T23:06:51Z

/test-e2e

antoninbas · 2019-11-25T23:07:23Z

I'm working on an additional commit for this that will enable a CI job for a Kind cluster.

antoninbas · 2019-11-26T02:00:56Z

The CI job was configured and works properly!

antoninbas · 2019-11-26T02:01:05Z

/test-e2e

jianjuns

Question - how could user switch from netdev to kernel datapath or the other way? Do they need some manual cleanup?

build/yamls/patches/kind/startOvs.yml

build/images/scripts/start_ovs_netdev

pkg/agent/util/ethtool.go

tnqn

LGTM, some minor comments

pkg/agent/util/ethtool.go

antoninbas · 2019-11-27T18:52:33Z

I have updated the PR now that #125 has been merged and I have squashed all my commits.

antoninbas · 2019-11-27T18:52:42Z

/test-e2e

jianjuns

LGTM.

Could you answer my question - what will happen if we switch between netdev and kernel datapath (not in a Kind env but a normal K8s cluster)? Do we need any manual cleanup before changing the datapath?

antoninbas · 2019-11-27T20:01:07Z

@jianjuns That's a good question. We definitely need to think about what should be cleaned up automatically when the agent Pod exits. At the moment we do not delete the br-int bridge. That means that you cannot change the datapath type without manually deleting the bridge. But if we delete the bridge on exit in start_ovs, then we won't be able to preserve connectivity on restarts / updates (which BTW is a problem we already have since OVS deletes flows from the datapath on exit).

I think ideally we want all of this:

deleting Antrea from a cluster (kubectl delete -f antrea.yml) should clean-up everything created by Antrea on the Nodes
agent Pod restarts (at least planned restarts for upgrades) should not delete OVS flows
on restart, if agent detects that the datapath type has changed, the existing bridge should be deleted and re-created

I can take care of 3 in a follow-up PR, but do you have any good ideas on handling 1 & 2 (assuming OVS can be patched so that flows are not deleted on daemon restart).

jianjuns · 2019-11-27T20:10:04Z

deleting Antrea from a cluster (kubectl delete -f antrea.yml) should clean-up everything created by Antrea on the Nodes
This could not be done by kubectl delete -f antrea.yml. We probably need a cleanup DS, or maybe CRD/CLI/ConfigMap to trigger cleanup.
agent Pod restarts (at least planned restarts for upgrades) should not delete OVS flows
This is the OVS change Ben is helping out?
on restart, if agent detects that the datapath type has changed, the existing bridge should be deleted and re-created
Actually I asked you this because I did not see you do any thing special for br-int and OVS port (except offload), and so wonder does it mean we do not need to recreate bridge and ports if the datapath type changes?

antoninbas · 2019-11-27T20:12:37Z

No, we need to. But this issue pre-dates this PR and is independent of Kind, so my plan is to do it in a follow-up PR.

antoninbas · 2019-11-27T20:17:25Z

/test-e2e

The following changes were required: * Disable TX HW checksum offload in containers. This is done in the Antrea CNI server when setting-up Pod networking, using an ioctl ethtool system call. * Disable TX HW checksum offload in the Linux host for the veth interface of each Kind Node. This must be done by invoking an additional script (hack/kind_linux.sh) after creating the Kind cluster. * Create a secondary br-phy bridge on each Node, as required by OVS userspace tunneling. * Use a new version of start_ovs (start_ovs_netdev) which modifies the ovs-ctl script in-place to avoid loading the kernel module. Refer to antrea-io#14 for the rationale for all the above bullet points. A new test "provider" was added to the e2e test framework so that all the e2e tests can be run on Kind clusters. As part of this, some changes to the framework had to be performed. For example it is impractical to run SSH commands on Kind Nodes - as they do not have an SSH server - so instead we use "docker exec". Fixes antrea-io#14 Fixes antrea-io#13

antoninbas · 2019-11-27T20:27:24Z

/test-e2e

antoninbas · 2019-11-27T20:28:01Z

@jianjuns I need a new approve for this, I renamed kind_linux.sh to kind-linux.sh to match other scripts.

jianjuns

Here you go.

vmwclabot added the cla-not-required label Nov 25, 2019

antoninbas commented Nov 25, 2019

View reviewed changes

antoninbas force-pushed the fix-kind-support branch 7 times, most recently from e9e11b4 to eb106dc Compare November 26, 2019 01:41

jianjuns reviewed Nov 26, 2019

View reviewed changes

build/yamls/patches/kind/startOvs.yml Show resolved Hide resolved

build/images/scripts/start_ovs_netdev Show resolved Hide resolved

build/images/scripts/start_ovs_netdev Show resolved Hide resolved

pkg/agent/util/ethtool.go Outdated Show resolved Hide resolved

tnqn reviewed Nov 26, 2019

View reviewed changes

pkg/agent/util/ethtool.go Outdated Show resolved Hide resolved

pkg/agent/util/ethtool.go Outdated Show resolved Hide resolved

pkg/agent/util/ethtool.go Outdated Show resolved Hide resolved

antoninbas force-pushed the fix-kind-support branch 2 times, most recently from e10f0dc to 7ed4b42 Compare November 26, 2019 18:59

antoninbas mentioned this pull request Nov 26, 2019

add e2e network policy test #140

Closed

antoninbas force-pushed the fix-kind-support branch from 7ed4b42 to 676797b Compare November 27, 2019 18:25

antoninbas mentioned this pull request Nov 27, 2019

Fix typo in kind.md #150

Closed

antoninbas force-pushed the fix-kind-support branch from 676797b to c7c497d Compare November 27, 2019 18:51

antoninbas force-pushed the fix-kind-support branch from c7c497d to fd42001 Compare November 27, 2019 18:57

jianjuns previously approved these changes Nov 27, 2019

View reviewed changes

antoninbas dismissed jianjuns’s stale review via d825508 November 27, 2019 20:27

antoninbas force-pushed the fix-kind-support branch from fd42001 to d825508 Compare November 27, 2019 20:27

antoninbas requested a review from jianjuns November 27, 2019 20:27

jianjuns approved these changes Nov 27, 2019

View reviewed changes

antoninbas merged commit 089b2d0 into antrea-io:master Nov 27, 2019

antoninbas deleted the fix-kind-support branch November 27, 2019 21:01

This was referenced Nov 28, 2019

Support Kind on macOS hosts #155

Closed

Deleting Antrea should do a complete cleanup #181

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Kind support (Linux hosts only) #137

Fix Kind support (Linux hosts only) #137

antoninbas commented Nov 25, 2019 •

edited

Loading

antrea-bot commented Nov 25, 2019

antoninbas Nov 25, 2019

antoninbas commented Nov 25, 2019

antoninbas commented Nov 25, 2019

antoninbas commented Nov 26, 2019

antoninbas commented Nov 26, 2019

jianjuns left a comment

tnqn left a comment

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

jianjuns left a comment

antoninbas commented Nov 27, 2019

jianjuns commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

jianjuns left a comment

Fix Kind support (Linux hosts only) #137

Fix Kind support (Linux hosts only) #137

Conversation

antoninbas commented Nov 25, 2019 • edited Loading

antrea-bot commented Nov 25, 2019

antoninbas Nov 25, 2019

Choose a reason for hiding this comment

antoninbas commented Nov 25, 2019

antoninbas commented Nov 25, 2019

antoninbas commented Nov 26, 2019

antoninbas commented Nov 26, 2019

jianjuns left a comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

jianjuns left a comment

Choose a reason for hiding this comment

antoninbas commented Nov 27, 2019

jianjuns commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

antoninbas commented Nov 27, 2019

jianjuns left a comment

Choose a reason for hiding this comment

antoninbas commented Nov 25, 2019 •

edited

Loading