request NET_RAW capabilities in CNI manifests #2063

JingmingGuo · 2022-08-15T23:26:19Z

What type of PR is this?

feature

Which issue does this PR fix:
#2061

What does this PR do / Why do we need it:
This PR request NET_RAW Capabilities in CNI manifests. Linux NET_RAW capabilities let us use RAW and PACKET sockets; bind to any address for transparent proxying which is quite powerful. Our application request NET_RAW capabilities that calls
iptables, which opens up a SOCK_RAW netlink socket to configure
netfilter settings. It's likely the VPC CNI will continue needing NET_RAW in the future and to be used by other applications.

If an issue # is not available please add repro steps and logs from IPAMD/CNI showing the issue:

Testing done on this change:

--- PASS: TestSetupHostNetworkIgnoringRpFilterUpdate (0.00s)
=== RUN   TestSetupHostNetworkUpdateLocalRule
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network_test.go:852","msg":"Setting up host network... "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Trying to find primary interface that has mac : "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Discovered interface: lo, mac: "}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:280","msg":"Discovered primary interface: lo"}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network_test.go:852","msg":"Skip updating RPF for primary interface: net/ipv4/conf/lo/rp_filter"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:307","msg":"Found the Link that uses mac address  and its index is 0 (attempt 1/5)"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Trying to find primary interface that has mac : "}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Discovered interface: lo, mac: "}
{"level":"info","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:383","msg":"Discovered primary interface: lo"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Total CIDRs to program - 0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Setup Host Network: iptables -N AWS-SNAT-CHAIN-0 -t nat"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"Setup Host Network: iptables -A POSTROUTING -m comment --comment \"AWS SNAT CHAIN\" -j AWS-SNAT-CHAIN-0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:714","msg":"Setup Host Network: loading existing iptables nat rules with chain prefix AWS-SNAT-CHAIN"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:509","msg":"Setup Host Network: computing stale iptables rules for %s table with chain prefix %s"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:403","msg":"iptableRules: [nat/POSTROUTING rule first SNAT rules for non-VPC outbound traffic shouldExist false rule [-m comment --comment AWS SNAT CHAIN -j AWS-SNAT-CHAIN-0] nat/AWS-SNAT-CHAIN-0 rule last SNAT rule for non-VPC outbound traffic shouldExist false rule [! -o vlan+ -m comment --comment AWS, SNAT -m addrtype ! --dst-type LOCAL -j SNAT --to-source 10.10.10.20] mangle/PREROUTING rule connmark for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i lo -m addrtype --dst-type LOCAL --limit-iface-in -j CONNMARK --set-mark 0x80/0x80] mangle/PREROUTING rule connmark restore for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i eni+ -j CONNMARK --restore-mark --mask 0x80] mangle/PREROUTING rule connmark restore for primary ENI from vlan shouldExist true rule [-m comment --comment AWS, primary ENI -i vlan+ -j CONNMARK --restore-mark --mask 0x80]]"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : first SNAT rules for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule nat/POSTROUTING rule first SNAT rules for non-VPC outbound traffic shouldExist false rule [-m comment --comment AWS SNAT CHAIN -j AWS-SNAT-CHAIN-0] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : last SNAT rule for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule nat/AWS-SNAT-CHAIN-0 rule last SNAT rule for non-VPC outbound traffic shouldExist false rule [! -o vlan+ -m comment --comment AWS, SNAT -m addrtype ! --dst-type LOCAL -j SNAT --to-source 10.10.10.20] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark for primary ENI"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i lo -m addrtype --dst-type LOCAL --limit-iface-in -j CONNMARK --set-mark 0x80/0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark restore for primary ENI"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark restore for primary ENI shouldExist true rule [-m comment --comment AWS, primary ENI -i eni+ -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"execute iptable rule : connmark restore for primary ENI from vlan"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:407","msg":"rule mangle/PREROUTING rule connmark restore for primary ENI from vlan shouldExist true rule [-m comment --comment AWS, primary ENI -i vlan+ -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Total CIDRs to exempt from connmark rules - 0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Setup Host Network: iptables -N AWS-CONNMARK-CHAIN-0 -t nat"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"Setup Host Network: iptables -t nat -A PREROUTING -i eni+ -m comment --comment \"AWS, outbound connections\" -m state --state NEW -j AWS-CONNMARK-CHAIN-0"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:714","msg":"Setup Host Network: loading existing iptables nat rules with chain prefix AWS-CONNMARK-CHAIN"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:639","msg":"Setup Host Network: computing stale iptables rules for %s table with chain prefix %s"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:411","msg":"iptableRules: [nat/PREROUTING rule connmark rule for non-VPC outbound traffic shouldExist false rule [-i eni+ -m comment --comment AWS, outbound connections -m state --state NEW -j AWS-CONNMARK-CHAIN-0] nat/AWS-CONNMARK-CHAIN-0 rule connmark rule for external  outbound traffic shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --set-xmark 0x80/0x80] nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80]]"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark rule for non-VPC outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark rule for non-VPC outbound traffic shouldExist false rule [-i eni+ -m comment --comment AWS, outbound connections -m state --state NEW -j AWS-CONNMARK-CHAIN-0] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark rule for external  outbound traffic"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/AWS-CONNMARK-CHAIN-0 rule connmark rule for external  outbound traffic shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --set-xmark 0x80/0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark to fwmark copy"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"execute iptable rule : connmark to fwmark copy"}
{"level":"debug","ts":"2022-08-15T07:06:46.432Z","caller":"networkutils/network.go:415","msg":"rule nat/PREROUTING rule connmark to fwmark copy shouldExist false rule [-m comment --comment AWS, CONNMARK -j CONNMARK --restore-mark --mask 0x80] exists false, err <nil>"}
--- PASS: TestSetupHostNetworkUpdateLocalRule (0.00s)
PASS
coverage: 70.8% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/networkutils	0.025s	coverage: 70.8% of statements
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/networkutils/mocks	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/nswrapper	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/nswrapper/mocks	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/procsyswrapper	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/procsyswrapper/mocks	[no test files]
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"logger/logger.go:52","msg":"Constructed new logger instance"}
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"awssession/session.go:39","msg":"Initialized new logger as an existing instance was not found"}
=== RUN   TestCloudWatchPublisherWithNoIMDS
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"publisher/publisher_test.go:42","msg":"Constructed new logger instance"}
{"level":"warn","ts":"2022-08-15T07:06:47.504Z","caller":"awssession/session.go:64","msg":"HTTP_TIMEOUT env is not set or set to less than 10 seconds, defaulting to httpTimeout to 10sec"}
{"level":"info","ts":"2022-08-15T07:06:47.504Z","caller":"publisher/publisher_test.go:49","msg":"Using REGION=us-west-2 and CLUSTER_ID=TEST_CLUSTER_ID"}
--- PASS: TestCloudWatchPublisherWithNoIMDS (0.00s)
=== RUN   TestCloudWatchPublisherWithSingleDatum
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:63","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithSingleDatum (0.00s)
=== RUN   TestCloudWatchPublisherWithMultipleDatum
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:86","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithMultipleDatum (0.00s)
=== RUN   TestCloudWatchPublisherWithGreaterThanMaxDatapoints
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:108","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
--- PASS: TestCloudWatchPublisherWithGreaterThanMaxDatapoints (0.00s)
=== RUN   TestCloudWatchPublisherWithGreaterThanMaxDatapointsAndStop
{"level":"info","ts":"2022-08-15T07:06:47.505Z","caller":"publisher/publisher_test.go:129","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.516Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.516Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.526Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.536Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.547Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher.go:173","msg":"Missing data for publishing CloudWatch metrics"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher_test.go:136","msg":"Stopping monitor loop for CloudWatch publisher"}
{"level":"info","ts":"2022-08-15T07:06:47.556Z","caller":"publisher/publisher.go:220","msg":"Stopping monitor loop for CloudWatch publisher"}
--- PASS: TestCloudWatchPublisherWithGreaterThanMaxDatapointsAndStop (0.10s)
=== RUN   TestCloudWatchPublisherWithSingleDatumWithError
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:161","msg":"Fetching CloudWatch dimensions"}
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher.go:191","msg":"Sending data to CloudWatch metrics"}
{"level":"warn","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher.go:173","msg":"Unable to publish CloudWatch metrics: test error"}
--- PASS: TestCloudWatchPublisherWithSingleDatumWithError (0.00s)
=== RUN   TestGetCloudWatchMetricNamespace
--- PASS: TestGetCloudWatchMetricNamespace (0.00s)
=== RUN   TestGetCloudWatchMetricDatumDimensions
--- PASS: TestGetCloudWatchMetricDatumDimensions (0.00s)
=== RUN   TestGetCloudWatchMetricDatumDimensionsWithMissingClusterID
--- PASS: TestGetCloudWatchMetricDatumDimensionsWithMissingClusterID (0.00s)
=== RUN   TestPublishWithNoData
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:209","msg":"Fetching CloudWatch dimensions"}
--- PASS: TestPublishWithNoData (0.00s)
=== RUN   TestPushWithMissingData
{"level":"info","ts":"2022-08-15T07:06:47.608Z","caller":"publisher/publisher_test.go:217","msg":"Missing data for publishing CloudWatch metrics"}
--- PASS: TestPushWithMissingData (0.00s)
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 70.8% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/publisher	0.112s	coverage: 70.8% of statements
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/publisher/mock_publisher	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/rpcwrapper	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/rpcwrapper/mocks	[no test files]
=== RUN   TestBuildHostVethNamePrefix
=== RUN   TestBuildHostVethNamePrefix/standard_mode_should_use_configured_vethNamePrefix
=== RUN   TestBuildHostVethNamePrefix/strict_mode_should_use_vlan_vethNamePrefix
--- PASS: TestBuildHostVethNamePrefix (0.00s)
    --- PASS: TestBuildHostVethNamePrefix/standard_mode_should_use_configured_vethNamePrefix (0.00s)
    --- PASS: TestBuildHostVethNamePrefix/strict_mode_should_use_vlan_vethNamePrefix (0.00s)
=== RUN   TestLoadEnforcingModeFromEnv
=== RUN   TestLoadEnforcingModeFromEnv/use_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_strict
=== RUN   TestLoadEnforcingModeFromEnv/use_standard_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_standard
=== RUN   TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_not_set
=== RUN   TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_incorrectly_configured
--- PASS: TestLoadEnforcingModeFromEnv (0.00s)
    --- PASS: TestLoadEnforcingModeFromEnv/use_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_strict (0.00s)
    --- PASS: TestLoadEnforcingModeFromEnv/use_standard_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_set_to_standard (0.00s)
    --- PASS: TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_not_set (0.00s)
    --- PASS: TestLoadEnforcingModeFromEnv/default_to_strict_mode_when_POD_SECURITY_GROUP_ENFORCING_MODE_incorrectly_configured (0.00s)
PASS
coverage: 88.9% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/sgpp	0.008s	coverage: 88.9% of statements
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/typeswrapper	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/typeswrapper/mocks	[no test files]
=== RUN   Test_FindInterfaceByName
=== RUN   Test_FindInterfaceByName/found_the_CNI_interface_at_index_0
=== RUN   Test_FindInterfaceByName/found_the_CNI_interface_at_index_1
=== RUN   Test_FindInterfaceByName/didn't_found_CNI_interface
--- PASS: Test_FindInterfaceByName (0.00s)
    --- PASS: Test_FindInterfaceByName/found_the_CNI_interface_at_index_0 (0.00s)
    --- PASS: Test_FindInterfaceByName/found_the_CNI_interface_at_index_1 (0.00s)
    --- PASS: Test_FindInterfaceByName/didn't_found_CNI_interface (0.00s)
=== RUN   Test_FindIPConfigsByIfaceIndex
=== RUN   Test_FindIPConfigsByIfaceIndex/single_matched_IPConfig
=== RUN   Test_FindIPConfigsByIfaceIndex/multiple_matched_IPConfig
=== RUN   Test_FindIPConfigsByIfaceIndex/none_matched_IPConfig
=== RUN   Test_FindIPConfigsByIfaceIndex/interface_is_not_set
--- PASS: Test_FindIPConfigsByIfaceIndex (0.00s)
    --- PASS: Test_FindIPConfigsByIfaceIndex/single_matched_IPConfig (0.00s)
    --- PASS: Test_FindIPConfigsByIfaceIndex/multiple_matched_IPConfig (0.00s)
    --- PASS: Test_FindIPConfigsByIfaceIndex/none_matched_IPConfig (0.00s)
    --- PASS: Test_FindIPConfigsByIfaceIndex/interface_is_not_set (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/cniutils	0.024s	coverage: 100.0% of statements
{"level":"info","ts":"2022-08-15T07:06:47.752Z","caller":"logger/logger.go:52","msg":"Constructed new logger instance"}
{"level":"info","ts":"2022-08-15T07:06:47.753Z","caller":"k8sapi/k8sutils.go:23","msg":"Initialized new logger as an existing instance was not found"}
=== RUN   TestBroadcastEvents
{"level":"debug","ts":"2022-08-15T07:06:47.763Z","caller":"eventrecorder/eventrecorder_test.go:99","msg":"Broadcasting event on pod mockPodWithLabelAndSpec"}
--- PASS: TestBroadcastEvents (0.01s)
PASS
coverage: 34.6% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/eventrecorder	0.035s	coverage: 34.6% of statements
=== RUN   TestEnvLogFilePath
--- PASS: TestEnvLogFilePath (0.00s)
=== RUN   TestLoggerGetSameInstance
--- PASS: TestLoggerGetSameInstance (0.00s)
=== RUN   TestLoggerNewAndGetSameInstance
--- PASS: TestLoggerNewAndGetSameInstance (0.00s)
=== RUN   TestGetLogFileLocationReturnsDefaultPath
--- PASS: TestGetLogFileLocationReturnsDefaultPath (0.00s)
=== RUN   TestLogLevelReturnsOverriddenLevel
--- PASS: TestLogLevelReturnsOverriddenLevel (0.00s)
=== RUN   TestLogLevelReturnsDefaultLevelWhenEnvNotSet
--- PASS: TestLogLevelReturnsDefaultLevelWhenEnvNotSet (0.00s)
=== RUN   TestLogLevelReturnsDefaultLevelWhenEnvSetToInvalidValue
--- PASS: TestLogLevelReturnsDefaultLevelWhenEnvSetToInvalidValue (0.00s)
=== RUN   TestGetPluginLogFilePathEmpty
--- PASS: TestGetPluginLogFilePathEmpty (0.00s)
=== RUN   TestGetPluginLogFilePathStdout
--- PASS: TestGetPluginLogFilePathStdout (0.00s)
=== RUN   TestGetPluginLogFilePath
--- PASS: TestGetPluginLogFilePath (0.00s)
PASS
coverage: 62.5% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/logger	0.008s	coverage: 62.5% of statements
=== RUN   TestSimpleBackoff
--- PASS: TestSimpleBackoff (0.00s)
=== RUN   TestJitter
--- PASS: TestJitter (0.00s)
=== RUN   TestRetryWithBackoff
=== RUN   TestRetryWithBackoff/retries
=== RUN   TestRetryWithBackoff/no_retries
--- PASS: TestRetryWithBackoff (0.00s)
    --- PASS: TestRetryWithBackoff/retries (0.00s)
    --- PASS: TestRetryWithBackoff/no_retries (0.00s)
=== RUN   TestRetryWithBackoffCtx
=== RUN   TestRetryWithBackoffCtx/retries
=== RUN   TestRetryWithBackoffCtx/no_retries
=== RUN   TestRetryWithBackoffCtx/cancel_context
--- PASS: TestRetryWithBackoffCtx (0.00s)
    --- PASS: TestRetryWithBackoffCtx/retries (0.00s)
    --- PASS: TestRetryWithBackoffCtx/no_retries (0.00s)
    --- PASS: TestRetryWithBackoffCtx/cancel_context (0.00s)
=== RUN   TestRetryNWithBackoff
=== RUN   TestRetryNWithBackoff/count_exceeded
=== RUN   TestRetryNWithBackoff/retry_succeeded
--- PASS: TestRetryNWithBackoff (0.00s)
    --- PASS: TestRetryNWithBackoff/count_exceeded (0.00s)
    --- PASS: TestRetryNWithBackoff/retry_succeeded (0.00s)
=== RUN   TestRetryNWithBackoffCtx
=== RUN   TestRetryNWithBackoffCtx/count_exceeded
=== RUN   TestRetryNWithBackoffCtx/retry_succeeded
=== RUN   TestRetryNWithBackoffCtx/cancel_context
--- PASS: TestRetryNWithBackoffCtx (0.00s)
    --- PASS: TestRetryNWithBackoffCtx/count_exceeded (0.00s)
    --- PASS: TestRetryNWithBackoffCtx/retry_succeeded (0.00s)
    --- PASS: TestRetryNWithBackoffCtx/cancel_context (0.00s)
PASS
coverage: 100.0% of statements
ok  	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/retry	0.004s	coverage: 100.0% of statements
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/utils/ttime/mocks	[no test files]
?   	github.com/aws/amazon-vpc-cni-k8s/pkg/version	[no test files]

Automation added to e2e:

Will this PR introduce any new dependencies?:

No.

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
No.

Does this change require updates to the CNI daemonset config files to work?:

Does this PR introduce any user-facing change?:

AWS VPC CNI requests NET_RAW capabilities by default.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

jayanthvn · 2022-08-15T23:35:14Z

CAP_NET_ADMIN
              Perform various network-related operations:
              * interface configuration;
              * administration of IP firewall, masquerading, and
                accounting;
              * modify routing tables;
              * bind to any address for transparent proxying;
              * set type-of-service (TOS);
              * clear driver statistics;
              * set promiscuous mode;
              * enabling multicasting;
              * use [setsockopt(2)](https://man7.org/linux/man-pages/man2/setsockopt.2.html) to set the following socket options:
                SO_DEBUG, SO_MARK, SO_PRIORITY (for a priority outside
                the range 0 to 6), SO_RCVBUFFORCE, and SO_SNDBUFFORCE.

CAP_NET_RAW
              * Use RAW and PACKET sockets;
              * bind to any address for transparent proxying.

The only additional capability will be this - "Use RAW and PACKET sockets;".

Thanks for the PR, we will look into it.

jayanthvn · 2022-08-15T23:57:12Z

Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?

renan-airbnb · 2022-08-16T21:17:27Z

@jayanthvn, I was the one that found this issue while working on internal stuff so I wanted to chime in.

This won't change the default permissions for anyone. If CAP_NET is not provided either explicitly or as part of the default permissions, aws-vpc-cni will simply not work. We wanted to surface this to make sure the permission is explicitly requested so there's no issues using runtimes that provide more restrictive permissions by default 😄.

Edit - Forgot to say, I'm from the same team as @JingmingGuo

JingmingGuo · 2022-08-17T02:03:29Z

Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?

Thanks for pointing! Yes we should add this additional capabilities for every user by default. I updated and I'm looking forward to the reply.

jayanthvn · 2022-08-17T03:18:50Z

Also instead of updating in the sample manifests since this would add additional capabilities for every user by default, will you be able to use helm and you can either update in https://github.com/aws/amazon-vpc-cni-k8s/blob/master/charts/aws-vpc-cni/values.yaml or use "set" to include the capability?

Thanks for pointing! Yes we should add this additional capabilities for every user by default. I updated and I'm looking forward to the reply.

Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.

S-Chan · 2022-08-19T18:46:31Z

Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.

Thanks for the response @jayanthvn!

I did a little testing (v1.10.2 of the plugin) to see if I could get the plugin running with NET_RAW explicitly dropped. Here's my securityContext:

        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            drop:
            - NET_RAW

When I set ENABLE_IPv4 set to "true" and ENABLE_IPv6 set to "false" I see the following logs from ipamd:

{"level":"error","ts":"2022-08-19T18:19:03.111Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}

I also tried the reversed configuration (enabling IPv6 and disabling IPv4) and got the following logs:

{"level":"error","ts":"2022-08-19T18:22:54.759Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: failed to enable IPv6: setupVeth network: failed to setup route to block pod access via IPv4 address: failed adding v4 drop route: running [/usr/sbin/iptables -t filter -I FORWARD 1 -d 169.254.172.0/22 -m conntrack --ctstate NEW -m comment --comment Block Node Local Pod access via IPv4 -j REJECT --wait]: exit status 3: conntrack: Could not determine whether revision 1 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 1 is supported, assuming it is.\niptables v1.8.4 (legacy): can't initialize iptables table `filter': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}

In both cases the logs seem to indicate https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.10.2/pkg/ipamd/ipamd.go#L457-L461 is causing the error. Just glancing at the code, I don't see any way to configure the plugin to avoid invoking iptables, which seems to require NET_RAW. Let me know if you can reproduce.

I didn't exhaustively test configuration, so I might be missing a case where iptables is not used.

jayanthvn · 2022-08-19T19:17:20Z

Sorry for the confusion, I meant not everyone would need NET_RAW capability. I feel this has to be added on need basis. Please let me know your thoughts.

Thanks for the response @jayanthvn!

I did a little testing (v1.10.2 of the plugin) to see if I could get the plugin running with NET_RAW explicitly dropped. Here's my securityContext:
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            drop:
            - NET_RAW
When I set ENABLE_IPv4 set to "true" and ENABLE_IPv6 set to "false" I see the following logs from ipamd:
{"level":"error","ts":"2022-08-19T18:19:03.111Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}
I also tried the reversed configuration (enabling IPv6 and disabling IPv4) and got the following logs:
{"level":"error","ts":"2022-08-19T18:22:54.759Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: ipamd init: failed to set up host network: failed to enable IPv6: setupVeth network: failed to setup route to block pod access via IPv4 address: failed adding v4 drop route: running [/usr/sbin/iptables -t filter -I FORWARD 1 -d 169.254.172.0/22 -m conntrack --ctstate NEW -m comment --comment Block Node Local Pod access via IPv4 -j REJECT --wait]: exit status 3: conntrack: Could not determine whether revision 1 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 3 is supported, assuming it is.\nconntrack: Could not determine whether revision 2 is supported, assuming it is.\nconntrack: Could not determine whether revision 1 is supported, assuming it is.\niptables v1.8.4 (legacy): can't initialize iptables table `filter': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}
In both cases the logs seem to indicate https://github.com/aws/amazon-vpc-cni-k8s/blob/v1.10.2/pkg/ipamd/ipamd.go#L457-L461 is causing the error. Just glancing at the code, I don't see any way to configure the plugin to avoid invoking iptables, which seems to require NET_RAW. Let me know if you can reproduce.

I didn't exhaustively test configuration, so I might be missing a case where iptables is not used.

Yes the plugin will require iptables and there is no scenario where we won't use.

regarding your use case -> "Our application request NET_RAW capabilities that calls
iptables, which opens up a SOCK_RAW netlink socket to configure
netfilter setting" and NET_RAW adds additional capability of "* Use RAW and PACKET sockets;" for the existing NET_ADMIN which would not be required to be added by default for all applications.

If you are on K8S slack, we can sync up on a time slot and get on a call to further discuss about this.

JingmingGuo · 2022-09-08T02:35:07Z

Thanks for taking the time to set the discussion meeting with us today. @S-Chan explained the reason and the fact why everyone needs NET_RAW. I just put the summary here for easier tracking:

AWS VPC CNI k8s use coreos/go-iptables/iptables for networking in multiple places(e.g.

amazon-vpc-cni-k8s/pkg/networkutils/network.go

Line 40 in d43309b

"github.com/coreos/go-iptables/iptables"

) which coreos/go-iptables/iptables trigger system iptables call: https://github.com/coreos/go-iptables/blob/d2b8608923d15b0800af7d9f4bb6dea90e03b7d5/iptables/iptables.go#L564-L570. In Amazon Linux AMI:

yum info iptables
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd

Installed Packages
Name        : iptables
Arch        : x86_64
Version     : 1.8.4
Release     : 10.amzn2.1.2
Size        : 1.6 M
Repo        : installed
Summary     : Tools for managing Linux kernel packet filtering capabilities
URL         : http://www.netfilter.org/
License     : GPLv2 and Artistic 2.0 and ISC
Description : The iptables utility controls the network packet filtering code in the
            : Linux kernel. If you need to set up firewalls and/or IP masquerading,
            : you should install this package.

Netfilter iptables includes libiptc: https://git.netfilter.org/iptables/tree/include/iptables.h#n6 which uses RAW sockets: https://git.netfilter.org/iptables/tree/libiptc/libiptc.c#n1312.

So that all the users use AWS VPC CNI k8s will need the CAP_NET_RAW which is reasonable to set as default. Also prevent the failures if users didn't notice it.

sushrk · 2022-10-14T17:01:24Z

Hi @JingmingGuo, thanks for the PR!

I was able to verify that NET_RAW capability is required in aws-node by experimenting with these values:

securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
            drop:
            - ALL

When I removed NET_RAW, ipamd fails to initialize with the following error(as already pointed out):

{"level":"error","ts":"2022-10-14T15:06:20.300Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd init: failed to set up host network: host network setup: failed to add chain: running [/usr/sbin/iptables -t nat -N AWS-SNAT-CHAIN-0 --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)\nPerhaps iptables or your kernel needs to be upgraded.\n"}

Also verified that NET_RAW capability is included by default in docker & containerd runtimes, see:

kubectl exec -it -n kube-system aws-node-r476b -- capsh --print
Defaulted container "aws-node" out of: aws-node, aws-vpc-cni-init (init)
Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Ambient set =
Current IAB: !cap_dac_read_search,!cap_linux_immutable,!cap_net_broadcast,!cap_ipc_lock,!cap_ipc_owner,!cap_sys_module,!cap_sys_rawio,!cap_sys_ptrace,!cap_sys_pacct,!cap_sys_admin,!cap_sys_boot,!cap_sys_nice,!cap_sys_resource,!cap_sys_time,!cap_sys_tty_config,!cap_lease,!cap_audit_control,!cap_mac_override,!cap_mac_admin,!cap_syslog,!cap_wake_alarm,!cap_block_suspend,!cap_audit_read
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=
Guessed mode: UNCERTAIN (0)

@renan-airbnb 's comment is valid:

We wanted to surface this to make sure the permission is explicitly requested so there's no issues using runtimes that provide more restrictive permissions by default 😄.

We will need to include NET_RAW to ensure that aws-node can work on runtimes that don't have it by default.
For example, cri-o does not include this by default cri-o/cri-o#3119

sushrk · 2022-10-14T18:55:17Z

@JingmingGuo can you please update the branch?

JingmingGuo · 2022-10-14T20:17:04Z

@JingmingGuo can you please update the branch?

Sure I updated the branch.

charts/aws-vpc-cni/README.md

config/master/aws-k8s-cni-cn.yaml

config/master/aws-k8s-cni-us-gov-east-1.yaml

config/master/aws-k8s-cni-us-gov-west-1.yaml

config/master/aws-k8s-cni.yaml

sushrk · 2022-10-17T17:54:05Z

@JingmingGuo thanks for updating, but you might still need to run make check-format as unit tests are failing here

JingmingGuo · 2022-10-25T20:49:53Z

@JingmingGuo thanks for updating, but you might still need to run make check-format as unit tests are failing here

Thanks for pointing! I run 'make format' and updated the PR. See the checks have passed now.

sushrk

Thanks, lgtm!

* request NET_RAW capabilities in CNI manifests * add request NET_RAW capabilities in chart and jsonnet * update format

JingmingGuo requested a review from a team as a code owner August 15, 2022 23:26

jayanthvn requested a review from sushrk October 11, 2022 20:16

jdn5126 assigned sushrk Oct 12, 2022

jdn5126 reviewed Oct 14, 2022

View reviewed changes

JingmingGuo force-pushed the net_raw branch 2 times, most recently from 4a47946 to f22b4d1 Compare October 14, 2022 21:33

jdn5126 force-pushed the net_raw branch from f22b4d1 to f93bfd3 Compare October 20, 2022 20:32

JingmingGuo added 3 commits October 24, 2022 17:01

request NET_RAW capabilities in CNI manifests

f161a69

add request NET_RAW capabilities in chart and jsonnet

0d24c61

update format

082f07a

JingmingGuo force-pushed the net_raw branch from 49e55a8 to 082f07a Compare October 25, 2022 00:02

sushrk approved these changes Oct 25, 2022

View reviewed changes

sushrk merged commit 9bee3e4 into aws:master Oct 25, 2022

jdn5126 pushed a commit to jdn5126/amazon-vpc-cni-k8s that referenced this pull request Oct 25, 2022

request NET_RAW capabilities in CNI manifests (aws#2063)

5429f38

* request NET_RAW capabilities in CNI manifests * add request NET_RAW capabilities in chart and jsonnet * update format

This was referenced Oct 28, 2022

🥳 aws-vpc-cni v1.12.0 Automated Release! 🥑 aws/eks-charts#835

Closed

🥳 aws-vpc-cni v1.12.0 Automated Release! 🥑 aws/eks-charts#836

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

request NET_RAW capabilities in CNI manifests #2063

request NET_RAW capabilities in CNI manifests #2063

JingmingGuo commented Aug 15, 2022

jayanthvn commented Aug 15, 2022 •

edited

jayanthvn commented Aug 15, 2022

renan-airbnb commented Aug 16, 2022 •

edited

JingmingGuo commented Aug 17, 2022

jayanthvn commented Aug 17, 2022 •

edited

S-Chan commented Aug 19, 2022 •

edited

jayanthvn commented Aug 19, 2022

JingmingGuo commented Sep 8, 2022

sushrk commented Oct 14, 2022

sushrk commented Oct 14, 2022

JingmingGuo commented Oct 14, 2022

sushrk commented Oct 17, 2022

JingmingGuo commented Oct 25, 2022

sushrk left a comment

request NET_RAW capabilities in CNI manifests #2063

request NET_RAW capabilities in CNI manifests #2063

Conversation

JingmingGuo commented Aug 15, 2022

jayanthvn commented Aug 15, 2022 • edited

jayanthvn commented Aug 15, 2022

renan-airbnb commented Aug 16, 2022 • edited

JingmingGuo commented Aug 17, 2022

jayanthvn commented Aug 17, 2022 • edited

S-Chan commented Aug 19, 2022 • edited

jayanthvn commented Aug 19, 2022

JingmingGuo commented Sep 8, 2022

sushrk commented Oct 14, 2022

sushrk commented Oct 14, 2022

JingmingGuo commented Oct 14, 2022

sushrk commented Oct 17, 2022

JingmingGuo commented Oct 25, 2022

sushrk left a comment

Choose a reason for hiding this comment

jayanthvn commented Aug 15, 2022 •

edited

renan-airbnb commented Aug 16, 2022 •

edited

jayanthvn commented Aug 17, 2022 •

edited

S-Chan commented Aug 19, 2022 •

edited