Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native nftables dataplane #8780

Merged
merged 58 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
99b8610
Initial nftables prototype implementation
caseydavenport Mar 18, 2024
e95cb6a
Add NFTables to FelixConfiguration, fix up UTs
caseydavenport May 2, 2024
b4adaae
Slightly better node Dockerfile implementation
caseydavenport May 20, 2024
11e1c8f
Properly clean up chains that aren't ours
caseydavenport May 21, 2024
d7b99ad
Handle overlapping ranges in nftables sets
caseydavenport May 21, 2024
8e964b8
Update match builder to use meta l4proto
caseydavenport May 21, 2024
99f311d
Start getting FV tests to run
caseydavenport May 21, 2024
64b9d31
Add triemasker prototype
caseydavenport Jun 5, 2024
3f54cc1
Fix overlapping network sets in nftables mode
caseydavenport Jun 5, 2024
230bc62
Fix node.get
caseydavenport Jun 7, 2024
1c7375d
Fix deduplicator in dual stack
caseydavenport Jun 7, 2024
9e4b9e5
Add UT for member deduplicator
caseydavenport Jun 7, 2024
1e3eb8d
Fix health tests for nft
caseydavenport Jun 7, 2024
f4aa164
HEP test fixed
caseydavenport Jun 7, 2024
fbce58c
Enable XDP :|
caseydavenport Jun 7, 2024
d565d72
Fix packet check
caseydavenport Jun 7, 2024
2675bd1
Remove remaining bookmarks
caseydavenport Jun 7, 2024
27693e4
Fix IP set deletion in deduplicator
caseydavenport Jun 10, 2024
9882fca
Use correct funcs for network sets FVs
caseydavenport Jun 11, 2024
a6b4595
Fix ipset command in nft mode
caseydavenport Jun 12, 2024
5082e87
Fix test
caseydavenport Jun 12, 2024
6fb158d
Fix ipv6 vxlan check
caseydavenport Jun 12, 2024
e42d07e
REVERT: Target flapping test in CI
caseydavenport Jun 12, 2024
657a91a
Fix VXLAN IPv6 set name
caseydavenport Jun 13, 2024
4857a66
Add log for missing IP set
caseydavenport Jun 13, 2024
5f9ba6f
Delete multiple IP sets at once
caseydavenport Jun 14, 2024
850deb1
Code review pt. 1
caseydavenport Jun 18, 2024
716d6a8
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jun 20, 2024
b230f1b
Further code review addressing
caseydavenport Jun 20, 2024
41cea00
Finish up postWriteInterval changes
caseydavenport Jun 21, 2024
2ea25a8
Rip out postWriteInterval logic, not needed in nftables
caseydavenport Jun 21, 2024
9cf774c
Use NumOperations()
caseydavenport Jun 21, 2024
49f63fc
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jun 21, 2024
33724ea
Fix build
caseydavenport Jun 21, 2024
ca785bc
Fix FV tests using wrong enablement var
caseydavenport Jun 21, 2024
7c88a4a
Fix arm64 build
caseydavenport Jun 22, 2024
8e3e474
Update generated files
caseydavenport Jun 22, 2024
634ad56
Add FV for nftables + bpf
caseydavenport Jun 24, 2024
df7c772
Fix IPSet naming bug
caseydavenport Jun 25, 2024
7972ae3
Update timeout for nftables mode
caseydavenport Jun 25, 2024
58451b3
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jun 26, 2024
cc6df21
Fix up XDP FV
caseydavenport Jun 26, 2024
4f00325
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jun 26, 2024
5cd8c7a
Fix build error
caseydavenport Jun 26, 2024
c3bfb8c
Switch off of knftables fork
caseydavenport Jun 27, 2024
be69b1d
Fix ipset resync performance
caseydavenport Jun 27, 2024
e38b4a3
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jun 28, 2024
5817a94
Disable resync for nftables set FV
caseydavenport Jun 28, 2024
33ebc1a
Disable ipset resync during FV
caseydavenport Jul 1, 2024
a8a0b4e
Fix that some protocol aliases don't exist on all system
caseydavenport Jul 1, 2024
8308a7e
use longer timeout for now
caseydavenport Jul 1, 2024
4b29f1c
timout -> timeout
caseydavenport Jul 2, 2024
7d37e97
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jul 2, 2024
b9f9f62
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jul 3, 2024
407fe9a
Fix reject with for nftables
caseydavenport Jul 3, 2024
dd920cc
Make fix
caseydavenport Jul 3, 2024
c1332c7
Merge remote-tracking branch 'origin/master' into casey-nft-proto
caseydavenport Jul 8, 2024
53da624
Fix UT post merge
caseydavenport Jul 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .semaphore/semaphore-scheduled-builds.yml
Original file line number Diff line number Diff line change
Expand Up @@ -478,6 +478,16 @@ blocks:
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
- name: nftables FV Test matrix
execution_time_limit:
minutes: 120
env_vars:
- name: FELIX_FV_NFTABLES
value: "true"
commands:
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
epilogue:
always:
commands:
Expand Down
10 changes: 10 additions & 0 deletions .semaphore/semaphore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -478,6 +478,16 @@ blocks:
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
- name: nftables FV Test matrix
execution_time_limit:
minutes: 120
env_vars:
- name: FELIX_FV_NFTABLES
value: "true"
commands:
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
epilogue:
always:
commands:
Expand Down
10 changes: 10 additions & 0 deletions .semaphore/semaphore.yml.d/blocks/20-felix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,16 @@
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
- name: nftables FV Test matrix
execution_time_limit:
minutes: 120
env_vars:
- name: FELIX_FV_NFTABLES
value: "true"
caseydavenport marked this conversation as resolved.
Show resolved Hide resolved
commands:
- make check-wireguard
- ../.semaphore/run-and-monitor fv-${SEMAPHORE_JOB_INDEX}.log make fv-no-prereqs FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
epilogue:
always:
commands:
Expand Down
10 changes: 10 additions & 0 deletions api/pkg/apis/projectcalico/v3/felixconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,13 @@ const (
IptablesBackendAuto = "Auto"
)

type NFTablesMode string

const (
NFTablesModeEnabled = "Enabled"
NFTablesModeDisabled = "Disabled"
)

// +kubebuilder:validation:Enum=DoNothing;Enable;Disable
type AWSSrcDstCheckOption string

Expand Down Expand Up @@ -439,6 +446,9 @@ type FelixConfigurationSpec struct {
// iptables. [Default: false]
GenericXDPEnabled *bool `json:"genericXDPEnabled,omitempty" confignamev1:"GenericXDPEnabled"`

// NFTablesMode configures nftables support in Felix. [Default: Disabled]
NFTablesMode *NFTablesMode `json:"nftablesMode,omitempty"`

// BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]
BPFEnabled *bool `json:"bpfEnabled,omitempty" validate:"omitempty"`
// BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable
Expand Down
5 changes: 5 additions & 0 deletions api/pkg/apis/projectcalico/v3/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions api/pkg/openapi/openapi_generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion calicoctl/calicoctl/commands/crds/crds.go

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions felix/.semaphore/semaphore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,13 @@ blocks:
- make check-wireguard
- make fv FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
- name: NFT FV Test matrix
execution_time_limit:
minutes: 120
commands:
- make check-wireguard
- make fv-nft FV_BATCHES_TO_RUN="${SEMAPHORE_JOB_INDEX}" FV_NUM_BATCHES=${SEMAPHORE_JOB_COUNT}
parallelism: 3
epilogue:
always:
commands:
Expand Down
5 changes: 5 additions & 0 deletions felix/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ fv-no-prereqs:
GINKGO_FOCUS="$(GINKGO_FOCUS)" \
ARCH=$(ARCH) \
FELIX_FV_ENABLE_BPF="$(FELIX_FV_ENABLE_BPF)" \
FELIX_FV_NFTABLES="$(FELIX_FV_NFTABLES)" \
FV_RACE_DETECTOR_ENABLED=$(FV_RACE_DETECTOR_ENABLED) \
FV_BINARY=$(FV_BINARY) \
FELIX_FV_WIREGUARD_AVAILABLE=`./wireguard-available >/dev/null && echo true || echo false` \
Expand All @@ -352,6 +353,9 @@ fv-no-prereqs:
fv-bpf:
$(MAKE) fv FELIX_FV_ENABLE_BPF=true

fv-nft:
$(MAKE) fv FELIX_FV_NFTABLES=true

check-wireguard:
fv/wireguard-available || ( echo "WireGuard not available."; exit 1 )

Expand Down Expand Up @@ -391,6 +395,7 @@ k8sfv-test: $(REMOTE_DEPS) $(FELIX_CONTAINER_CREATED) $(FV_BINARY) bin/k8sfv.tes
FV_FELIXIMAGE=$(FV_FELIXIMAGE) \
FV_K8SIMAGE=$(FV_K8SIMAGE) \
FV_BINARY=$(FV_BINARY) \
FELIX_FV_NFTABLES=$(FELIX_FV_NFTABLES) \
CERTS_PATH=$(CERTS_PATH) \
PRIVATE_KEY=`pwd`/fv/private.key \
k8sfv/run-test
Expand Down
3 changes: 2 additions & 1 deletion felix/calc/calc_graph.go
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ func NewCalculationGraph(callbacks PipelineCallbacks, conf *config.Config, liveC
// |
// <dataplane>
//
ipsetMemberIndex := labelindex.NewSelectorAndNamedPortIndex()
ipsetMemberIndex := labelindex.NewSelectorAndNamedPortIndex(conf.NFTablesMode == "Enabled")
ipsetMemberIndex.OnAlive = liveCallback
// Wire up the inputs to the IP set member index.
ipsetMemberIndex.RegisterWith(allUpdDispatcher)
Expand All @@ -311,6 +311,7 @@ func NewCalculationGraph(callbacks PipelineCallbacks, conf *config.Config, liveC
callbacks.OnIPSetRemoved(ipSet.UniqueID())
gaugeNumActiveSelectors.Dec()
}

// Send the IP set member index's outputs to the dataplane.
ipsetMemberIndex.OnMemberAdded = func(ipSetID string, member labelindex.IPSetMember) {
if log.GetLevel() >= log.DebugLevel {
Expand Down
4 changes: 4 additions & 0 deletions felix/config/config_params.go
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,10 @@ type Config struct {
WireguardHostEncryptionEnabled bool `config:"bool;false"`
WireguardPersistentKeepAlive time.Duration `config:"seconds;0"`

// nftables configuration.
NFTablesMode string `config:"oneof(Enabled,Disabled);Disabled"`

// BPF configuration.
BPFEnabled bool `config:"bool;false"`
BPFDisableUnprivileged bool `config:"bool;true"`
BPFLogLevel string `config:"oneof(off,info,debug);off;non-zero"`
Expand Down
1 change: 1 addition & 0 deletions felix/dataplane/common/ipsets_mgr.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ type IPSetsDataplane interface {
QueueResync()
ApplyUpdates()
ApplyDeletions() (reschedule bool)
SetFilter(neededIPSets set.Set[string])
}

// Except for domain IP sets, IPSetsManager simply passes through IP set updates from the datastore
Expand Down
5 changes: 3 additions & 2 deletions felix/dataplane/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ func StartDataplaneDriver(configParams *config.Config,
healthAggregator *health.HealthAggregator,
configChangedRestartCallback func(),
fatalErrorCallback func(error),
k8sClientSet *kubernetes.Clientset) (DataplaneDriver, *exec.Cmd) {

k8sClientSet *kubernetes.Clientset,
) (DataplaneDriver, *exec.Cmd) {
if !configParams.IsLeader() {
// Return an inactive dataplane, since we're not the leader.
log.Info("Not the leader, using an inactive dataplane")
Expand Down Expand Up @@ -207,6 +207,7 @@ func StartDataplaneDriver(configParams *config.Config,
NetlinkTimeout: configParams.NetlinkTimeoutSecs,
},
RulesConfig: rules.Config{
NFTables: configParams.NFTablesMode == "Enabled",
WorkloadIfacePrefixes: configParams.InterfacePrefixes(),

IPSetConfigV4: ipsets.NewIPVersionConfig(
Expand Down
Loading