Implement the controller for API BGPPolicy #6203

hongliangl · 2024-04-09T02:45:02Z

This commit implements the controller of API BGPPolicy, designed to advertise
Service IPs, Egress IPs, and Pod IPs to BGP peers from selected Nodes.

According to the spec of BGPPolicy, the Node selector is used to select Nodes
to which a BGPPolicy is applied. Multiple BGPPolicies can be applied to the
same Node. However, only one BGPPolicy can be effective on a Node, with others
serving as alternatives. The first BGPPolicy applied to a Node will be the
effective one, and the latter ones will serve as alternatives. The effective one
may be changed in the following cases:

The current effective BGPPolicy is updated and not applied to the Node.
The current effective BGPPolicy is deleted.
The antrea-agent is rebooted, and an original alternative BGPPolicy is synced
first and becomes effective.

The BGP server instance is only started for the effective BGPPolicy on a Node.
Currently, the BGP server instance is implemented by goBGP, and more BGP
server implementations might be added later. If the effective BGPPolicy is changed,
the corresponding BGP server instance will be terminated by calling the Stop()
method, and a new BGP server instance will be created and started by calling the
Start() method for the new effective BGPPolicy.

To start a BGP server instance, ASN, routerID, and listen port must be specified.
ASN and listen port are specified in the spec of the effective BGPPolicy. For
routerID, if the cluster is IPv4 only or dual stack, the IPv4 NodeIP is used as
the routerID; if the cluster is IPv6 only, the routerID must be specified in the
Node annotation antrea.io/bgp-route-id. Additionally, a new BGP server instance
should be created and started when any of ASN, routerID, or listen port changes.

The information of the BGP peers is specified in the effective BGPPolicy. The
unique identification of a BGP peer is the peer IP address.

To reconcile the latest BGP peers:

Get the BGP peers to be added and add them by calling the AddPeers method of
the BGP server instance.
Get the BGP peers to be deleted and delete them by calling the RemovePeers
method of the BGP server instance.
Get the remaining BGP peers and calculate the updated BGP peers, then update
them by calling the UpdatePeers method of the BGP server instance.

The information of the IPs to be advertised can be calculated from the spec of
the effective BGPPolicy. Currently, we advertise the IPs and CIDRs to all the
BGP peers.

To reconcile the latest IPs to all BGP peers:

If the BGP server instance is newly created and started, advertise all the IPs
by calling the AdvertiseRoutes method.
If the BGP server instance is not newly created and started:
- Get the IPs/CIDRs to be added and advertise them by calling the
  AdvertiseRoutes method.
- Get the IPs/CIDRs to be removed and withdraw them by calling the
  WithdrawRoutes method.

The feature is gated by the alpha BGPPolicy feature gate.

Depends on #6009

luolanzone · 2024-06-25T09:46:57Z

Hi @hongliangl could you explain more about advertise Service IPs, Egress IPs, and Pod IPs to BGP peers from selected Nodes.? According to the API, there is enum of ipTypes, Does it mean users can choose what's kind of Service can be advertised? Will there be a default one? It would be better to provide a sample BGPPolicy and explain the behavior to help reviewers. Thanks.

hongliangl · 2024-06-25T09:48:54Z

Hi @hongliangl could you explain more about advertise Service IPs, Egress IPs, and Pod IPs to BGP peers from selected Nodes.? According to the API, there is enum of ipTypes, Does it mean users can choose what's kind of Service can be advertised? Will there be a default one? It would be better to provide a sample BGPPolicy and explain the behavior to help reviewers. Thanks.

will add

luolanzone · 2024-06-25T06:51:07Z

build/charts/antrea/conf/antrea-agent.conf

+bgpPolicy:
+  # The name of the Secret storing passwords of the BGP peers. For each BGP peer, the Secret key is generated by
+  # concatenating its IP address and AS number, e.g., `192.168.1.1-65521`.
+  secretName: {{ .Values.bgpPolicy.secretName | quote }}


Should we allow user to change this Secret name? I mean how would Antrea guarantee the ClusterRole being updated after this is changed?

If users only modify the name but don't update the ClusterRole, we don't have a good solution to resolve that. The error logs may help users. What we can do is to document it well to remind users. I will refine the comment here to remind users about that.

In other similar cases, we just hardcode the secret name and grant the secret's permission to agent, and no one has ever complained it. I think it could follow that, instead of making the secret name configurable.

pkg/agent/controller/networkpolicy/l7engine/reconciler.go

luolanzone · 2024-06-25T09:12:40Z

pkg/agent/controller/bgp/controller.go

+		}
+
+		// Stop the stale BGP server if it exists.
+		if bpState.bgpServer != nil {


Why to stop stale BGP server after new BGP server? Won't be port conflict if other configs changes but not port?

Good question! I should stop the server first.

pkg/agent/controller/bgp/controller.go

luolanzone · 2024-06-25T09:48:23Z

pkg/agent/controller/bgp/controller.go

+
+func (c *Controller) getRouterID() (string, error) {
+	var routerID string
+	// For IPv6 only environment, the BGP routerID should be specified by K8s Node annotation `antrea. io/ bgp-route-id`.


Suggested change

// For IPv6 only environment, the BGP routerID should be specified by K8s Node annotation `antrea. io/ bgp-route-id`.

// For IPv6 only environment, the BGP routerID should be specified by K8s Node annotation `antrea.io/bgp-route-id`.

Question: if this is provided by users, how to avoid duplicate IDs?

We cannot avoid duplicate IDs.

This commit implements the controller of API `BGPPolicy`, designed to advertise Service IPs, Egress IPs, and Pod IPs to BGP peers from selected Nodes. According to the spec of `BGPPolicy`, the Node selector is used to select Nodes to which a `BGPPolicy` is applied. Multiple `BGPPolicies` can be applied to the same Node. However, only one `BGPPolicy` can be effective on a Node, with others serving as alternatives. The first `BGPPolicy` applied to a Node will be the effective one, and the latter ones will serve as alternatives. The effective one may be changed in the following cases: - The current effective BGPPolicy is updated and not applied to the Node. - The current effective BGPPolicy is deleted. - The antrea-agent is rebooted, and an original alternative BGPPolicy is synced first and becomes effective. The BGP server instance is only started for the effective BGPPolicy on a Node. If the effective BGPPolicy is changed, the corresponding BGP server instance will be terminated by calling the `Stop` method, and a new BGP server instance will be created and started by calling the `Start` method for the new effective BGPPolicy. To start a BGP server instance, ASN, routerID, and listen port must be specified. ASN and listen port are specified in the spec of the effective BGPPolicy. For routerID, if the cluster is IPv4 only or dual stack, the IPv4 NodeIP is used as the routerID; if the cluster is IPv6 only, the routerID must be specified in the Node annotation `antrea.io/bgp-route-id`. Additionally, a new BGP server instance should be created and started when any of ASN, routerID, or listen port changes. The information of the BGP peers is specified in the effective BGPPolicy. The unique identification of a BGP peer is the peer IP address. To reconcile the latest BGP peers: - Get the BGP peers to be added and add them by calling the `AddPeer` method of the BGP server instance. - Get the BGP peers to be deleted and delete them by calling the `RemovePeer` method of the BGP server instance. - Get the remaining BGP peers and calculate the updated BGP peers, then update them by calling the `UpdatePeer` method of the BGP server instance. The information of the IPs to be advertised can be calculated from the spec of the effective BGPPolicy. Currently, we advertise the IPs and CIDRs to all the BGP peers. To reconcile the latest IPs to all BGP peers: - If the BGP server instance is newly created and started, advertise all the IPs by calling the `AdvertiseRoutes` method. - If the BGP server instance is not newly created and started: - Get the IPs/CIDRs to be added and advertise them by calling the `AdvertiseRoutes` method. - Get the IPs/CIDRs to be removed and withdraw them by calling the `WithdrawRoutes` method. The feature is gated by the alpha `BGPPolicy` feature gate. Signed-off-by: Hongliang Liu <lhongliang@vmware.com>

tnqn

I haven't finished reviewing all code, but I feel the patch spends too many words and code for the unexpected case: multiple policies applied to a Node, making it like a normal case.
It's not a meaningful configuration and we should discourage users to do it, and there is no need to explain how we deal with such configurations in detail. "We could just have a note for users: If more than one BGPPolicy applies to a Node, only one will take effect."

tnqn · 2024-07-03T03:36:01Z

build/charts/antrea/conf/antrea-agent.conf

+bgpPolicy:
+  # The name of the Secret storing passwords of the BGP peers. For each BGP peer, the Secret key is generated by
+  # concatenating its IP address and AS number, e.g., `192.168.1.1-65521`.
+  secretName: {{ .Values.bgpPolicy.secretName | quote }}


In other similar cases, we just hardcode the secret name and grant the secret's permission to agent, and no one has ever complained it. I think it could follow that, instead of making the secret name configurable.

tnqn · 2024-07-03T03:38:26Z

build/charts/antrea/templates/agent/bgp-secret.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Secret


why it needs to create the secret in advance?

Do you mean that we should let users create the Secret? Is it a convention or something else?

tnqn · 2024-07-03T03:42:08Z

docs/feature-gates.md

@@ -31,7 +31,7 @@ edit the Agent configuration in the
 ## List of Available Features

 | Feature Name                  | Component          | Default | Stage | Alpha Release | Beta Release | GA Release | Extra Requirements | Notes                                         |
-| ----------------------------- | ------------------ | ------- | ----- | ------------- | ------------ | ---------- | ------------------ | --------------------------------------------- |
+|-------------------------------| ------------------ | ------- | ----- |---------------| ------------ | ---------- |--------------------| --------------------------------------------- |


I'm not sure which is correct, having space or not having space. But this change makes them inconsistent.

WIll rever this.

tnqn · 2024-07-03T03:45:59Z

pkg/agent/controller/bgp/controller.go

+)
+
+const (
+	bgpRouteIDAnnotation = "antrea.io/bgp-route-id"


I will have comment to understand why we need annotation for IPv6 in another thread. But if this is really needed, please move it to pkg/agent/types/annotations.go and name it "bgp.antrea.io/router-id" (it should be router, not route, right?)

pkg/agent/controller/bgp/controller.go

tnqn · 2024-07-03T03:48:37Z

pkg/agent/controller/bgp/controller.go

+	newBGPServerFn = func(globalConfig *bgp.GlobalConfig) bgp.Interface {
+		return gobgp.NewGoBGPServer(globalConfig)
+	}


Suggested change

newBGPServerFn = func(globalConfig *bgp.GlobalConfig) bgp.Interface {

return gobgp.NewGoBGPServer(globalConfig)

}

newBGPServerFn = gobgp.NewGoBGPServer

And I think it's better to make it a member variable of Controller and assign gobgp.NewGoBGPServer to it by default, then you don't need to revert it in tests.

tnqn · 2024-07-03T03:53:45Z

pkg/agent/controller/bgp/controller.go

+	// peers maps IP families to concatenated strings of BGP peer IP addresses and ASNs.
+	// Example: "192.168.77.100-65000", "2001::1-65000".


The comment is wrong

Will update.

tnqn · 2024-07-03T03:54:54Z

pkg/agent/controller/bgp/controller.go

+	peerKeys sets.Set[string]
+	// peerConfigs maps concatenated string of BGP peer IP addresses and ASN to the configuration of the peer.
+	peerConfigs map[string]bgp.PeerConfig


Aren't peerKeys redundant with peerConfigs' keys?

peerKeys can be removed. If so, we still need to collect all the keys of the current and the stale peerConfigs to two sets, and then get the peers to be added or deleted.

It can be easily got by sets.KeySet(peerConfigs), it's not a common "optimization" to use a separate set to store another map's keys, which may not be even more efficient as it still needs to maintain an extra set when writing.

tnqn · 2024-07-03T04:06:47Z

pkg/agent/controller/bgp/controller.go

+	for i := 0; i < defaultWorkers; i++ {
+		go wait.Until(c.worker, time.Second, stopCh)
+	}


There is no point to process multiple policies in parallel and have many code to handle the race condition.
In normal scenario, we expect only zero or single policy applied to the Node, we should just get one policy and implement it, instead of dealing with multiple policies like it's a normal case. If all jobs of implementing a single policy should be done sequentially, it just needs a single worker with a dummy key.

The worker's implementation in my mind:

func syncBGPPolicy() error { // Get 0 or 1 policy applied to the Node. bgpPolicy, exists := c.getBGPPolicy() if !exists { // cleanup } // Implement the bgpPolicy }

How should we choose a BGPPolicy if users apply multiple ones to a Node anyway?

Is it hard to get one item from a list of items?

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from e986dd8 to 16f2901 Compare April 9, 2024 02:46

hongliangl added kind/feature Categorizes issue or PR as related to a new feature. action/release-note Indicates a PR that should be included in release notes. labels Apr 9, 2024

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from e93d476 to 7508c38 Compare April 10, 2024 07:20

rajnkamr added the area/transit/bgp Issues or PRs related to BGP support. label Apr 11, 2024

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from 7508c38 to e1e246a Compare May 22, 2024 05:30

hongliangl added this to the Antrea v2.1 release milestone May 24, 2024

hongliangl force-pushed the 20240222-bgp-controller-v2 branch 2 times, most recently from f209e2a to 3063737 Compare May 30, 2024 10:40

hongliangl force-pushed the 20240222-bgp-controller-v2 branch 3 times, most recently from 57773d9 to c9b727a Compare June 6, 2024 12:25

hongliangl mentioned this pull request Jun 14, 2024

Add BGP datapath interface and implement goBGP integration #6447

Merged

hongliangl force-pushed the 20240222-bgp-controller-v2 branch 11 times, most recently from 095d7da to 5086b34 Compare June 19, 2024 07:23

hongliangl marked this pull request as ready for review June 19, 2024 07:26

hongliangl force-pushed the 20240222-bgp-controller-v2 branch 2 times, most recently from 3bab09b to 69c31a2 Compare June 20, 2024 05:55

hongliangl changed the title ~~Add BGPPolicy controller~~ Implement the controller for API BGPPolicy Jun 20, 2024

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from 69c31a2 to 18c6f27 Compare June 21, 2024 10:38

hongliangl mentioned this pull request Jun 21, 2024

To support BGP stack integration #6189

Open

4 tasks

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from 18c6f27 to 57bd548 Compare June 25, 2024 03:10

luolanzone reviewed Jun 25, 2024

View reviewed changes

This was referenced Jun 26, 2024

Update docs/network-requirements.md #6482

Open

Add BGPPolicy to docs/api.md #6457

Open

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from 57bd548 to 11e6f39 Compare June 27, 2024 12:03

hongliangl force-pushed the 20240222-bgp-controller-v2 branch from 11e6f39 to 2e762c8 Compare June 28, 2024 05:25

tnqn reviewed Jul 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement the controller for API BGPPolicy #6203

Implement the controller for API BGPPolicy #6203

hongliangl commented Apr 9, 2024 •

edited

Loading

luolanzone commented Jun 25, 2024

hongliangl commented Jun 25, 2024

luolanzone Jun 25, 2024

hongliangl Jun 28, 2024

tnqn Jul 3, 2024

luolanzone Jun 25, 2024

hongliangl Jun 28, 2024

luolanzone Jun 25, 2024

hongliangl Jun 28, 2024

tnqn left a comment

tnqn Jul 3, 2024

tnqn Jul 3, 2024

hongliangl Jul 3, 2024

tnqn Jul 3, 2024

hongliangl Jul 3, 2024

tnqn Jul 3, 2024

tnqn Jul 3, 2024

tnqn Jul 3, 2024

hongliangl Jul 3, 2024

tnqn Jul 3, 2024

hongliangl Jul 3, 2024

tnqn Jul 3, 2024

tnqn Jul 3, 2024

tnqn Jul 3, 2024

hongliangl Jul 3, 2024

tnqn Jul 3, 2024

	// For IPv6 only environment, the BGP routerID should be specified by K8s Node annotation `antrea. io/ bgp-route-id`.
	// For IPv6 only environment, the BGP routerID should be specified by K8s Node annotation `antrea.io/bgp-route-id`.

		// peers maps IP families to concatenated strings of BGP peer IP addresses and ASNs.
		// Example: "192.168.77.100-65000", "2001::1-65000".

Implement the controller for API BGPPolicy #6203

Are you sure you want to change the base?

Implement the controller for API BGPPolicy #6203

Conversation

hongliangl commented Apr 9, 2024 • edited Loading

luolanzone commented Jun 25, 2024

hongliangl commented Jun 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongliangl commented Apr 9, 2024 •

edited

Loading