[CONTCNTR-3779] Restarting k8s-bigip-ctlr when using f5-ipam-controller causes all VirtualServers to be recreated with new IP addresses, or even removed

[klippo]

Setup Details

CIS Version : 2.11.1
Build: f5networks/k8s-bigip-ctlr:2.11.1
IPAM controller: f5networks/f5-ipam-controller:0.1.8
BIGIP Version: Big IP 15.1.7
AS3 Version: 3.38.8
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: 1.24.4
Pool Mode: Cluster
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc>

Description

We're trying to understand why all our VirtualServers gets a new IP on every restart of f5-bigip-ctlr . We're using it together with the f5-ipam-controller .

We've noticed that every time f5-bigip-ctlr is restarted, the ipams.fic.f5.com/f5-bigip-ctlr.k8s-cluster01.ipam is deleted and re-created , this will cause f5-ipam-controller to empty it's SQLite database.

Observed with:

• Deletion of ipams resource from f5-bigip-ctrl logs:

f5-bigip-ctlr-7d76559868-4m69l f5-bigip-ctlr 2023/01/30 08:39:06 [DEBUG] [ipam] Created IPAM Custom Resource: 

https://github.com/F5Networks/k8s-bigip-ctlr/blob/v2.11.1/pkg/controller/controller.go#L246

• Monitoring ipam database and noticing that it's emptied.

$ watch -n 0.5 'sqlite3 cis_ipam.sqlite3 "SELECT * FROM ipaddress_range"'

• Logs from F5-Ipam-Controller

f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:18 [DEBUG] Processing Key: &{0xc000352000 <nil> Delete}

f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:18 [ERROR] Unable to find IPAM: kube-system/f5-bigip-ctlr.k8s-dev.ipam to update
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:18 [ERROR] Unable to find IPAM: kube-system/f5-bigip-ctlr.k8s-dev.ipam to update
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:21 [DEBUG] Enqueueing on Create: kube-system/f5-bigip-ctlr.k8s-dev.ipam
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:21 [DEBUG] Processing Key: &{0xc0003534a0 <nil> Create}
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:28 [DEBUG] Enqueueing on Update: kube-system/f5-bigip-ctlr.k8s-dev.ipam
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:28 [DEBUG] Processing Key: &{0xc000353600 0xc0003534a0 Update}
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:28 [DEBUG] Enqueueing on Update: kube-system/f5-bigip-ctlr.k8s-dev.ipam
f5-ipam-controller-77884566c8-fjx94 f5-ipam-controller 2023/01/26 15:11:28 [DEBUG] Processing Key: &{0xc0003dc000 0xc000353600 Update}

This means that every VirtualServer now have a new IP address that needs to be updated in F5 and in the DNS.

We've also noticed that if we restart f5-bigip-ctlr without f5-ipam-controller running and using a mix of ipamLabel and virtualServerAddress, all ipams VirtualServers are removed from F5.

Is this by design, or have we done a major configuration mishap somewhere?

Expected Result

Persist allocated IP addresses during restarts of service

Actual Result

Diagnostic Information

Runtime args:

- args:
        - --credentials-directory
        - /tmp/creds
        - --bigip-partition=k8s-dev
        - --bigip-url=https://f5/
        - --custom-resource-mode=true
        - --insecure=true
        - --ipam=true
        - --log-as3-response=true
        - --log-level=DEBUG
        - --manage-ingress=false
        - --pool-member-type=cluster
        command:
        - /app/bin/k8s-bigip-ctlr

  - args:
    - --orchestration=kubernetes
    - --ipam-provider=f5-ip-provider
    - --ip-range={"OTA-TEST-LB-K8S-DEV":"10.0.0.1-10.0.0.20"}
    - --log-level=DEBUG
   command:
     - /app/bin/f5-ipam-controller

<Configuration files, error messages, logs>
Note: Sanitize the data. For example, be mindful of IPs, ports, application names and URLs
Note: The following F5 article outlines the information required when opening an issue.
https://support.f5.com/csp/article/K60974137

Observations (if any)

[trinaths]

Created [CONTCNTR-3779] for internal tracking.

[vklohiya]

Fix will be available in CIS 2.12 release and FIC release 0.1.9.

PR: F5Networks/f5-ipam-controller#142.