Shared Pool not working with v2 #2151

rgarcia89 · 2021-12-16T15:28:05Z

Setup Details

CIS Version : 2.6.1
Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: BIG-IP v16.1.1 (Build 0.0.16)
AS3 Version: 3.31.0
Agent Mode: AS3
Orchestration: K8S
Orchestration Version: 1.20.14
Pool Mode: Nodeport
Additional Setup details: CentOS 7.9

Description

When trying deploy a shared pool, controller v2 seems to run into an issue, even having the service port number defined correctly.
The issue appears as followed in the logs:

021/12/15 15:11:00 [ERROR] [AS3] Template is not valid. see errors
2021/12/15 15:11:00 [ERROR] - (root): Must validate one and only one schema (oneOf)
2021/12/15 15:11:00 [ERROR] - declaration.Shared: Must validate "then" as "if" was valid
2021/12/15 15:11:00 [ERROR] - declaration.Shared: Must validate "then" as "if" was valid
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0: Must validate "then" as "if" was valid
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0: Must validate one and only one schema (oneOf)
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0.serverAddresses: array items[0,3] must be unique
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0.serverAddresses: array items[1,4] must be unique
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0.serverAddresses: array items[2,5] must be unique
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0: Must validate all the schemas (allOf)
2021/12/15 15:11:00 [ERROR] - declaration.Shared.members.0: Must validate all the schemas (allOf)
2021/12/15 15:11:00 [ERROR] - declaration.Shared: Must validate all the schemas (allOf)

Steps To Reproduce

Create a Service with the following CIS labels

cis.f5.com/as3-app=Shared
cis.f5.com/as3-pool=alertmanager-cluster_pool
cis.f5.com/as3-tenant=sensu-system_infs_tenant

Create a AS3 declaration that creates a shared pool and uses it in two VS

kind: ConfigMap
apiVersion: v1
metadata:
  name: f5-as3-sensu-system
  namespace: f5
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
    "class": "AS3",
    "action": "deploy",
    "persist": false,
    "declaration": {
        "class": "ADC",
        "schemaVersion": "3.28.0",
        "id": "f5-as3-sensu-system",
        "label": "f5-as3-sensu-system",
        "remark": "AS3 Declaration - NS sensu-system",
        "sensu-system_infs_tenant": {
            "class": "Tenant",
            "Shared": {
                "class": "Application",
                "template": "shared",
                "alertmanager_serviceAddress": {
                    "class": "Service_Address",
                    "virtualAddress": "10.10.10.1",
                    "arpEnabled": true,
                    "icmpEcho": "enable",
                    "routeAdvertisement": "all",
                    "trafficGroup":"/Common/traffic-1"
                },
                "alertmanager-cluster_pool": {
                    "class": "Pool",
                    "loadBalancingMode": "round-robin",
                    "monitors": [
                        "tcp"
                    ],
                    "members": [
                        {
                            "servicePort": 9094,
                            "serverAddresses": [],
                            "shareNodes": true
                        }
                    ]
                }
            },
            "alertmanager-cluster_tcp": {
                "class": "Application",
                "template": "generic",
                "alertmanager-cluster_tcp": {
                    "class": "Service_TCP",
                    "remark": "alertmanager.x.x.x",
                    "virtualAddresses": [
                        {
                            "use": "/sensu-system_infs_tenant/Shared/alertmanager_serviceAddress"
                        }
                    ],
                    "virtualPort": 9094,
                    "allowVlans": [
                        "/Common/VL-lb-int"
                    ],
                    "snat": "auto",
                    "pool": "/sensu-system_infs_tenant/Shared/alertmanager-cluster_pool"
                }
            },
            "alertmanager-cluster_udp": {
                "class": "Application",
                "template": "generic",
                "alertmanager-cluster_udp": {
                    "class": "Service_UDP",
                    "remark": "alertmanager.x.x.x",
                    "virtualAddresses": [
                        {
                            "use": "/sensu-system_infs_tenant/Shared/alertmanager_serviceAddress"
                        }
                    ],
                    "virtualPort": 9094,
                    "allowVlans": [
                        "/Common/VL-lb-int"
                    ],
                    "snat": "auto",
                    "pool": "/sensu-system_infs_tenant/Shared/alertmanager-cluster_pool"
                }
            },
        }
    }
}

Expected Result

Two virtual servers should be created with a shared pool

Actual Result

CIS runs into an issue

Diagnostic Information

<Configuration files, error messages, logs>
Note: Sanitize the data. For example, be mindful of IPs, ports, application names and URLs
Note: The following F5 article outlines the information required when opening an issue.
https://support.f5.com/csp/article/K60974137

Observations (if any)

The text was updated successfully, but these errors were encountered:

mdditt2000 · 2021-12-21T19:14:11Z

@rgarcia89 validated with the following configuration that the nodes are getting created in /Common using sharednodes = true in the declaration.

https://github.com/mdditt2000/kubernetes-1-19/blob/master/cis%202.7/github/2151/as3-configmap/default/f5-as3-configmap.yaml

This is what is expected from CIS 2.6.1

Here is my repo - https://github.com/mdditt2000/kubernetes-1-19/tree/master/cis%202.7/github/2151

rgarcia89 · 2021-12-21T22:29:08Z

Hi @mdditt2000 I think you picked the wrong ticket?
This one is about creating a shared Application Pool.

Your answer seems more to be about #2101

mdditt2000 · 2021-12-23T05:56:20Z

@rgarcia89 understand you using a reference to shared pool which most likely CIS doesn't understand. I will look into that tomorrow. Can we chat tomorrow? I want to understand what you trying to achieve. Please email me at m.dittmer@f5.com

BTW checkout this new features coming in CIS 2.7. Its shared-pools etc but even better https://github.com/mdditt2000/kubernetes-1-19/blob/master/cis%202.7/tenant-configmap/README.md

rgarcia89 · 2021-12-23T08:46:38Z

Of course I will send you an email to arrange a slot to chat.
Thanks

mdditt2000 · 2022-02-09T18:43:44Z

@rgarcia89 where you able to resolve this issue with Kostas?

rgarcia89 · 2022-02-09T18:46:12Z

Not yet unfortunately. He wanted to come back to me

mdditt2000 · 2022-02-09T18:58:14Z

Will reach out to Kostas. Thanks!

skenderidis · 2022-02-11T05:20:38Z

Please find the steps to replicate the issue:

git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus
kubectl create -f manifests/setup
kubectl create -f manifests/

you should see a service (namespace: monitoring)

alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 11h

edit the service and add the labels shared by Raul

cis.f5.com/as3-app=Shared 
cis.f5.com/as3-pool=alertmanager-cluster_pool
cis.f5.com/as3-tenant=sensu-system_infs_tenant

apiVersion: v1
kind: ConfigMap
metadata:
  name: f5-as3-declaration
  namespace: monitoring
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
        "class": "AS3",
        "action": "deploy",
        "persist": false,        
        "declaration": {
            "class": "ADC",
            "schemaVersion": "3.28.0",
            "id": "f5-as3-sensu-system",
            "label": "f5-as3-sensu-system",
            "remark": "AS3 Declaration - NS sensu-system",
            "sensu-system_infs_tenant": {
                "class": "Tenant",
                "Shared": {
                    "class": "Application",
                    "template": "shared",
                    "alertmanager_serviceAddress": {
                            "class": "Service_Address",
                            "virtualAddress": "10.10.10.1",
                            "arpEnabled": true,
                            "icmpEcho": "enable",
                            "routeAdvertisement": "all",
                            "trafficGroup":"/Common/traffic-group-1"
                        },                    
                    "alertmanager-cluster_pool": {
                        "class": "Pool",
                        "loadBalancingMode": "round-robin",
                        "monitors": [
                            "tcp"
                        ],
                        "members": [
                            {
                                "servicePort": 9094,
                                "serverAddresses": [],
                                "shareNodes": true
                            }
                        ]
                    }
                },                
                "alertmanager-cluster_tcp": {
                    "class": "Application",
                    "template": "generic",
                    "alertmanager-cluster_tcp": {
                        "class": "Service_TCP",
                        "remark": "alertmanager.x.x.x",
                        "virtualAddresses": [
                            {
                                "use": "/sensu-system_infs_tenant/Shared/alertmanager_serviceAddress"
                            }
                        ],
                        "virtualPort": 9094,
                        "allowVlans": [
                            "/Common/VL-lb-int"
                        ],         
                        "snat": "auto",
                        "pool": "/sensu-system_infs_tenant/Shared/alertmanager-cluster_pool"
                    }
                },   
                "alertmanager-cluster_udp": {
                    "class": "Application",
                    "template": "generic",
                    "alertmanager-cluster_udp": {
                        "class": "Service_UDP",
                        "remark": "alertmanager.x.x.x",
                        "virtualAddresses": [
                            {
                                "use": "/sensu-system_infs_tenant/Shared/alertmanager_serviceAddress"
                            }
                        ],
                        "virtualPort": 9094,
                        "allowVlans": [
                            "/Common/VL-lb-int"
                        ],                        
                        "snat": "auto",
                        "pool": "/sensu-system_infs_tenant/Shared/alertmanager-cluster_pool"
                    }
                }
            }
        }
    }

you should see the error mentioned by Raul

trinaths · 2022-02-11T09:29:08Z

Created [CONTCNTR-3158] for internal tracking.

mdditt2000 · 2022-02-14T23:14:12Z

@rgarcia89 @skenderidis @trinaths please note the following works fine

https://github.com/mdditt2000/kubernetes-1-19/blob/master/cis%202.7.1/github/2151/configmap/configmap.yaml

2022/02/14 23:04:23 [DEBUG] [AS3] posting request to https://192.168.200.60/mgmt/shared/appsvcs/declare/
2022/02/14 23:04:28 [DEBUG] [AS3] Response from BIG-IP: code: 200 --- tenant:k8s --- message: no change
2022/02/14 23:04:28 [DEBUG] [AS3] Response from BIG-IP: code: 200 --- tenant:sensu-system_infs_tenant --- message: success
2022/02/14 23:04:28 [DEBUG] [AS3] Preparing response message to response handler for arp and fdb config
2022/02/14 23:04:28 [DEBUG] [AS3] AppManager wrote endpoints to VxlanMgr
2022/02/14 23:04:28 [DEBUG] [AS3] Sent response message to response handler for arp and fdb config

My changes

removed "trafficGroup":"/Common/traffic-group-1" --- not on BIG-IP
removed "allowVlans": [. -- not on BIG-IP
"/Common/VL-lb-int"
],
changed port to 80 --- only because my services/pod is listening on port 80

@rgarcia89 please try my configmap. Change your ports back.

rgarcia89 · 2022-02-15T10:44:27Z

@mdditt2000 I do not see any changes other than your mentioned removes, which need to be defined on my side.
Did it work for you with trafficGroup and allowedVlans configured?

mdditt2000 · 2022-03-09T06:06:12Z

Investigation in the next sprint starting this week.

vklohiya · 2022-03-10T11:26:04Z

@rgarcia89 , we have identified the root cause of the issue. It's happening because service "alertmanager-operated" in namespace "monitoring" is using the same port no for TCP & UDP protocol. And CIS is adding the duplicate pool members for both of these endpoints, which is rejected by the AS3 JSON schema.

apiVersion: v1
kind: Service
metadata:
  labels:
    cis.f5.com/as3-app: Shared
    cis.f5.com/as3-pool: alertmanager-cluster_pool
    cis.f5.com/as3-tenant: sensu-system_infs_tenant
    operated-alertmanager: "true"
  name: alertmanager-operated
  namespace: monitoring
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  - name: tcp-mesh
    port: 9094
    protocol: TCP
    targetPort: 9094
  - name: udp-mesh
    port: 9094
    protocol: UDP
    targetPort: 9094
  selector:
    app.kubernetes.io/name: alertmanager
  sessionAffinity: None
  type: ClusterIP

Would it be possible for you to use the different TCP & UDP port no in this service?

rgarcia89 · 2022-03-10T15:40:00Z

Hi @vklohiya these ports are coming from the Prometheus operator / helm chart and are unfortunately not configurable 😕

rgarcia89 · 2022-03-16T08:49:02Z

@vklohiya also just to make you aware. This works totally fine in version f5networks/k8s-bigip-ctlr:1.14

vklohiya · 2022-03-30T08:48:52Z

Thanks for the update @rgarcia89 , we are working on it.

rgarcia89 · 2022-04-20T08:20:46Z

@skenderidis @vklohiya @mdditt2000 same issue with v2.8.1

trinaths · 2022-04-20T08:45:55Z

Created CONTCNTR-3253 to fix this issue. ETA 2.9

sebidude · 2022-06-29T14:28:58Z

Hi all,
this is still not working as expected with v2.9.0. The VirtualServers and the Pools do not get created when you use the service below.

apiVersion: v1
kind: Service
metadata:
  labels:
    cis.f5.com/as3-app: Shared
    cis.f5.com/as3-pool: the-shared-pool
    cis.f5.com/as3-tenant: the-shared-tenant
  name: the-shared-app
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
  - name: mesh-tcp
    port: 9094
    protocol: TCP
    targetPort: 9094
  - name: mesh-udp
    port: 9094
    protocol: UDP
    targetPort: 9094
  selector:
    app: TheUDPApp
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

As soon as we remove one port from the service, everything gets created. That's why it was working for @mdditt2000 as stated here
We cannot use different ports for tcp and udp like @vklohiya has already asked, as this is part of the alertmanager code. https://github.com/prometheus/alertmanager#high-availability

vklohiya · 2022-06-30T04:57:08Z

@sebidude , Let me check it and get back to you.

vklohiya · 2022-06-30T05:38:24Z

@sebidude , it's working fine with following service definition in CIS 2.9.0:

apiVersion: v1
kind: Service
metadata:
  labels:
    cis.f5.com/as3-app: Shared
    cis.f5.com/as3-pool: alertmanager-cluster_pool
    cis.f5.com/as3-tenant: sensu-system_infs_tenant
    operated-alertmanager: "true"
  name: alertmanager-operated
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  - name: tcp-mesh
    port: 9094
    protocol: TCP
    targetPort: 9094
  - name: udp-mesh
    port: 9094
    protocol: UDP
    targetPort: 9094
  selector:
    app.kubernetes.io/name: alertmanager
  type: ClusterIP

Could you share the reproduction steps and your configMap configuration along with service?

sebidude · 2022-07-06T10:43:34Z

Hey @vklohiya
can you test this with a service of type NodePort and not ClusterIP?

Here is the service:

apiVersion: v1
kind: Service
metadata:
  labels:
    cis.f5.com/as3-app: Shared
    cis.f5.com/as3-pool: the-shared-pool
    cis.f5.com/as3-tenant: the-shared-tenant
  name: the-shared-app
spec:
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
  - name: mesh-tcp
    port: 9094
    protocol: TCP
    targetPort: 9094
  - name: mesh-udp
    port: 9094
    protocol: UDP
    targetPort: 9094
  selector:
    app: ThePodsThatMatch
  sessionAffinity: None
  type: NodePort

And here is the configmap that contains the corresponding as3 declaration:

kind: ConfigMap
apiVersion: v1
metadata:
  name: f5-as3-share-test
  namespace: f5lab
  labels:
    f5type: virtual-server
    as3: "true"
data:
  template: |
    {
        "class": "AS3",
        "action": "deploy",
        "persist": false,
        "declaration": {
            "class": "ADC",
            "schemaVersion": "3.28.0",
            "id": "f5-as3-share-test",
            "label": "f5-as3-share-test",
            "remark": "testing shared multiport svc",
            "the-shared-tenant": {
                "class": "Tenant",
                "Shared": {
                    "class": "Application",
                    "template": "shared",
                    "TheSharedServiceAddress": {
                        "class": "Service_Address",
                        "virtualAddress": "10.10.10.1",
                        "arpEnabled": true,
                        "icmpEcho": "enable",
                        "routeAdvertisement": "all"
                    },
                    "the-shared-pool": {
                        "class": "Pool",
                        "loadBalancingMode": "round-robin",
                        "monitors": [
                            "tcp"
                        ],
                        "members": [
                            {
                                "servicePort": 9094,
                                "serverAddresses": [],
                                "shareNodes": true
                            }
                        ]
                    }
                },
                "TheTCPApp": {
                    "class": "Application",
                    "template": "generic",
                    "TheTCPApp": {
                        "class": "Service_TCP",
                        "remark": "remark A",
                        "virtualAddresses": [
                            {
                                "use": "/the-shared-tenant/Shared/TheSharedServiceAddress"
                            }
                        ],
                        "virtualPort": 9094,
                        "snat": "auto",
                        "pool": "/the-shared-tenant/Shared/the-shared-pool"
                    }
                },
                "TheUDPApp": {
                    "class": "Application",
                    "template": "generic",
                    "TheUDPApp": {
                        "class": "Service_UDP",
                        "remark": "remark B",
                        "virtualAddresses": [
                            {
                                "use": "/the-shared-tenant/Shared/TheSharedServiceAddress"
                            }
                        ],
                        "virtualPort": 9094,
                        "snat": "auto",
                        "pool": "/the-shared-tenant/Shared/the-shared-pool"
                    }
                }
            }
        }
    }

vklohiya · 2022-07-07T04:24:09Z

@sebidude , CIS is deployed in ClusterIP mode or NodePort mode?

sebidude · 2022-07-07T04:48:05Z

NodePort mode.
This works totally fine with version f5networks/k8s-bigip-ctlr:1.14

sebidude · 2022-07-07T10:56:02Z

@vklohiya we ran it in the debugger and noticed, that the fix which was merged with #2396 does not handle duplicate pool members when the controller is deployed in NodePort mode.

we just added

} else { // Controller is in NodePort mode.
    if service.Spec.Type == v1.ServiceTypeNodePort {
        for _, port := range service.Spec.Ports {
            endpointMembers := appMgr.getEndpointsForNodePort(port.NodePort, port.Port)
            for _, newMember := range endpointMembers {
                if _, ok := uniqueMembersMap[newMember]; !ok {
                        uniqueMembersMap[newMember] = struct{}{}
                        members = append(members, newMember)
                }
            }
        }
    }
}

and made it working.

Please verify and release a bugfix as soon as possible.

Cheers
Sebi

vklohiya · 2022-07-07T12:04:01Z

@sebidude , thanks for identifying the fix. Can you please raise a separate github issue to track this?

rgarcia89 added bug untriaged no JIRA created labels Dec 16, 2021

mdditt2000 self-assigned this Dec 21, 2021

trinaths added awaiting response Awaiting response and removed untriaged no JIRA created labels Jan 5, 2022

trinaths added JIRA and removed awaiting response Awaiting response labels Feb 11, 2022

mdditt2000 added the In progress Fix in progress label Mar 9, 2022

trinaths added this to the 2.9 milestone Apr 20, 2022

vklohiya mentioned this issue May 10, 2022

Fix for adding unique pool members only to as3 declaration #2396

Merged

2 tasks

vklohiya closed this as completed in #2396 May 16, 2022

sebidude mentioned this issue Jul 7, 2022

Shared pool not working when cis is deployed in NodePort mode #2492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shared Pool not working with v2 #2151

Shared Pool not working with v2 #2151

rgarcia89 commented Dec 16, 2021

mdditt2000 commented Dec 21, 2021

rgarcia89 commented Dec 21, 2021

mdditt2000 commented Dec 23, 2021

rgarcia89 commented Dec 23, 2021

mdditt2000 commented Feb 9, 2022

rgarcia89 commented Feb 9, 2022

mdditt2000 commented Feb 9, 2022

skenderidis commented Feb 11, 2022

trinaths commented Feb 11, 2022

mdditt2000 commented Feb 14, 2022

rgarcia89 commented Feb 15, 2022

mdditt2000 commented Mar 9, 2022

vklohiya commented Mar 10, 2022 •

edited

rgarcia89 commented Mar 10, 2022

rgarcia89 commented Mar 16, 2022

vklohiya commented Mar 30, 2022

rgarcia89 commented Apr 20, 2022

trinaths commented Apr 20, 2022

sebidude commented Jun 29, 2022 •

edited

vklohiya commented Jun 30, 2022

vklohiya commented Jun 30, 2022 •

edited

sebidude commented Jul 6, 2022 •

edited

vklohiya commented Jul 7, 2022

sebidude commented Jul 7, 2022

sebidude commented Jul 7, 2022

vklohiya commented Jul 7, 2022 •

edited

Shared Pool not working with v2 #2151

Shared Pool not working with v2 #2151

Comments

rgarcia89 commented Dec 16, 2021

Setup Details

Description

Steps To Reproduce

Expected Result

Actual Result

Diagnostic Information

Observations (if any)

mdditt2000 commented Dec 21, 2021

rgarcia89 commented Dec 21, 2021

mdditt2000 commented Dec 23, 2021

rgarcia89 commented Dec 23, 2021

mdditt2000 commented Feb 9, 2022

rgarcia89 commented Feb 9, 2022

mdditt2000 commented Feb 9, 2022

skenderidis commented Feb 11, 2022

trinaths commented Feb 11, 2022

mdditt2000 commented Feb 14, 2022

rgarcia89 commented Feb 15, 2022

mdditt2000 commented Mar 9, 2022

vklohiya commented Mar 10, 2022 • edited

rgarcia89 commented Mar 10, 2022

rgarcia89 commented Mar 16, 2022

vklohiya commented Mar 30, 2022

rgarcia89 commented Apr 20, 2022

trinaths commented Apr 20, 2022

sebidude commented Jun 29, 2022 • edited

vklohiya commented Jun 30, 2022

vklohiya commented Jun 30, 2022 • edited

sebidude commented Jul 6, 2022 • edited

vklohiya commented Jul 7, 2022

sebidude commented Jul 7, 2022

sebidude commented Jul 7, 2022

vklohiya commented Jul 7, 2022 • edited

vklohiya commented Mar 10, 2022 •

edited

sebidude commented Jun 29, 2022 •

edited

vklohiya commented Jun 30, 2022 •

edited

sebidude commented Jul 6, 2022 •

edited

vklohiya commented Jul 7, 2022 •

edited