Resource Type and Namespace Filter Support for HubMode

[sectoreleven]

Title

Resource Type and Namespace Filter Support for HubMode

Description

Enable resource type filtering by namespace when using HubMode

Actual Problem

At enterprise scale using shared BIG-IP devices, we need to better separate roles and responsibilities for deploying BIG-IP configurations via CIS in a given Kubernetes cluster. Our model is to have app teams run their Deployments and Services in their own namespaces, while CIS and ConfigMaps live in an mutually-exclusive namespace owned and managed by our load-balancer administrators. App teams do not have access to this load-balancer namespace directly.

Given that CIS utilizes only administrator-level credentials for BIG-IP, we need to allow CIS to watch for ConfigMaps and Ingress in this load-balancer namespace only. Our teams are not granted admin credentials to BIG-IP directly, so even if they deploy a ConfigMap in their own namespace, it should not be picked up by CIS. Instead, a pipeline operated by our load-balancer admins will deploy ConfigMaps from preapproved templates into the CIS namespace.

Currently, the namespace filter is global, when specifying namespaces it watches only the specified namespaces for any resource. However, this breaks HubMode if only the namespace with the ConfigMap is specified.

Given the following topology:
CIS and ConfigMap Namespace - f5-cis
App Namespace - app-webservice

And the following CIS configuration:

args:
  bigip_url: #.#.#.#
  insecure: true
  bigip_partition: f5-bigip-ctlr
  pool_member_type: nodeport
  manage_routes: false
  manage_ingress: false
  custom-resource-mode: false
  manage-configmaps: true
  hubmode: true
  log-as3-response: true
  filter-tenants: true
  log_level: INFO
  namespaces: ["f5-cis"]

And an app deployed like:

apiVersion: v1
kind: Service
metadata:
  name: f5-hello-world-web
  namespace: app-webservice
  labels:
    app: f5-hello-world-web
    cis.f5.com/as3-tenant: app-team-partition
    cis.f5.com/as3-app: app
    cis.f5.com/as3-pool: app-webservice-pool
spec:
  ports:
  - name: f5-hello-world-web
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: NodePort
  selector:
    app: f5-hello-world-web

The above config will successfully create BIG-IP virtual servers from ConfigMaps deployed in the f5-cis namespace, but fails to wire the service up to the pool.

Solution Proposed

Enable namespace filtering by resource type when utilizing HubMode.
Instead of simple namespace filtering like namespaces: ["f5-cis"], it would helpful to define by type.
This could be either by custom properties in the YAML, i.e.:

configMapNamespaces: ["f5-cis"]
serviceNamespaces: ["app-webservice"]

or by embedding the type in the definition via arrays or some other syntax

namespaces:
- name: "f5-cis"
  resourceTypes:
  - configMap
- name: "app-webservice"
  resourceTypes:
  - service

[trinaths]

Created [CONTCNTR-3677] for internal tracking

[sectoreleven]

Any update or planned release for this feature request? Would appreciate an update here.

[trinaths]

Created [CONTCNTR-3683] for internal tracking. ETA 2.13 ~ May'23

[vklohiya]

In CIS namespace filter is global means CIS will watch all the resources in that namespace, CIS also does not monitor configMap and other resources in other namespaces when --namespace is specified in the CIS deployment. However when --hub-mode is set to true CIS looks only for services in all the namespaces instead of specified namespace. If there are multiple services with the same label is created in multiple namespaces, CIS considers the oldest service.

As per the your use-case, you can define the namespace where you wants to create the configMap in CIS deployment parameters along with the hub-mode parameter. Respective services/deployment can be defined in any other namespace. It is recommended that you do not create the multiple services with the same labels, if it happens CIS will consider the oldest service in this scenario.

@sectoreleven , Can you please try this approach and share if anything is blocking you?

[sectoreleven]

As per the your use-case, you can define the namespace where you wants to create the configMap in CIS deployment parameters along with the hub-mode parameter. Respective services/deployment can be defined in any other namespace. It is recommended that you do not create the multiple services with the same labels, if it happens CIS will consider the oldest service in this scenario.

@sectoreleven , Can you please try this approach and share if anything is blocking you?

@vklohiya - we tried this originally and listed the results in the original ticket submission. In such configuration, CIS creates the pool from the ConfigMap but then does not successfully populate it with the Pods belonging to the Service.

(Note: we run NodePortLocal mode, so it is Pods we expect in the BIG-IP pool, not cluster nodes.)

[vklohiya]

@sectoreleven Thanks for the additional info. Currently hub mode is broken with NodePortLocal in this particular use-case.

[sectoreleven]

Two questions, then.

First, when you state:

However when --hub-mode is set to true CIS looks only for services in all the namespaces instead of specified namespace.

Does that mean if this worked as intended with our config and NodePortLocal, then teams that attempt to deploy ConfigMap or Ingress resources to their own namespaces will not get associated objects created on the BIG-IP?

Second, now as a stated bug with NodePortLocal, would this be targeted for a 2.12.1 fix, the 2.13 release as stated prior, or later?

[vklohiya]

Two questions, then.

First, when you state:

However when --hub-mode is set to true CIS looks only for services in all the namespaces instead of specified namespace.

Does that mean if this worked as intended with our config and NodePortLocal, then teams that attempt to deploy ConfigMap or Ingress resources to their own namespaces will not get associated objects created on the BIG-IP?

Second, now as a stated bug with NodePortLocal, would this be targeted for a 2.12.1 fix, the 2.13 release as stated prior, or later?

Ans-1: Yes if teams attempt to create configMap in their own namespaces no object will be created on BIGIP. Only the configMap in specified namespace will be considered.

Note: --hub-mode feature is only supported with configMap resource, not with Ingress.

Ans-2: Fix will be available in 2.13 release.