Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CIS cannot discover service endpoints when the targetPort is referred name instead of number #2729

Closed
rajbaratht opened this issue Jan 17, 2023 · 9 comments · Fixed by #2748
Closed

CIS cannot discover service endpoints when the targetPort is referred name instead of number #2729

rajbaratht opened this issue Jan 17, 2023 · 9 comments · Fixed by #2748
Labels
feature-request JIRA

Comments

@rajbaratht
Copy link

Setup Details

CIS Version : 2.10.1
Build: f5networks/k8s-bigip-ctlr:2.10.1
BIGIP Version: BIG-IP 15.1.3 Build 0.0.11 Final
AS3 Version: f5-appsvcs-3.38.0-4.noarch
Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version: Kubernetes v1.22.13
Pool Mode: Cluster/Nodeport
Additional Setup details: Rancher/Calico

Description

When the service has name in TargetPort then the members in the load balancer is empty.

Steps To Reproduce

  1. Deploy rabbitmq app using helm; chart version rabbitmq-8.31.5 and app version 3.9.15. And look at the service created by the app.
kg svc/rabbitmq -o yaml -n data
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: rabbitmq
    meta.helm.sh/release-namespace: data
  creationTimestamp: "2022-04-19T18:52:58Z"
  labels:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-8.31.5
  name: rabbitmq
  namespace: data
  resourceVersion: "240109529"
  uid: 55d1b599-1895-460d-b403-0c7eb7cc66f0
spec:
  clusterIP: 10.43.198.109
  clusterIPs:
  - 10.43.198.109
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: amqp
  - name: amqp-ssl
    port: 5671
    protocol: TCP
    targetPort: amqp-ssl
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: epmd
  - name: dist
    port: 25672
    protocol: TCP
    targetPort: dist
  - name: http-stats
    port: 15672
    protocol: TCP
    targetPort: stats
  selector:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/name: rabbitmq
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
  1. Create the following vip
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  creationTimestamp: "2023-01-17T16:19:47Z"
  generation: 1
  labels:
    f5cr: "true"
  name: dev1.messagebus.p2.internal
  namespace: data
  resourceVersion: "369930540"
  uid: 5927e4c0-73b6-4c6c-9491-1fdbfa48b319
spec:
  pools:
  - monitor:
      interval: 20
      recv: ""
      send: /
      timeout: 31
      type: https
    path: /
    service: rabbitmq
    servicePort: 15672
  tlsProfileName: dev1.messagebus.p2.internal
  virtualServerAddress: 10.40.14.207
  virtualServerName: dev1_messagebus_p2_internal_vs
status:
  status: Ok
  vsAddress: None
  1. Check the logs in the f5 pod
2023/01/17 15:00:41 [DEBUG] Processing Key: &{data TLSProfile dev1.messagebus.p2.internal 0xc0005df500 Create}
2023/01/17 15:00:41 [INFO] Enqueueing TLSProfile: &{{ } {dev1.messagebus.p2.internal  data  4d6c3de4-a72a-486b-a5af-bd34c8665d23 369880900 1 2023-01-17 15:00:41 +0000 UTC <nil> <nil> map[f5cr:true] map[] [] []  [{kubectl-create Update cis.f5.com/v1 2023-01-17 15:00:41 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:tls":{".":{},"f:clientSSL":{},"f:reference":{},"f:serverSSL":{},"f:termination":{}}}}}]} {[] {reencrypt /Common/dev1.messagebus.p2.internal_ssl_client /Common/dev1.messagebus.p2.internal_ssl_server bigip}}}
2023/01/17 15:00:41 [INFO] Change in TLSProfile dev1.messagebus.p2.internal does not effect any VirtualServer
2023/01/17 15:00:42 [DEBUG] Enqueueing VirtualServer: &{{ } {dev1.messagebus.p2.internal  data  5e503746-f5c8-4444-b7d5-60cd3b2b3714 369880910 1 2023-01-17 15:00:42 +0000 UTC <nil> <nil> map[f5cr:true] map[] [] []  [{kubectl-create Update cis.f5.com/v1 2023-01-17 15:00:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}}]} {  10.40.14.207  dev1_messagebus_p2_internal_vs 0 0 [{ / rabbitmq 15672  {https /  20 31 0  } []   }] dev1.messagebus.p2.internal     [] [] []      {{ }      []  } []} { }}
2023/01/17 15:00:42 [DEBUG] Processing Key: &{data VirtualServer dev1.messagebus.p2.internal 0xc001862000 Create}
2023/01/17 15:00:42 [DEBUG] Process all the Virtual Servers which share same VirtualServerAddress
2023/01/17 15:00:42 [DEBUG] Processing Virtual Server dev1.messagebus.p2.internal for port 443
2023/01/17 15:00:42 [DEBUG] Configured rule: {vs__rabbitmq_15672_data  0 [0xc0002812c0] []}
2023/01/17 15:00:42 [DEBUG] Configured policy: {dev1_messagebus_p2_internal_vs_443__policy data  [forwarding]  true [http] [0xc000281440] /Common/first-match}
2023/01/17 15:00:42 [DEBUG] Processing  BIGIP referenced profiles for 'VirtualServer' 'data'/'dev1.messagebus.p2.internal'
2023/01/17 15:00:42 [DEBUG] Updated BIGIP referenced profiles for 'VirtualServer' 'data'/'dev1.messagebus.p2.internal'
2023/01/17 15:00:42 [DEBUG] Updated Virtual dev1.messagebus.p2.internal with TLSProfile dev1.messagebus.p2.internal
2023/01/17 15:00:42 [ERROR] [CORE]Endpoints could not be fetched for service rabbitmq with targetPort 0

Expected Result

The endpoints should be detected by f5

Actual Result

Endpoints could not be fetched for service rabbitmq with targetPort 0

Diagnostic Information


2023/01/17 15:01:06 [DEBUG] Enqueueing VirtualServer: &{{ } {dev1.messagebus.p2.internal  data  5e503746-f5c8-4444-b7d5-60cd3b2b3714 369881142 1 2023-01-17 15:00:42 +0000 UTC <nil> <nil> map[f5cr:true] map[] [] []  [{kubectl-create Update cis.f5.com/v1 2023-01-17 15:00:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {k8s-bigip-ctlr.real Update cis.f5.com/v1 2023-01-17 15:01:06 +0000 UTC FieldsV1 {"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} {  10.40.14.207  dev1_messagebus_p2_internal_vs 0 0 [{ / rabbitmq 15672  {https /  20 31 0  } []   }] dev1.messagebus.p2.internal     [] [] []      {{ }      []  } []} {None Ok}}
2023/01/17 15:01:06 [DEBUG] Processing Key: &{data VirtualServer dev1.messagebus.p2.internal 0xc001862700 Update}
2023/01/17 15:01:06 [DEBUG] Process all the Virtual Servers which share same VirtualServerAddress
2023/01/17 15:01:06 [DEBUG] Processing Virtual Server dev1.messagebus.p2.internal for port 443
2023/01/17 15:01:06 [DEBUG] Configured rule: {vs__rabbitmq_15672_data  0 [0xc001d8a540] []}
2023/01/17 15:01:06 [DEBUG] Configured policy: {dev1_messagebus_p2_internal_vs_443__policy data  [forwarding]  true [http] [0xc001d8a5a0] /Common/first-match}
2023/01/17 15:01:06 [DEBUG] Processing  BIGIP referenced profiles for 'VirtualServer' 'data'/'dev1.messagebus.p2.internal'
2023/01/17 15:01:06 [DEBUG] Updated BIGIP referenced profiles for 'VirtualServer' 'data'/'dev1.messagebus.p2.internal'
2023/01/17 15:01:06 [DEBUG] Updated Virtual dev1.messagebus.p2.internal with TLSProfile dev1.messagebus.p2.internal
2023/01/17 15:01:06 [ERROR] [CORE]Endpoints could not be fetched for service rabbitmq with targetPort 0
2023/01/17 15:01:06 [DEBUG] Finished syncing virtual servers &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:dev1.messagebus.p2.internal GenerateName: Namespace:data SelfLink: UID:5e503746-f5c8-4444-b7d5-60cd3b2b3714 ResourceVersion:369881142 Generation:1 CreationTimestamp:2023-01-17 15:00:42 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[f5cr:true] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:kubectl-create Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-17 15:00:42 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {Manager:k8s-bigip-ctlr.real Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-17 15:01:06 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} Spec:{Host: HostGroup: VirtualServerAddress:10.40.14.207 IPAMLabel: VirtualServerName:dev1_messagebus_p2_internal_vs VirtualServerHTTPPort:0 VirtualServerHTTPSPort:0 Pools:[{Name: Path:/ Service:rabbitmq ServicePort:15672 NodeMemberLabel: Monitor:{Type:https Send:/ Recv: Interval:20 Timeout:31 TargetPort:0 Name: Reference:} Monitors:[] Rewrite: Balance: ServiceNamespace:}] TLSProfileName:dev1.messagebus.p2.internal HTTPTraffic: SNAT: WAF: RewriteAppRoot: AllowVLANs:[] IRules:[] ServiceIPAddress:[] PolicyName: PersistenceProfile: ProfileMultiplex: DOS: BotDefense: Profiles:{TCP:{Client: Server:} UDP: HTTP: HTTP2: RewriteProfile: PersistenceProfile: LogProfiles:[] ProfileL4: ProfileMultiplex:} AllowSourceRange:[]} Status:{VSAddress:None StatusOk:Ok}} (200.862µs)


Note: Sanitize the data. For example, be mindful of IPs, ports, application names and URLs
Note: The following F5 article outlines the information required when opening an issue.
https://support.f5.com/csp/article/K60974137

Observations (if any)
@rajbaratht rajbaratht added bug untriaged no JIRA created labels Jan 17, 2023
@rajbaratht rajbaratht changed the title CIS cannot discover service endpoints when the servicePort is referred name instead of number CIS cannot discover service endpoints when the targetPort is referred name instead of number Jan 17, 2023
@trinaths
Copy link
Contributor

trinaths commented Jan 18, 2023 •
edited

@rajbaratht the service is not configured properly.

- name: http-stats
    port: 15672
    protocol: TCP
    targetPort: stats     <--- http-stats ? In the given service YAML, there is no reference to stats.

@trinaths trinaths added awaiting response Awaiting response and removed untriaged no JIRA created labels Jan 18, 2023
@rajbaratht
Copy link
Author

@trinaths I updated the svc and still I get the same error.

Service

 kg svc/rabbitmq -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: rabbitmq
    meta.helm.sh/release-namespace: data
  creationTimestamp: "2022-09-05T20:08:06Z"
  labels:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-8.31.5
  name: rabbitmq
  namespace: data
  resourceVersion: "220276321"
  uid: 0bc80adc-996b-4f98-962c-8798e5e1b2a9
spec:
  clusterIP: 10.43.124.161
  clusterIPs:
  - 10.43.124.161
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: amqp
  - name: amqp-ssl
    port: 5671
    protocol: TCP
    targetPort: amqp-ssl
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: epmd
  - name: dist
    port: 25672
    protocol: TCP
    targetPort: dist
  - name: http-stats
    port: 15672
    protocol: TCP
    targetPort: http-stats
  selector:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/name: rabbitmq
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Logs from the cis controller pod

2023/01/18 18:48:40 [DEBUG] Updating VirtualServer Status with { Ok} for resource name:dev2.messagebus.p2.internal , namespace: data
2023/01/18 18:48:40 [DEBUG] Enqueueing VirtualServer: &{{ } {dev2.messagebus.p2.internal  data  7ee7a089-faf6-4bed-b1aa-5e5283e5730b 220277508 1 2023-01-18 18:48:20 +0000 UTC <nil> <nil> map[f5cr:true] map[] [] []  [{kubectl-create Update cis.f5.com/v1 2023-01-18 18:48:20 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {k8s-bigip-ctlr.real Update cis.f5.com/v1 2023-01-18 18:48:40 +0000 UTC FieldsV1 {"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} {  10.40.15.207  dev2_messagebus_p2_internal_vs 0 0 [{ / rabbitmq 15672  {https /  20 31 0  } []   }] dev2.messagebus.p2.internal     [] [] []      {{ }      []  } []} {None Ok}}
2023/01/18 18:48:40 [DEBUG] Processing Key: &{data VirtualServer dev2.messagebus.p2.internal 0xc000e31180 Update}
2023/01/18 18:48:40 [DEBUG] Process all the Virtual Servers which share same VirtualServerAddress
2023/01/18 18:48:40 [DEBUG] Processing Virtual Server dev2.messagebus.p2.internal for port 443
2023/01/18 18:48:40 [DEBUG] Configured rule: {vs__rabbitmq_15672_data  0 [0xc0004b7aa0] []}
2023/01/18 18:48:40 [DEBUG] Configured policy: {dev2_messagebus_p2_internal_vs_443__policy data  [forwarding]  true [http] [0xc0004b7b00] /Common/first-match}
2023/01/18 18:48:40 [DEBUG] Processing  BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:48:40 [DEBUG] Updated BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:48:40 [DEBUG] Updated Virtual dev2.messagebus.p2.internal with TLSProfile dev2.messagebus.p2.internal
2023/01/18 18:48:40 [ERROR] [CORE]Endpoints could not be fetched for service rabbitmq with targetPort 0
2023/01/18 18:48:40 [DEBUG] Finished syncing virtual servers &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:dev2.messagebus.p2.internal GenerateName: Namespace:data SelfLink: UID:7ee7a089-faf6-4bed-b1aa-5e5283e5730b ResourceVersion:220277508 Generation:1 CreationTimestamp:2023-01-18 18:48:20 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[f5cr:true] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:kubectl-create Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:20 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {Manager:k8s-bigip-ctlr.real Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:40 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} Spec:{Host: HostGroup: VirtualServerAddress:10.40.15.207 IPAMLabel: VirtualServerName:dev2_messagebus_p2_internal_vs VirtualServerHTTPPort:0 VirtualServerHTTPSPort:0 Pools:[{Name: Path:/ Service:rabbitmq ServicePort:15672 NodeMemberLabel: Monitor:{Type:https Send:/ Recv: Interval:20 Timeout:31 TargetPort:0 Name: Reference:} Monitors:[] Rewrite: Balance: ServiceNamespace:}] TLSProfileName:dev2.messagebus.p2.internal HTTPTraffic: SNAT: WAF: RewriteAppRoot: AllowVLANs:[] IRules:[] ServiceIPAddress:[] PolicyName: PersistenceProfile: ProfileMultiplex: DOS: BotDefense: Profiles:{TCP:{Client: Server:} UDP: HTTP: HTTP2: RewriteProfile: PersistenceProfile: LogProfiles:[] ProfileL4: ProfileMultiplex:} AllowSourceRange:[]} Status:{VSAddress:None StatusOk:Ok}} (157.741µs)

There are no current members in the pool

image

Kindly advice

@rajbaratht
Copy link
Author

@trinaths Any updates ?

@trinaths
Copy link
Contributor

trinaths commented Jan 19, 2023 •
edited

@rajbaratht Can you replace the name reference in targetPort and give normal integer in service definition. Refer the port name in the VS CRD. Share your findings.

@rajbaratht
Copy link
Author

@trinaths Yes, it works if I replace the name reference in targetPort to normal integer in service definition.
I'm not sure how to add port name in VS CRD; can you please share an example ?

VS

kg vs/dev2.messagebus.p2.internal -o yaml
apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  creationTimestamp: "2023-01-18T18:48:20Z"
  generation: 1
  labels:
    f5cr: "true"
  name: dev2.messagebus.p2.internal
  namespace: data
  resourceVersion: "220277508"
  uid: 7ee7a089-faf6-4bed-b1aa-5e5283e5730b
spec:
  pools:
  - monitor:
      interval: 20
      recv: ""
      send: /
      timeout: 31
      type: https
    path: /
    service: rabbitmq
    servicePort: 15672
  tlsProfileName: dev2.messagebus.p2.internal
  virtualServerAddress: 10.40.15.207
  virtualServerName: dev2_messagebus_p2_internal_vs
status:
  status: Ok
  vsAddress: None

service

 kg svc/rabbitmq -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: rabbitmq
    meta.helm.sh/release-namespace: data
  creationTimestamp: "2022-09-05T20:08:06Z"
  labels:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-8.31.5
  name: rabbitmq
  namespace: data
  resourceVersion: "220282944"
  uid: 0bc80adc-996b-4f98-962c-8798e5e1b2a9
spec:
  clusterIP: 10.43.124.161
  clusterIPs:
  - 10.43.124.161
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: amqp
    port: 5672
    protocol: TCP
    targetPort: 5672
  - name: amqp-ssl
    port: 5671
    protocol: TCP
    targetPort: 5671
  - name: epmd
    port: 4369
    protocol: TCP
    targetPort: 4369
  - name: dist
    port: 25672
    protocol: TCP
    targetPort: 25672
  - name: http-stats
    port: 15672
    protocol: TCP
    targetPort: 15672
  selector:
    app.kubernetes.io/instance: rabbitmq
    app.kubernetes.io/name: rabbitmq
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

logs from cis

2023/01/18 18:57:59 [DEBUG] [CORE] NodePoller (0xc0001d15f0) notifying listener: {l:0xc00055b800 s:0xc00055b860}
2023/01/18 18:57:59 [DEBUG] [CORE] NodePoller (0xc0001d15f0) listener callback - num items: 7 err: <nil>
2023/01/18 18:58:13 [DEBUG] [2023-01-18 18:58:13,680 __main__ DEBUG] config handler woken for reset
2023/01/18 18:58:13 [DEBUG] [2023-01-18 18:58:13,680 __main__ DEBUG] loaded configuration file successfully
2023/01/18 18:58:13 [DEBUG] [2023-01-18 18:58:13,680 __main__ DEBUG] NET Config: {}
2023/01/18 18:58:13 [DEBUG] [2023-01-18 18:58:13,680 __main__ DEBUG] loaded configuration file successfully
2023/01/18 18:58:13 [DEBUG] [2023-01-18 18:58:13,681 __main__ DEBUG] updating tasks finished, took 0.0007417201995849609 seconds
2023/01/18 18:58:26 [DEBUG] Enqueueing Old Service: &Service{ObjectMeta:{rabbitmq  data  0bc80adc-996b-4f98-962c-8798e5e1b2a9 220276321 0 2022-09-05 20:08:06 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:rabbitmq app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:rabbitmq helm.sh/chart:rabbitmq-8.31.5] map[meta.helm.sh/release-name:rabbitmq meta.helm.sh/release-namespace:data] [] []  [{helm Update v1 2022-09-05 20:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":4369,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":5671,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":5672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":15672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":25672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}} {kubectl-edit Update v1 2023-01-17 17:02:41 +0000 UTC FieldsV1 {"f:spec":{"f:ports":{"k:{\"port\":4369,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":5671,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":5672,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":15672,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":25672,\"protocol\":\"TCP\"}":{"f:targetPort":{}}}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:amqp,Protocol:TCP,Port:5672,TargetPort:{1 0 amqp},NodePort:0,AppProtocol:nil,},ServicePort{Name:amqp-ssl,Protocol:TCP,Port:5671,TargetPort:{1 0 amqp-ssl},NodePort:0,AppProtocol:nil,},ServicePort{Name:epmd,Protocol:TCP,Port:4369,TargetPort:{1 0 epmd},NodePort:0,AppProtocol:nil,},ServicePort{Name:dist,Protocol:TCP,Port:25672,TargetPort:{1 0 dist},NodePort:0,AppProtocol:nil,},ServicePort{Name:http-stats,Protocol:TCP,Port:15672,TargetPort:{1 0 http-stats},NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/instance: rabbitmq,app.kubernetes.io/name: rabbitmq,},ClusterIP:10.43.124.161,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalanc
erSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[10.43.124.161],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},}
2023/01/18 18:58:26 [DEBUG] Enqueueing Updated Service: &Service{ObjectMeta:{rabbitmq  data  0bc80adc-996b-4f98-962c-8798e5e1b2a9 220282944 0 2022-09-05 20:08:06 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:rabbitmq app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:rabbitmq helm.sh/chart:rabbitmq-8.31.5] map[meta.helm.sh/release-name:rabbitmq meta.helm.sh/release-namespace:data] [] []  [{helm Update v1 2022-09-05 20:08:06 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":4369,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":5671,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":5672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":15672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}},"k:{\"port\":25672,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}} {kubectl-edit Update v1 2023-01-17 17:02:41 +0000 UTC FieldsV1 {"f:spec":{"f:ports":{"k:{\"port\":4369,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":5671,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":5672,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":15672,\"protocol\":\"TCP\"}":{"f:targetPort":{}},"k:{\"port\":25672,\"protocol\":\"TCP\"}":{"f:targetPort":{}}}}}}]},Spec:ServiceSpec{Ports:[]ServicePort{ServicePort{Name:amqp,Protocol:TCP,Port:5672,TargetPort:{0 5672 },NodePort:0,AppProtocol:nil,},ServicePort{Name:amqp-ssl,Protocol:TCP,Port:5671,TargetPort:{0 5671 },NodePort:0,AppProtocol:nil,},ServicePort{Name:epmd,Protocol:TCP,Port:4369,TargetPort:{0 4369 },NodePort:0,AppProtocol:nil,},ServicePort{Name:dist,Protocol:TCP,Port:25672,TargetPort:{0 25672 },NodePort:0,AppProtocol:nil,},ServicePort{Name:http-stats,Protocol:TCP,Port:15672,TargetPort:{0 15672 },NodePort:0,AppProtocol:nil,},},Selector:map[string]string{app.kubernetes.io/instance: rabbitmq,app.kubernetes.io/name: rabbitmq,},ClusterIP:10.43.124.161,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,TopologyKeys:[],IPFamilyPolicy:*SingleStack,ClusterIPs:[10.43.124.161],IPFamilies:[IPv4],AllocateLoadBalancerNodePorts:nil,LoadBalancerClass:nil,InternalTrafficPolicy:*Cluster,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[]LoadBalancerIngress{},},Conditions:[]Condition{},},}
2023/01/18 18:58:26 [DEBUG] Processing Key: &{data Service rabbitmq 0xc0001d8780 Delete}
2023/01/18 18:58:26 [DEBUG] Process all the Virtual Servers which share same VirtualServerAddress
2023/01/18 18:58:26 [DEBUG] Processing Virtual Server dev2.messagebus.p2.internal for port 443
2023/01/18 18:58:26 [DEBUG] Configured rule: {vs__rabbitmq_15672_data  0 [0xc0007f4780] []}
2023/01/18 18:58:26 [DEBUG] Configured policy: {dev2_messagebus_p2_internal_vs_443__policy data  [forwarding]  true [http] [0xc0007f47e0] /Common/first-match}
2023/01/18 18:58:26 [DEBUG] Processing  BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:58:26 [DEBUG] Updated BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:58:26 [DEBUG] Updated Virtual dev2.messagebus.p2.internal with TLSProfile dev2.messagebus.p2.internal
2023/01/18 18:58:26 [DEBUG] Finished syncing virtual servers &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:dev2.messagebus.p2.internal GenerateName: Namespace:data SelfLink: UID:7ee7a089-faf6-4bed-b1aa-5e5283e5730b ResourceVersion:220277508 Generation:1 CreationTimestamp:2023-01-18 18:48:20 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[f5cr:true] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:kubectl-create Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:20 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {Manager:k8s-bigip-ctlr.real Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:40 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} Spec:{Host: HostGroup: VirtualServerAddress:10.40.15.207 IPAMLabel: VirtualServerName:dev2_messagebus_p2_internal_vs VirtualServerHTTPPort:0 VirtualServerHTTPSPort:0 Pools:[{Name: Path:/ Service:rabbitmq ServicePort:15672 NodeMemberLabel: Monitor:{Type:https Send:/ Recv: Interval:20 Timeout:31 TargetPort:0 Name: Reference:} Monitors:[] Rewrite: Balance: ServiceNamespace:}] TLSProfileName:dev2.messagebus.p2.internal HTTPTraffic: SNAT: WAF: RewriteAppRoot: AllowVLANs:[] IRules:[] ServiceIPAddress:[] PolicyName: PersistenceProfile: ProfileMultiplex: DOS: BotDefense: Profiles:{TCP:{Client: Server:} UDP: HTTP: HTTP2: RewriteProfile: PersistenceProfile: LogProfiles:[] ProfileL4: ProfileMultiplex:} AllowSourceRange:[]} Status:{VSAddress:None StatusOk:Ok}} (148.434µs)
2023/01/18 18:58:26 [DEBUG] Change in Service rabbitmq does not effect any VirtualServer for TransportServer
2023/01/18 18:58:26 [DEBUG] Processing Key: &{data Service rabbitmq 0xc00077a280 Create}
2023/01/18 18:58:26 [INFO] No IngressLink found in namespace data
2023/01/18 18:58:26 [DEBUG] Process all the Virtual Servers which share same VirtualServerAddress
2023/01/18 18:58:26 [DEBUG] Processing Virtual Server dev2.messagebus.p2.internal for port 443
2023/01/18 18:58:26 [DEBUG] Configured rule: {vs__rabbitmq_15672_data  0 [0xc0007f4960] []}
2023/01/18 18:58:26 [DEBUG] Configured policy: {dev2_messagebus_p2_internal_vs_443__policy data  [forwarding]  true [http] [0xc0007f49c0] /Common/first-match}
2023/01/18 18:58:26 [DEBUG] Processing  BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:58:26 [DEBUG] Updated BIGIP referenced profiles for 'VirtualServer' 'data'/'dev2.messagebus.p2.internal'
2023/01/18 18:58:26 [DEBUG] Updated Virtual dev2.messagebus.p2.internal with TLSProfile dev2.messagebus.p2.internal
2023/01/18 18:58:26 [ERROR] [CORE]Endpoints could not be fetched for service rabbitmq with targetPort 15672
2023/01/18 18:58:26 [INFO] No IngressLink found in namespace data
2023/01/18 18:58:26 [DEBUG] Finished syncing virtual servers &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:dev2.messagebus.p2.internal GenerateName: Namespace:data SelfLink: UID:7ee7a089-faf6-4bed-b1aa-5e5283e5730b ResourceVersion:220277508 Generation:1 CreationTimestamp:2023-01-18 18:48:20 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[f5cr:true] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:kubectl-create Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:20 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:f5cr":{}}},"f:spec":{".":{},"f:pools":{},"f:tlsProfileName":{},"f:virtualServerAddress":{},"f:virtualServerName":{}}}} {Manager:k8s-bigip-ctlr.real Operation:Update APIVersion:cis.f5.com/v1 Time:2023-01-18 18:48:40 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:status":{},"f:vsAddress":{}}}}]} Spec:{Host: HostGroup: VirtualServerAddress:10.40.15.207 IPAMLabel: VirtualServerName:dev2_messagebus_p2_internal_vs VirtualServerHTTPPort:0 VirtualServerHTTPSPort:0 Pools:[{Name: Path:/ Service:rabbitmq ServicePort:15672 NodeMemberLabel: Monitor:{Type:https Send:/ Recv: Interval:20 Timeout:31 TargetPort:0 Name: Reference:} Monitors:[] Rewrite: Balance: ServiceNamespace:}] TLSProfileName:dev2.messagebus.p2.internal HTTPTraffic: SNAT: WAF: RewriteAppRoot: AllowVLANs:[] IRules:[] ServiceIPAddress:[] PolicyName: PersistenceProfile: ProfileMultiplex: DOS: BotDefense: Profiles:{TCP:{Client: Server:} UDP: HTTP: HTTP2: RewriteProfile: PersistenceProfile: LogProfiles:[] ProfileL4: ProfileMultiplex:} AllowSourceRange:[]} Status:{VSAddress:None StatusOk:Ok}} (105.124µs)
2023/01/18 18:58:26 [DEBUG] Change in Service rabbitmq does not effect any VirtualServer for TransportServer
2023/01/18 18:58:26 [DEBUG] [CCCL] ConfigWriter (0xc0004ff320) writing section name gtm
2023/01/18 18:58:26 [DEBUG] [CCCL] ConfigWriter (0xc0004ff320) successfully wrote section (gtm)
2023/01/18 18:58:26 [DEBUG] Wrote gtm config section: map[]
2023/01/18 18:58:26 [DEBUG] [AS3] No change in dev2-p2 tenant configuration
2023/01/18 18:58:26 [DEBUG] Enqueueing Endpoints: &Endpoints{ObjectMeta:{rabbitmq  data  8e55aa3c-cd37-4afe-8a27-1abc4136febc 220282945 0 2022-09-05 20:08:06 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:rabbitmq app.kubernetes.io/managed-by:Helm app.kubernetes.io/name:rabbitmq helm.sh/chart:rabbitmq-8.31.5] map[] [] []  [{kube-controller-manager Update v1 2022-10-28 12:09:07 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:helm.sh/chart":{}}},"f:subsets":{}}}]},Subsets:[]EndpointSubset{EndpointSubset{Addresses:[]EndpointAddress{EndpointAddress{IP:10.42.129.48,TargetRef:&ObjectReference{Kind:Pod,Namespace:data,Name:rabbitmq-0,UID:70d0707e-fea9-44b0-8631-06cb1e95b381,APIVersion:,ResourceVersion:165158358,FieldPath:,},Hostname:,NodeName:*d2-cont-wkr1,},EndpointAddress{IP:10.42.209.239,TargetRef:&ObjectReference{Kind:Pod,Namespace:data,Name:rabbitmq-2,UID:7d3107a8-c4d2-4c01-9622-55ea06f37390,APIVersion:,ResourceVersion:149542348,FieldPath:,},Hostname:,NodeName:*d2-cont-wkr3,},EndpointAddress{IP:10.42.92.104,TargetRef:&ObjectReference{Kind:Pod,Namespace:data,Name:rabbitmq-1,UID:00461768-68f5-4ff3-a04c-a51ec7434d80,APIVersion:,ResourceVersion:214939094,FieldPath:,},Hostname:,NodeName:*d2-cont-wkr2,},},NotReadyAddresses:[]EndpointAddress{},Ports:[]EndpointPort{EndpointPort{Name:amqp,Port:5672,Protocol:TCP,AppProtocol:nil,},EndpointPort{Name:amqp-ssl,Port:5671,Protocol:TCP,AppProtocol:nil,},EndpointPort{Name:http-stats,Port:15672,Protocol:TCP,AppProtocol:nil,},EndpointPort{Name:epmd,Port:4369,Protocol:TCP,AppProtocol:nil,},EndpointPort{Name:dist,Port:25672,Protocol:TCP,AppProtocol:nil,},},},},}
2023/01/18 18:58:26 [DEBUG] Processing Key: &{data Endpoints rabbitmq 0xc001252140 Update}
2023/01/18 18:58:26 [DEBUG] [CCCL] ConfigWriter (0xc0004ff320) writing section name gtm
2023/01/18 18:58:26 [DEBUG] [CCCL] ConfigWriter (0xc0004ff320) successfully wrote section (gtm)
2023/01/18 18:58:26 [DEBUG] Wrote gtm config section: map[]
2023/01/18 18:58:26 [DEBUG] [AS3] PostManager Accepted the configuration
2023/01/18 18:58:26 [DEBUG] [AS3] posting request to https://10.40.12.16/mgmt/shared/appsvcs/declare/dev2-p2

@trinaths
Copy link
Contributor

@rajbaratht from your VS CRD example, please update servicePort with port name rather than number.

like,

apiVersion: cis.f5.com/v1
kind: VirtualServer
metadata:
  creationTimestamp: "2023-01-18T18:48:20Z"
  generation: 1
  labels:
    f5cr: "true"
  name: dev2.messagebus.p2.internal
  namespace: data
  resourceVersion: "220277508"
  uid: 7ee7a089-faf6-4bed-b1aa-5e5283e5730b
spec:
  pools:
  - monitor:
      interval: 20
      recv: ""
      send: /
      timeout: 31
      type: https
    path: /
    service: rabbitmq
    servicePort: http-stats               <=========== change here
  tlsProfileName: dev2.messagebus.p2.internal
  virtualServerAddress: 10.40.15.207
  virtualServerName: dev2_messagebus_p2_internal_vs
status:
  status: Ok
  vsAddress: None

@rajbaratht
Copy link
Author

@trinaths I get the following error If I use string instead of integer

k create -f dev2.messagebus.p2.internal_vs.yml
tlsprofile.cis.f5.com/dev2.messagebus.p2.internal created
error: error validating "dev2.messagebus.p2.internal_vs.yml": error validating data: ValidationError(VirtualServer.spec.pools[0].servicePort): invalid type for com.f5.cis.v1.VirtualServer.spec.pools.servicePort: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false

@trinaths
Copy link
Contributor

@rajbaratht - This feature is support for ServiceType LB. So this requirement is still a TODO for VS and TS CRD.

Please mail this requirement to automation_toolchain_pm at f5 dot com

@trinaths trinaths added feature-request and removed bug awaiting response Awaiting response labels Jan 20, 2023
@trinaths
Copy link
Contributor

Created CONTCNTR-3778 for internal tracking

@trinaths trinaths added the JIRA label Jan 25, 2023
@lavanya-f5 lavanya-f5 mentioned this issue Jan 31, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request JIRA
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

named port support for CRD lavanya-f5/k8s-bigip-ctlr
2 participants
@trinaths @rajbaratht