Bug: Network Security Group rules cleared once an hour

[sinap-se]

Version of Azure Service Operator
v2.0.0-beta.5

Describe the bug
Network Security Group (NSG) security rules are removed at least once an hour. The rules are re-added some time afterwards.

To Reproduce

• Have two resource groups: A and B.
• Have an Azure Kubernetes Service (AKS) cluster in resource group A.
• Have Azure Service Operator (ASO) installed on resource group A.
• Apply an NSG to the AKS cluster that adds an NSG to resource group B.
• Note that the NSG has appeared in the Azure portal in resource group B.
• Add an NSG security rule targeting the NSG.
• Note that the NSG rule appears in the Azure portal on the newly created NSG.
• Wait for approximately an hour.
• Note that the rules disappear from the NSG in the portal.
• Note that the activity log shows NSG and its rules changing despite there being no changes to the Kubernetes resources.

Expected behavior

• Rules must persist indefinitely when the resources defining them do not change.

Screenshots

Additional context

• When ASO is removed from the cluster, the erroneous behavior stops.
• The rules are re-added by the Create or Update Network Security Group event OR the Create or Update Security Rule event depending on timing.

Activity Log JSON

{
    "authorization": {
        "action": "Microsoft.Network/networkSecurityGroups/write",
        "scope": "/subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter"
    },
    "caller": "24401744-526c-46dd-a3b5-83477c9c737a",
    "channels": "Operation",
    "claims": {
        "aud": "https://management.core.windows.net/",
        "iss": "https://sts.windows.net/<tenant-id>/",
        "iat": "1686068081",
        "nbf": "1686068081",
        "exp": "1686071981",
        "aio": "E2ZgYPj5/JRq++TISTIRPrJf99uGROT5X0s6djdeukVe4tbZLYIA",
        "appid": "53957710-c8aa-442d-a03c-fad52de7127d",
        "appidacr": "1",
        "http://schemas.microsoft.com/identity/claims/identityprovider": "https://sts.windows.net/<tenant-id>/",
        "idtyp": "app",
        "http://schemas.microsoft.com/identity/claims/objectidentifier": "24401744-526c-46dd-a3b5-83477c9c737a",
        "rh": "0.ARsAMb8yLbvUnkKAS0IX-055TEZIf3kAutdPukPawfj2MBMbAAA.",
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier": "24401744-526c-46dd-a3b5-83477c9c737a",
        "http://schemas.microsoft.com/identity/claims/tenantid": "<tenant-id>",
        "uti": "uUGDu9kFzkyEqrU940AKAA",
        "ver": "1.0",
        "xms_tcdt": "1471612579"
    },
    "correlationId": "f46eb54a-dbe1-42d9-b741-aa3b9ef90ab2",
    "description": "",
    "eventDataId": "1b2fb287-3e65-44a3-9956-3cd0fff3b544",
    "eventName": {
        "value": "EndRequest",
        "localizedValue": "End request"
    },
    "category": {
        "value": "Administrative",
        "localizedValue": "Administrative"
    },
    "eventTimestamp": "2023-06-06T16:50:09.5268259Z",
    "id": "/subscriptions/<subscription-id>/resourcegroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/events/1b2fb287-3e65-44a3-9956-3cd0fff3b544/ticks/638216670095268259",
    "level": "Informational",
    "operationId": "149b2ffc-79fd-4ce3-802b-3d1e027fd68c",
    "operationName": {
        "value": "Microsoft.Network/networkSecurityGroups/write",
        "localizedValue": "Create or Update Network Security Group"
    },
    "resourceGroupName": "nsgtest-worker",
    "resourceProviderName": {
        "value": "Microsoft.Network",
        "localizedValue": "Microsoft.Network"
    },
    "resourceType": {
        "value": "Microsoft.Network/networkSecurityGroups",
        "localizedValue": "Microsoft.Network/networkSecurityGroups"
    },
    "resourceId": "/subscriptions/<subscription-id>/resourcegroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter",
    "status": {
        "value": "Succeeded",
        "localizedValue": "Succeeded"
    },
    "subStatus": {
        "value": "",
        "localizedValue": ""
    },
    "submissionTimestamp": "2023-06-06T16:53:10Z",
    "subscriptionId": "<subscription-id>",
    "tenantId": "<tenant-id>",
    "properties": {
        "eventCategory": "Administrative",
        "entity": "/subscriptions/<subscription-id>/resourcegroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter",
        "message": "Microsoft.Network/networkSecurityGroups/write",
        "hierarchy": "<tenant-id>/<org>/Development/<subscription-id>"
    },
    "relatedEvents": []
}

Resource YAML:

apiVersion: network.azure.com/v1beta20201101
kind: NetworkSecurityGroupsSecurityRule
metadata:
  annotations:
    io.javaoperatorsdk/primary-name: demo-filter-rule
    io.javaoperatorsdk/primary-namespace: default
    serviceoperator.azure.com/operator-namespace: azureserviceoperator-system
    serviceoperator.azure.com/resource-id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/securityRules/samplerule1
  creationTimestamp: "2023-05-30T20:43:08Z"
  finalizers:
    - serviceoperator.azure.com/finalizer
  generation: 1
  labels:
    NetworkFilterRule: "true"
  name: samplerule1
  namespace: default
  ownerReferences:
    - apiVersion: network.azure.com/v1beta20201101storage
      kind: NetworkSecurityGroup
      name: demo-filter
      uid: 5f5948ab-db98-46c3-887a-4a96e58e8f3f
  resourceVersion: "8672992"
  uid: 5ae2c407-f4d3-46f1-a086-59dc9759b28c
spec:
  access: Deny
  azureName: samplerule1
  description: Allow access to source port 23-45 and destination port 45-56
  destinationAddressPrefixes:
    - 0.0.0.0/0
  destinationPortRange: 46-56
  direction: Inbound
  owner:
    name: demo-filter
  priority: 1001
  protocol: Esp
  sourceAddressPrefixes:
    - 0.0.0.0/0
  sourcePortRange: '*'
status:
  access: Deny
  conditions:
    - lastTransitionTime: "2023-06-01T16:01:40Z"
      observedGeneration: 1
      reason: Succeeded
      status: "True"
      type: Ready
  description: Allow access to source port 23-45 and destination port 45-56
  destinationAddressPrefixes:
    - 0.0.0.0/0
  destinationPortRange: 46-56
  direction: Inbound
  etag: W/"e217983a-ebb7-4165-bc1c-4295ab334a2b"
  id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/securityRules/samplerule1
  name: samplerule1
  priority: 1001
  protocol: Esp
  provisioningState: Succeeded
  sourceAddressPrefixes:
    - 0.0.0.0/0
  sourcePortRange: '*'
  type: Microsoft.Network/networkSecurityGroups/securityRules

apiVersion: network.azure.com/v1beta20201101
kind: NetworkSecurityGroup
metadata:
  annotations:
    io.javaoperatorsdk/primary-name: demo-filter
    io.javaoperatorsdk/primary-namespace: default
    serviceoperator.azure.com/operator-namespace: azureserviceoperator-system
    serviceoperator.azure.com/resource-id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter
  creationTimestamp: "2023-05-30T20:03:53Z"
  finalizers:
    - serviceoperator.azure.com/finalizer
  generation: 1
  labels:
    NetworkFilter: "true"
  name: demo-filter
  namespace: default
  ownerReferences:
    - apiVersion: resources.azure.com/v1beta20200601
      kind: ResourceGroup
      name: nsgtest-worker
      uid: 4f3a7866-6fcf-4400-8825-2ad0f0e28aee
  resourceVersion: "8675708"
  uid: 5f5948ab-db98-46c3-887a-4a96e58e8f3f
spec:
  azureName: demo-filter
  location: canadacentral
  owner:
    name: nsgtest-worker
status:
  conditions:
    - lastTransitionTime: "2023-06-01T16:09:35Z"
      observedGeneration: 1
      reason: Succeeded
      status: "True"
      type: Ready
  defaultSecurityRules:
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/AllowVnetInBound
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/AllowAzureLoadBalancerInBound
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/DenyAllInBound
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/AllowVnetOutBound
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/AllowInternetOutBound
    - id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter/defaultSecurityRules/DenyAllOutBound
  etag: W/"c83fda4a-d988-447b-ac20-a4ccc63687ea"
  id: /subscriptions/<subscription-id>/resourceGroups/nsgtest-worker/providers/Microsoft.Network/networkSecurityGroups/demo-filter
  location: canadacentral
  name: demo-filter
  provisioningState: Succeeded
  resourceGuid: resource-id
  type: Microsoft.Network/networkSecurityGroups

[matthchr]

I've confirmed this is a bug by reviewing the networking teams Swagger specification.

We need to fix this the same way we've fixed VNET/Subnet and RouteTable/Route, (with a customized extension).