AddressGroup is not updated when Pod IP changes due to Node restart

[Dyanngg]

Describe the bug
Special thanks to @tnqn for initial triage on this issue.
In a cluster where Pods are running on multiple Nodes, if a Node is restarted, the Pods on that Node might get new IPs assigned to them. When this happens, NetworkPolicies that refer to these Pods in ingress/egress peers are expected to be correctly enforced. Antrea Controller should compute updated IPs for corresponding addressGroups and sent to Antrea Agents who reconciles internal NPs that refer to these addressGroups. However, these update events are not getting generated, because since #1467 the GroupMember key becomes entity namespaced-name for both appliedToGroup and addressGroup members. Across the update, even though IPs have changed, each GroupMember will still have the same key. Thus addressGroup patches are skipped, and Antrea agents will fail to correctly enforce NetworkPolicies.

To Reproduce

1. Create a src Namespace

# kubectl get ns test-ns --show-labels=true
NAME      STATUS   AGE     LABELS
test-ns   Active   2d23h   project=test

2. Create a src Pod

# kubectl get pods -n test-ns --show-labels=true -o wide | grep tcp-3-rc-zzbgs
tcp-3-rc-zzbgs   1/1     Running   2          2d23h   10.128.80.79     scale-workload-cluster-new-md-0-74d4678-d5rlf   <none>           <none>            app=test

3. Create a dst Pod

# kubectl get pods -n demo-99-zwlilc -o wide --show-labels=true | grep scale-ns-zwlilc-rc-1-7v69z
scale-ns-zwlilc-rc-1-7v69z   1/1     Running            1          8h    10.128.12.59    scale-workload-cluster-new-md-0-74d4678-q6g9s   <none>           <none>            app=scale

4. Apply an ANP to DENY traffic from src to dst

# cat ingress-np.yaml
apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
kind: NetworkPolicy
metadata:
  name: test-anp-1
spec:
    priority: 5
    tier: securityops
    appliedTo:
      - podSelector:
          matchLabels:
            app: scale
    ingress:
      - action: Drop
        from:
            namespaceSelector:
              matchLabels:
                project: test

5. Restart the Node where tcp-3-rc-zzbgs (ingress peer) is running on.

Expected
After Node restart, traffic from src Pod to dst Pod should be denied.

Actual behavior
After Node restart, traffic from src Pod to dst Pod is not denied.
AddressGroup info is out of sync in Controller and Agent. The IP of the Pod in one of antrea-agents:

# kubectl exec -it antrea-agent-vg9kx -n kube-system -- antctl get ag be2afa10-1020-5702-b929-95b4845648e6 -o yaml |grep -C 2 scale-ns-zwlilc-rc-1-7v69z
- ip: 10.128.12.214
  pod:
    name: scale-ns-zwlilc-rc-1-7v69z
    namespace: demo-99-zwlilc

The IP of the same Pod in antrea-controller:

# kubectl exec -it deploy/antrea-controller -n kube-system -- antctl get addressgroup be2afa10-1020-5702-b929-95b4845648e6 -o yaml |grep -C 2 scale-ns-zwlilc-rc-1-7v69z
- ip: 10.128.12.59
  pod:
    name: scale-ns-zwlilc-rc-1-7v69z
    namespace: demo-99-zwlilc

Versions:
Antrea v0.12 release