chaining: source security ID in hairpin might lead to policy denies in >=v1.7.4

[aanm]

Bisecting blame 60b4210

xx drop (Policy denied) flow 0x53352ab to endpoint 694, identity 4417->56410: 172.16.166.201:53662 -> 172.16.166.200:80 tcp SYN

The security ID in the endpoint is different from the one being generated by the datapath:

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                       IPv6   IPv4             STATUS   
           ENFORCEMENT        ENFORCEMENT                                                                                                            
366        Disabled           Disabled          4428       k8s:class=tiefighter                                                     172.16.166.201   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                          
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                   
                                                           k8s:io.kubernetes.pod.namespace=default                                                           
                                                           k8s:org=empire

[tgraf]

Is this a potential collision in the mark value?

[tgraf]

It looks like some bits changed:

4417 1000101000001
4428 1000101001100

[tgraf]

@aanm How did you get into this state?

[aanm]

In that case yes, but it is more common to be off by 1

[tgraf]

This could be a --set 1/0xF or similar where 4 bits get cleared and one bit gets set.

[jrfastab]

To close out discussion, this was caused by a conflicting mark value. So added a PR to opt-out of setting identity field in the mark. This will force the ingress path to do another lookup and obfuscate the trace logs some, but will allow policy to work correctly.

[borkmann]

[Only thing left here is a doc follow-up to update the chaining guide.]

[borkmann]

Fixed via #12185 and #12194.