Traffic shadowing not working as expected

[aggarwala3]

Describe the bug
We're looking to replicate the traffic from original jenkins to another parallel running jenkins. We have added zalando.org/skipper-filter in original jenkins with tee predicate to replicate the traffic to another-jenkins.company.org but as soon as we add this annotation, jenkins API calls stop working and result in timeout.
Annotation added

    zalando.org/skipper-filter: tee("https://another-jenkins.com.org")

To Reproduce
Add context, for example:

1. route or ingress definition
2. http request with headers (truncate your secrets)
3. response with headers

Expected behavior
We'd expect same traffic to reach another-jenkins instance as well

Observed behavior
But we only see timeout in original jenkins post this addition of annotation on skipper ingress.

[szuecs]

Please show the logs that show the timeouts.
Does the request have a body?
Does another Jenkins see the http request headers?

I am pretty sure that the problem is that the target application is not able to read the body fast enough and therefore the other call hangs in the I/O pipe. That's expected behavior enforced by the current efficient implementation which we likely won't change. It's likely a side effect of being efficient.

[aggarwala3]

Here are the logs

│ [APP]time="2024-12-02T12:57:20Z" level=warning msg="tee: error while tee request Post \"https://another-jenkins.com.org/github-webhook/\": dial tcp x.x.x.x:443: connect: connection timed o │
│ [APP]time="2024-12-02T12:57:20Z" level=error msg="tee: error while tee requestio: read/write on closed pipe"                                                                                                                     │

I doubt if its about reading body fast enough, since both applications are in same network and there shouldnt be much of delay.

Plus, adding this annotation on ingress meant for original jenkins breaks the original jenkins itself and all webhook calls meant for "original" jenkins are broken post that with above logs.

[MustafaSaber]

Hi, I tried to reproduce but I can't.

Made 2 deployments (deployment, service & ingress) and added tee to one of them but I get requests on both services. So it would be a bit hard to debug without more details about the setup. The bigger the body the more probable you will face the issue as @szuecs mentioned because we split request into buffers and if one of the backends is slow it will make both slow

reproducer.yaml

apiVersion: v1
kind: Namespace
metadata:
  name:  foo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: simple2
  namespace: foo
spec:
  selector:
    matchLabels:
      application: app-test
  template:
    metadata:
      labels:
        application: app-test
    spec:
      containers:
      - name: simple2
        image: registry.opensource.zalan.do/teapot/skipper:latest
        args:
            - skipper
            - -inline-routes
            - '* -> logHeader() -> inlineContent("Busted!") -> <shunt>'
            - -address=:80
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
          name:  http
---
apiVersion: v1
kind: Service
metadata:
  name: simple2
  namespace: foo
spec:
  selector:
    application: app-test
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple2
  namespace: foo
  annotations:
    zalando.org/skipper-filter: tee("https://another-simple2.example.com")
spec:
  rules:
  - host: simple2.example.com
    http:
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: simple2
            port:
              number: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: another-simple2
  namespace: foo
spec:
  selector:
    matchLabels:
      application: app
  template:
    metadata:
      labels:
        application: app
    spec:
      containers:
      - name: another-simple2
        image: registry.opensource.zalan.do/teapot/skipper:latest
        args:
            - skipper
            - -inline-routes
            - '* -> logHeader() -> inlineContent("another-simple2!") -> <shunt>'
            - -address=:80
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
          name:  http
---
apiVersion: v1
kind: Service
metadata:
  name: another-simple2
  namespace: foo
spec:
  selector:
    application: app
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: another-simple2  
  namespace: foo
spec:
  rules:
  - host: another-simple2.example.com
    http:
      paths:
      - pathType: ImplementationSpecific
        backend:
          service:
            name: another-simple2 
            port:
              number: 80

[szuecs]

The known fact is that the io.PipeWriter that does the tee of the body can block the main request in case the shadow is slow.
We see connection timeout to "https://another-jenkins.com.org/github-webhook/", which will likely block the main request and therefore it's down.