read: connection reset by peer at high load

[ashrafguitoni]

What version of Knative?

1.7.1

Note: I was able to reproduce the issue in Knative 1.9.X as well.

Expected Behavior

We have a microservices architecture implemented, where some Knative services call other Knative services. At high load, we expect the system to still perform normally.

Note: some of the services haves two user containers: a python application run via dumb-init and an instance of Nvidia Triton. The python application run via dumb-init does terminate gracefully (based on the container's log messages when a pod is terminated).

Actual Behavior

This error appears (rarely but consistently):

read tcp 127.0.0.1:49906->127.0.0.1:8877: read: connection reset by peer

Note: 8877 is the port of our application container

Steps to Reproduce the Problem

I'm not sure if this will be easily reproducible with bare-bone services, but I can try to create a minimum reproducible example if the error isn't very well known.

[dprotaso]

This was fixed in v1.10 - can you try upgrading to the latest version and retest

The PR with the fix - #13815

/triage needs-user-input

[ashrafguitoni]

@dprotaso

Thank you for the reply. I had tried that version and a similar error was appearing actually: #12387

Perhaps I'll have to wait for that issue to be fixed, then upgrade, if there's no other solution.

[tikr7]

@ashrafguitoni do you have workaround or how do you deal with this situation at the moment?

[ashrafguitoni]

@ashrafguitoni do you have workaround or how do you deal with this situation at the moment?

Unfortunately, the only thing we could do is have a certain number of retries or the requests. This is "acceptable" for us because we use knative services only internally, so whatever consumes them can just retry the request a few times. It's really not ideal at all.

[tikr7]

Okay, thank you for the info. We use at the moment more the Knative Eventing with Broker and Trigger. There we configured the retry mechanism. Something like:

  delivery:
    deadLetterSink:
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: blurring-eventdisplay-dlq
    backoffDelay: PT3S
    backoffPolicy: exponential
    retry: 5

Of course the eventing architecture is different in applications in comparison to the normal api consumption.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[ajstewart]

Hi @ashrafguitoni I'm also wondering if you managed to resolve this? I also have some connection reset by peer messages in the queue proxy logs. Also a bit lost as to why and relying on consumer retries.

[dprotaso]

@ajstewart the other thing it could be is that go doesn't enable full duplex by default - you can do that using this annotations

See: https://knative.dev/docs/serving/services/http-protocol/#configure-http1-full-duplex-support

Can you try that out - and create an issue if you can still reproduce your issue

[ajstewart]

@ajstewart the other thing it could be is that go doesn't enable full duplex by default - you can do that using this annotations

See: https://knative.dev/docs/serving/services/http-protocol/#configure-http1-full-duplex-support

Can you try that out - and create an issue if you can still reproduce your issue

Thanks for the suggestion! It turns out I was not being smart with my concurrency settings (as in I was trying to be concurrent when it couldn't cope with it). Pretty sure this was what was causing the errors as after sorting that out (hard limiting to 1) I no longer see the random failures when under load.