Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow configuring idle_timeout Envoy parameter on HTTP Connection Manager #2155

Open
neufeldtech opened this issue Jan 8, 2020 · 0 comments
Open

Comments

@neufeldtech
Copy link

@neufeldtech neufeldtech commented Jan 8, 2020

Please describe your use case / problem.
We currently have Ambassador deployed on-premise behind a firewall and fronted by a CDN. We encountered an issue where on low-traffic services, our clients were receiving some 503 first byte timeouts.

The issue is present because when the firewall chose to remove idle tcp sessions from its session table, it would not send a FIN or RST to the CDN. On the next request, the CDN thinks that it still has a tcp connection open and attempts to re-use the now-closed TCP connection, resulting in the request getting 'black-holed', and leads to an HTTP 503.

Because Envoy doesn't ever attempt to close idle tcp connections by default, we're left with Envoy thinking that sessions are open, but our firewall has closed them after a 30 minute timeout (without notifying the CDN that the connection is closed), and the CDN still thinks that the connection is open.

Our "options" for remediating this scenario were:

  • Configure tcp idle timeout on the CDN side to control connection idle time
    • This turned out to not be an option that we can readily control on our CDN. They have 'dynamic' connection times.
  • Make the firewall send a FIN or RST to the CDN when it removes a session from its session table
    • We investigated this as well, but on our firewall this is not an option that is configurable.
  • Have Envoy close idle tcp connections gracefully after a configurable timeout (in our case, some timer that is lower than our firewall idle session timeout)
    • We believe that this is the most appropriate way to solve the issue, and gives us control when tuning idle timeouts across each network device in the path, so this issue requests making the idle_timeout Envoy setting configurable on the Envoy http listener.

Describe the solution you'd like
Would like to be able to have the idle_timeout on the downstream HTTP Listener configurable for the reason mentioned above.

Describe alternatives you've considered
See description above.

Additional context
Looks like #1738 originally brought this up, but was closed as stale

I originally opened #2126 for this, but because of Github shenanigans the PR got closed when I was updating my fork. Created this issue for tracking, then will link PR's to this when I have one ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.