Queries failing after 30s despite changing dataproxy.timeout to > 30s

[kahinton]

What happened:
Long running queries to Azure Log Analytics are failed after 30 seconds with a 504 Gateway timeout issue despite dataproxy.timeout set above 30s when run behind a network proxy.

What you expected to happen:
We expect that queries should not appear to time out in less time than has been configured in the dataproxy.timeout setting.

How to reproduce it (as minimally and precisely as possible):
Run a computationally heavy query to Azure Log Analytics with the dataproxy.timeout set above 30 seconds. This appears to be occurring only when the query is run through a network proxy. When running locally on a machine not utilizing the proxy the issue does not seem to be present.

Anything else we need to know?:
I'm adding a pull request that will make the dataproxy keepalive and idle connection timeout configurable, while keeping the defaults the same as they currently are. It appears that the keepalive request always fails when this proxy is utilized, and having the ability to change the keepalive time should help us to resolve this. I also figured I would add the setting for idle connection timeout for anyone that may have a reason to change it. There seem to have been a few issues reported previously that I think may have been caused by similar issues, however they had managed to simply lower the query time.

Environment:

• Grafana version: At least since 7.x.x
• Data source type & version: Azure Monitor 0.3.0
• OS Grafana is installed on: Running Docker 7.2.0-ubuntu
• User OS & Browser: Reported across multiple OS and Browser types
• Grafana plugins:
• Others:

[marefr]

Interesting. To resolve your problem you have to change keepalive and idle connection timeout to what?

[kahinton]

In our situation we need to set the keepalive to essentially match the dataproxy timeout. It seems that the network proxy in use isn't handling the keep alive message well and it leads to the connection being marked as closed. I added the setting for the idle connection timeout just for anyone that has a need to more tightly control open connections to a datasource.

[anarcher]

I have the same issue. dataproxy timeout is above 30s but always datasource slow queries (loki, >= 1m) are failed with bad gateway(http: proxy error: EOF) and there are no error logs from query-frontend in loki.

[morvencao]

I'm using grafana v7.4.2, still have the same issue.

Although I have updated the timeout and keep_alive_seconds to 300 seconds

[dataproxy]
timeout = 300
keep_alive_seconds = 300
logging = true

The requests are always timeout after 30 seconds, see: