Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent 502 proxy errors when running behind Apache #2687

Open
panovitch opened this issue Jan 25, 2018 · 16 comments

Comments

@panovitch
Copy link

@panovitch panovitch commented Jan 25, 2018

Hi there! Was wondering if you have some insight on the following problem:

Long story short

We are running an aiohttp app behind an Apache proxy. Around 20% of requests to the app cause Apache to return 502 Proxy Error.

These happen with requests to all endpoints of the server, and there seems to be no pattern to errors of any kind. The proxy error is returned immediately upon request.
Other servers/apps (built on other frameworks, for example, tornado) run behind the same Apache proxy and do not show the same behavior.

The example of Apache logs:

[Tue Jan 23 12:38:51.050529 2018] [proxy_http:error] [pid <pid>:tid <tid>] (20014)Internal error (specific information not available): [client <IP address>] AH01102: error reading status line from remote server <hostname>

Full error response example:

HTTP/1.1 502 Proxy Error
Connection: Keep-Alive
Content-Length: 502
Content-Type: text/html; charset=iso-8859-1
Date: Tue, 23 Jan 2018 13:47:29 GMT
Keep-Alive: timeout=15, max=100
Server: Apache/2.4.18 (Ubuntu)

502 Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request.
Reason: Error reading from remote server

Your environment

aiohttp==2.3.9
python 3.6.3
Apache 2.4.18

@samuelcolvin

This comment has been minimized.

Copy link
Member

@samuelcolvin samuelcolvin commented Jan 25, 2018

Do you see anything in the logs from aiohttp?

How are you running aiohttp? With gunicorn or directly?

I guess debug level logs from aiohttp (and gunicorn if it's running) would help.

Also how is the aiohttp server addressed by apache (by an ip or a hostname)? I've had similar issues that turned out to be DNS problems which were fixed by using static IPs.

@panovitch

This comment has been minimized.

Copy link
Author

@panovitch panovitch commented Jan 25, 2018

Hi @samuelcolvin,
We are running aiohttp directly.
The Apache addresses the server by a hostname, but I doubt that issue is DNS-related, since other apps also run on the same host and dont have any problems.
There are no records in aiohttp logs for the problematic requests at all.

@asvetlov

This comment has been minimized.

Copy link
Member

@asvetlov asvetlov commented Jan 26, 2018

Maybe related to #2682

@aalhour

This comment has been minimized.

Copy link

@aalhour aalhour commented Feb 1, 2018

Hello @asvetlov,

Do you have any recommendation for us to validate the source of the issue? How can we help you guys resolve this problem?

@asvetlov

This comment has been minimized.

Copy link
Member

@asvetlov asvetlov commented Feb 1, 2018

I'm working on it

@asvetlov

This comment has been minimized.

Copy link
Member

@asvetlov asvetlov commented Feb 1, 2018

Sorry, after digging in I did not found expected bug.
Cannot reproduce locally.

aiohttp error log is required (with DEBUG level enabled) for getting more information about the problem.

@mariuszluciow

This comment has been minimized.

Copy link

@mariuszluciow mariuszluciow commented Feb 1, 2018

I'm having something similar, running aiohttp behind zuul proxy. After several good responses, and a while of doing nothing, server starts responding with incorrect response, which starts with \r\nTP/1.1 200 OK instead of HTTP/1.1 200 OK

@asvetlov

This comment has been minimized.

Copy link
Member

@asvetlov asvetlov commented Feb 2, 2018

  1. Not sure if you have the same problem.
  2. I need reliable way for reproducing the error -- otherwise a change for fix is very low. Python test would be awesome.
@panovitch

This comment has been minimized.

Copy link
Author

@panovitch panovitch commented Feb 2, 2018

Hi @asvetlov,

We tried to check the aiohttp debug logs, that's how we tried to do it, and how we are running the app:

def setup_loggers():
    stream_handler = logging.StreamHandler(stream=sys.stderr) 
    _stream_handler.setLevel(logging.DEBUG)

    loggers = [getLogger('aiohttp.internal'), getLogger('aiohttp.server')]
    for logger in loggers:
        logger.setLevel(logging.DEBUG)
        logger.addHandler(_stream_handler)

def run_app():
    """
    Application runner function.
    """
    setup_loggers()
    app = create_app()
    host = server_option("host")
    port = int(server_option("port"))

    web.run_app(
        app=app, host=host, port=port, access_log=make_logger(is_development=True),
        access_log_format=access_log_format)

We ran the app with this configuration, but were unable to see anything related to the problematic requests - they do not even register in the access log.

I'm posting the Apache config below, hope it might help you reproduce it.

Apache Configuration

apache2.conf

Default config. Nothing special here.

sites-enabled

ServerName mine.localhost.com
ProxyRequests Off

<Location /aiohttp_app/>
ProxyPass http://localhost:8137/ retry=0
ProxyPassReverse http://localhost:8137/
</Location>

Apache Error Logs

[Fri Feb 02 16:05:41.865610 2018] [proxy_http:error] [pid 6146:tid 140638813112064] (20014)Internal error (specific information not available): [client 127.0.0.1:39048] AH01102: error reading status line from remote server localhost:8137
[Fri Feb 02 16:05:41.865899 2018] [proxy:error] [pid 6146:tid 140638813112064] [client 127.0.0.1:39048] AH00898: Error reading from remote server returned by /aiohttp_app/ping
[Fri Feb 02 16:05:41.875609 2018] [proxy_http:error] [pid 6147:tid 140638787933952] (20014)Internal error (specific information not available): [client 127.0.0.1:39050] AH01102: error reading status line from remote server localhost:8137
[Fri Feb 02 16:05:41.875797 2018] [proxy:error] [pid 6147:tid 140638787933952] [client 127.0.0.1:39050] AH00898: Error reading from remote server returned by /aiohttp_app/ping
@asvetlov

This comment has been minimized.

Copy link
Member

@asvetlov asvetlov commented Feb 2, 2018

Errors are stored in logger named aiohttp.server.
I suggest calling logging.basicConfig(level=logging.DEBUG) to see all logs.

@mariuszluciow

This comment has been minimized.

Copy link

@mariuszluciow mariuszluciow commented Feb 2, 2018

I spend few hours to produce some small replicable example with no luck. If anything changes I will let you know, thanks anyway!

@panovitch

This comment has been minimized.

Copy link
Author

@panovitch panovitch commented Feb 5, 2018

Hi @asvetlov, running the app with logging.basicConfig(level=logging.DEBUG) also doesn't show anything. To help reproduce the problem, I've created this: https://github.com/panovitch/test_aiohttp_502. The repo contains a sample apache config (main config apache.conf, which is not different from the default one AFAIK, and a site config for sites-enabled), a way to test for error, and an app (a copy of the example app really). Please note that error occurs pretty randomly so give the test some time to run.

@panovitch

This comment has been minimized.

Copy link
Author

@panovitch panovitch commented Mar 2, 2018

Hi everyone,

We eventually managed to solve the problem by adding this to our apache proxy config: disablereuse=on -hope this would be useful!

@pfreixes

This comment has been minimized.

Copy link
Contributor

@pfreixes pfreixes commented Apr 7, 2018

@panovitch do you have the chance to test if the error is reproducible with Aiohttp 3.x?

Regarding the disablereuse=on you are forcing to open a TCP connection at each time. is this stack running in local or are separated pieces within a network?

@lordmauve

This comment has been minimized.

Copy link

@lordmauve lordmauve commented May 4, 2018

I am seeing this with aiohttp==3.1.3 and Apache 2.4.10.

@rcupic

This comment has been minimized.

Copy link

@rcupic rcupic commented Jul 26, 2019

Hi everyone,

We eventually managed to solve the problem by adding this to our apache proxy config: disablereuse=on -hope this would be useful!

Hello,
Thank you for letting us know your solution. My question is if there is any downgrade in performance with this setup? I have something like 20-30 requests per minute. This setting sounds like disabling pooled connections which means that every of this 20-30 requests will open new connection for itself even if they are from a same client.
@panovitch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.