APISIX occasionally returns 500 for httpbin.org

[sshniro]

Issue description

For testing, I use httpbin.org as an upstream address to echo the parameters. But I'm starting to get 500 for consequent hits for the same address. And occasionally it returns 200 again, the behavior is unpredictable.

Does anyone know why is this happening? Is this something related to caching or timeouts?

I tested httpbin.org without proxying via APISIX with a 1-second loop and the response is always 200.

The following is the screenshot when I access httpbin via apisix with 1-second interval.

The following is the screenshot when I access httpbin without any proxy with 1-second interval.

The following is my configuration.

curl "http://127.0.0.1:9080/apisix/admin/upstreams/50" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "type": "roundrobin",
    "nodes": {
        "httpbin.org:443": 1
    }
}'

curl http://127.0.0.1:9080/apisix/admin/routes/5 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "/samplePrefix/get",
    "plugins": {
        "proxy-rewrite": {
          "scheme": "https",
          "regex_uri": ["^/samplePrefix/get(.*)", "/get$1"]
        }
    },
    "upstream_id": 50
}'

curl -i -X GET http://127.0.0.1:9080/samplePrefix/get?param1=foo&param2=bar

curl -i -X GET https://httpbing.org

[membphis]

@sshniro please provide your error log when you got 500 responses code, I need to confirm it.

I guess you are using an older version of Apache APISIX. I have fixed this error for a few days.

You can make a try with the latest version of APISIX.

[sshniro]

@membphis , I'm using the master branch, There are no helpful messages printed when I get 500, This is what I get during startup.

[sshniro]

@membphis This is reproducible with the latest master branch, What does RLLIMIT mean in this context?

[membphis]

I'm using the master branch, There are no helpful messages printed when I get 500, This is what I get during startup.

If we can't find any associated error logs from the local file, then this 500 error response is from httpbin and it was not generated by APISIX.

Your Nginx worker does not have permissions, which limits the number of open files.

[membphis]

I tried your example, I got sucess

$ curl "http://127.0.0.1:9080/apisix/admin/upstreams/50" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "type": "roundrobin",
    "nodes": {
        "httpbin.org:443": 1
    }
}'
{"node":{"value":{"nodes":{"httpbin.org:443":1},"hash_on":"vars","type":"roundrobin"},"createdIndex":2103,"key":"\/apisix\/upstreams\/50","modifiedIndex":2103},"action":"set"}

$ curl http://127.0.0.1:9080/apisix/admin/routes/5 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri": "/samplePrefix/get",
    "plugins": {
        "proxy-rewrite": {
          "scheme": "https",
          "regex_uri": ["^/samplePrefix/get(.*)", "/get$1"]
        }
    },
    "upstream_id": 50
}'
{"node":{"value":{"upstream_id":50,"plugins":{"proxy-rewrite":{"regex_uri":["^\/samplePrefix\/get(.*)","\/get$1"],"scheme":"https"}},"uri":"\/samplePrefix\/get","priority":0},"createdIndex":2104,"key":"\/apisix\/routes\/5","modifiedIndex":2104},"action":"set"}

$ curl -i -X GET http://127.0.0.1:9080/samplePrefix/get\?param1\=foo\&param2\=bar
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 367
Connection: keep-alive
Date: Sat, 14 Mar 2020 13:42:50 GMT
Server: APISIX web server
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {
    "param1": "foo",
    "param2": "bar"
  },
  "headers": {
    "Accept": "*/*",
    "Host": "127.0.0.1",
    "User-Agent": "curl/7.66.0",
    "X-Amzn-Trace-Id": "Root=1-5e6cdf59-6808716820e92f54dfa60584",
    "X-Forwarded-Host": "127.0.0.1"
  },
  "origin": "127.0.0.1, 223.73.235.21",
  "url": "https://127.0.0.1/get?param1=foo&param2=bar"
}

[sshniro]

Thanks, Yes it was working but gave 500 intermittently, I will try with another service and get back to you.

BTW, a suggestion why don't we host an echo service for APISIX itself?

I see some of the docs use this IP http://39.97.63.215/, I assume this IP belongs to the APISIX team, in that case, we can add an endpoint to echo the calls (+domain name), thus we can use it in all our docs. Without referring to external domains.

[membphis]

BTW, a suggestion why don't we host an echo service for APISIX itself?

#1225

https://docs.google.com/document/d/1BNMWjXB5_yFsuP8WcJFH-a2mF9PfaH8Md-wUXGFG4MY/edit#heading=h.ovyshqokq27

@Akayeshmantha is working on this, you can talk to him directly.

[membphis]

I see some of the docs use this IP http://39.97.63.215/

we should change it to another address. PR welcome

[sshniro]

Sure, will update!