bug: upstream proxy with long host name which was setup by k8s-ingress.

[yangtian9999]

Issue description

bug: upstream proxy with long host name which was setup by k8s-ingress.

Environment

I used docker whole in one to install apisix gateway and etcd.

$ make build-all-in-one
$ docker run -v pwd/all-in-one/apisix/config.yaml:/usr/local/apisix/conf/config.yaml -p 9080:9080 -p 9091:9091 -p 2379:2379 -d apache/apisix:whole

• apisix version (cmd: apisix version): 2.8

• OS (cmd: uname -a): Linux 845a20fd4211 5.8.0-63-generic doc: maybe a mistake in architecture-design-cn.md #71~20.04.1-Ubuntu SMP Thu Jul 15 17:46:08 UTC 2021 x86_64 Linux

• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
  nginx version: openresty/1.19.3.1
  built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
  built with OpenSSL 1.1.1k 25 Mar 2021
  TLS SNI support enabled

• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
  {"hostname":"845a20fd4211","id":"c3486945-9d97-4829-99ac-d9d6df6a9732","boot_time":1629105851,"etcd_version":"3.4.0","up_time":232296,"last_report_time":1629338105,"version":"2.8"}

• apisix-dashboard version, if have: not installed

• luarocks version, if the issue is about installation (cmd: luarocks --version):
  /usr/local/openresty/luajit/bin/luarocks 3.7.0
  LuaRocks main command-line interface

Steps to reproduce

The following scrips worked fine with ip address "10.3.4.105".
But after I used host name "xxxx.xxxxxx-xxxxx.xxxxxxx.com" which was setup by k8s-ingress and back-end service deployed by k8s, curl resulted with "404".
Usually, I add "server_names_hash_bucket_size 128" in nginx to resolve the same long host name, and this workes fine.

curl "http://127.0.0.1:9080/apisix/admin/upstreams/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"type": "roundrobin",
"nodes": {
"10.3.4.105:8088": 1
}
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/system/list",
"host": "10.3.4.105",
"upstream_id": "1"
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/system/list",
"plugins": {
"proxy-rewrite": {
"regex_uri": ["^/system/(.*)", "/system/$1"]
}
},
"upstream_id": "1"
}'

curl http://127.0.0.1:9080/system/list

Actual result

after I used host name "xxxx.xxxxxx-xxxxx.xxxxxxx.com" which was setup by k8s-ingress, curl resulted with "404".

Error log

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Date: Thu, 19 Aug 2021 02:24:20 GMT
Server: APISIX/2.8

404 page not found

Expected result

same api result for both ip and long hostname.

[tao12345666333]

Can you show your current routes list in APISIX?

[yangtian9999]

ingress address service is:

curl http://xxxx.xxxxxx-xxxxx.xxxxxxx.com/aaa (this works fine)

route list result:

{"action":"get","node":{"key":"/apisix/routes","dir":true,"nodes":[{"modifiedIndex":994,"createdIndex":16,"value":{"plugins":{"proxy-rewrite":{"regex_uri":["^/aaa","/aaa"]}},"create_time":1629283092,"uri":"/aaa","id":"1","priority":0,"status":1,"update_time":1629339836,"upstream_id":"1"},"key":"/apisix/routes/1"}]},"count":1}

[yangtian9999]

these are for /aaa config

curl "http://127.0.0.1:9080/apisix/admin/upstreams/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"type": "roundrobin",
"nodes": {
"xxxx.xxxxxx-xxxxx.xxxxxxx.com:80": 1
}
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"host": "xxxx.xxxxxx-xxxxx.xxxxxxx.com",
"upstream_id": "1"
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"plugins": {
"proxy-rewrite": {
"regex_uri": ["^/aaa", "/aaa"]
}
},
"upstream_id": "1"
}'

[tokers]

Your descriptions are confusing, not sure whether my understanding is correct.

So the accesses of /aaa and /system/list with host xxxx.xxxxxx-xxxxx.xxxxxxx.com are both failed?

Could you paste the route with the host xxxx.xxxxxx-xxxxx.xxxxxxx.com and show the request you sent.

[yangtian9999]

Hi, @tokers

Just ignore /system/list.

xxxx.xxxxxx-xxxxx.xxxxxxx.com only have /aaa interface.

curl "http://127.0.0.1:9080/apisix/admin/upstreams/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"type": "roundrobin",
"nodes": {
"xxxx.xxxxxx-xxxxx.xxxxxxx.com:80": 1
}
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"host": "xxxx.xxxxxx-xxxxx.xxxxxxx.com",
"upstream_id": "1"
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"plugins": {
"proxy-rewrite": {
"regex_uri": ["^/aaa", "/aaa"]
}
},
"upstream_id": "1"
}'

Finally, curl -i http://127.0.0.1:9080/aaa
, we got

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Date: Thu, 19 Aug 2021 02:24:20 GMT
Server: APISIX/2.8

[tokers]

Strange, are there any suspicious error logs?

[yangtian9999]

Hi, @tokers no error message when submit by apisix admin.

Any other message you want? And please share the command that can get some other info.

Thanks.

[tokers]

Finally, curl -i http://127.0.0.1:9080/aaa
, we got

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
Content-Length: 19
Connection: keep-alive
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Date: Thu, 19 Aug 2021 02:24:20 GMT
Server: APISIX/2.8

What does the response body like?

[yangtian9999]

this is the 404 from the apisix gateway.

this is the direct call from long host name, and the correct result.

[yangtian9999]

We add host in the route config.
Problem fixed. But we rarely found any doc page that specifically guided us to know where to put hostname.
But luckily, we found apisix can resolve the long hostname based on our environment.
Thanks for your help @tokers

curl "http://127.0.0.1:9080/apisix/admin/upstreams/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"type": "roundrobin",
"nodes": {
"xxxx.xxxxxx-xxxxx.xxxxxxx.com:80": 1
}
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"host": "xxxx.xxxxxx-xxxxx.xxxxxxx.com",
"upstream_id": "1"
}'

curl "http://127.0.0.1:9080/apisix/admin/routes/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
"uri": "/aaa",
"plugins": {
"proxy-rewrite": {
"regex_uri": ["^/aaa", "/aaa"],
"host": "xxxx.xxxxxx-xxxxx.xxxxxxx.com" -----------here is the missing one-----------------
}
},
"upstream_id": "1"
}'