404 for some sizes at particular time - Docker

[amitjoseph]

I have observed this strange problem, some sizes I'm getting a 404 for some sizes at a particular time.
I'm using the default configuration. I'm not able point out where actually this problem is occurring.

Error Message:

{"status":"error","code":404,"message":"The hostname of the origin is unresolvable (DNS) or blocked by policy."}

Suppose http://localhost:81/?url=https://images.unsplash.com/photo-1554204428-01ab5f412a28&w=750
gives me this error but http://localhost:81/?url=https://images.unsplash.com/photo-1554204428-01ab5f412a28&w=700 works fine
This is what I don't understand.

Docker log:

172.17.0.1 - - [18/Dec/2019:07:35:54 +0100] "GET /?url=https://images.unsplash.com/photo-1576379676103-b31985f96ae2&w=5500 HTTP/1.1" 404 132 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" 1576650954.447

172.17.0.1 - - [18/Dec/2019:07:36:02 +0100] "GET /?url=https://images.unsplash.com/photo-1576379676103-b31985f96ae2&w=5500/ HTTP/1.1" 200 3547059 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" 1576650962.910

172.17.0.1 - - [18/Dec/2019:08:03:35 +0100] "GET /?url=https://images.unsplash.com/photo-1554204428-01ab5f412a28&w=888 HTTP/1.1" 200 163246 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" 1576652615.460

172.17.0.1 - - [18/Dec/2019:08:03:46 +0100] "GET /?url=https://images.unsplash.com/photo-1554204428-01ab5f412a28&w=850 HTTP/1.1" 200 151278 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" 1576652626.628

172.17.0.1 - - [18/Dec/2019:08:03:52 +0100] "GET /?url=https://images.unsplash.com/photo-1554204428-01ab5f412a28&w=750 HTTP/1.1" 404 132 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0" 1576652632.757

[kleisauke]

If you see such errors:

YYYY/MM/DD HH:MM:SS [LEVEL] PID#TID: *CID connect() to [2001:db8:85a3::8a2e:370:7334]:443 failed (99: Cannot assign requested address) while connecting to upstream

Within the weserv module error log (/var/log/nginx/weserv-local-error.log), then it's usually a misconfiguration of the IPv6 network interface. You can use these commands to view the error logs in Docker:

# Global nginx / libvips errors
docker exec imagesweserv tail /var/log/nginx/error.log

# weserv module errors
docker exec imagesweserv tail /var/log/nginx/weserv-local-error.log

# rate limiting errors
docker exec imagesweserv tail /var/log/nginx/weserv-error.log

If looking up of IPv6 addresses is not desired, the ipv6=off parameter can be specified within the nginx config:

--- a/ngx_conf/imagesweserv.conf
+++ b/ngx_conf/imagesweserv.conf
@@ -54,7 +54,7 @@ server {
     weserv on;
 
     location / {
-        resolver 8.8.8.8; # Use Google's open DNS server
+        resolver 8.8.8.8 ipv6=off; # Use Google's open DNS server
         weserv_mode proxy; # Default
     }
 

[kleisauke]

@amitjoseph Were you able to make any progress with this?

[andrieslouw]

@amitjoseph I'm also curious if you discovered the root cause, or need any help debugging this issue, please let us know by re-opening the issue if so!

[RichardSieg]

We had the same issue and your suggestion @kleisauke fixed it for us! Thanks!

[lincolnthree]

@andrieslouw We just tried this fix but are still getting the IP assignment errors -- seems like the ipv6=off setting is not being used:

2021/04/27 22:20:53 [crit] 35#0: *152 connect() to [XXXX:XXXX:XX::XXXX:XXX]:443 failed (99: Cannot assign requested address) while connecting to upstream, client: 127.0.0.1, server: images.weserv.local, request: "GET /?url=https%3A%2F%2Fimga.exam.com%2Flarge2Fb4d54342.jpg&fit=cover&dpr=1 HTTP/1.1", upstream: "https://[XXXX:XXXX:XX::XXXX:XXX]:443", host: "images.weserv.local", referrer: "http://localhost:8100/"

imagesweserv.conf modifications look like this:

proxy_cache_path /dev/shm/proxy_cache inactive=8h levels=1:2 keys_zone=images:512m max_size=250m loader_files=5000 use_temp_path=off;
...
        resolver 8.8.8.8 ipv6=off; # Use Google's open DNS server

Is there something else we can adjust or dig into to find the source of these errors?

It looks like the IPV6 address upstream that fails is a cloudfair address.

[lincolnthree]

Is this possibly related? https://trac.nginx.org/nginx/ticket/723

[andrieslouw]

If the bug is in NGINX's proxy_pass: A really ugly hack would be to remove the IPv6 addr corresponding to the .local prefix from the /etc/hosts, or using a different hostname (instead of images.weserv.local); one which has no AAAA-records at all.

If the bug is in resolving and connecting to IPv6-origins, a ugly solution would be to use a DNS-resolver that doesn't provide AAAA-records, or strips them from all responses.

But I'm more concerned by the lack of IPv6-support in your setup, is there no other way to fix that? We see more and more hosts having IPv6-only support, and/or really broken and CGNAT-ed IPv4. IPv6 is the way of the future, and the preferred way of connecting networks nowadays. The only right way to solve this is to get IPv6 working in your stack 😉

[lincolnthree]

@andrieslouw The weird thing is it works some of the time. I'll look into using a different hostname and report back. AFAIK IPV6 should be enabled in the stack.

docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXX.XX.XX.XXX  netmask 255.255.0.0  broadcast XXX.XX.XXX.XXX
        inet6 XXXX::XXX:XXXX:XXXX:XXXX  prefixlen 64  scopeid 0x20<link>
        ether XX:XX:XX:XX:XX:XX  txqueuelen 0  (Ethernet)
        RX packets 85448  bytes 98877536 (98.8 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 87741  bytes 234467855 (234.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXX.XX.XX.XXX  netmask 255.255.240.0  broadcast XXX.XX.XX.XXX
        inet6 XXXX::XXX:XXXX:XXXX:XXXX  prefixlen 64  scopeid 0x20<link>
        ether XX:XX:XX:XX:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 535429  bytes 725402041 (725.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 81379  bytes 11382751 (11.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

#and of course:

$ ping6 2001:4860:4860::8888
PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=118 time=2.05 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=2 ttl=118 time=1.39 ms

[lincolnthree]

@andrieslouw I'm not a networking expert, but it looks like IPV6 is enabled. There does not appear to be a .local-prefixed host defined:

127.0.1.1 xxxxxxxxxx xxxxxxxxxxx
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

[lincolnthree]

Oh no. This could be the issue: https://docs.digitalocean.com/products/networking/ipv6/#:~:text=Load%20balancers%2C%20Floating%20IPs%2C%20and,to%20(like%20a%20Droplet).

Not supported for load balancers. Will test connecting directly to the server.

[lincolnthree]

Update, load balancer should not be the issue.

@andrieslouw Interestingly enough. I can ping6 addresses from my host, but not from inside the imagesweserv docker container:

[root@f25f05c71967 imagesweserv]# ping6 XXXX:XXXX:XX::XXXX:XXX
connect: Cannot assign requested address
[root@f25f05c71967 imagesweserv]# 

[lincolnthree]

@andrieslouw OK. That was the issue. Docker networking itself was not configured for IPV6. (Seems this is not the default for docker for some reason? Possibly a botched install?)

Either way --- For the time being I have run the docker container using --network host option to use the host network, since these servers are single-purpose and there will not be any overlap between containers, but I will likely experiment with figuring out how to get this configured anyway, and post here if I find the right docker config.

[andrieslouw]

Please let us know if there is something specific missing in our proposed docker config. Glad you got IPv6 working, hope we can help other users with your findings. Thank you for taking us with you while debugging this, it's really helpful to get some insight!

@kleisauke: Could you take a look into the IPv6 stack that Docker is using by default? Is something missing at our side?

[lincolnthree]

@andrieslouw I will do my best. Thanks for your assistance with this. I likely would not have figured out IPV6 was not working for a LONG time.

I do wonder if it has to do with the order that the system configuration happens in. E.g. if IPV6 is configured after docker packages are installed, I'm wondering if it needs to be re-packaged/installed to pick that up. I need to do more digging to see if docker is even supposed to work with IPV6 out of the box.

[kleisauke]

@kleisauke: Could you take a look into the IPv6 stack that Docker is using by default? Is something missing at our side?

For anyone experiencing this issue with Docker, I recommend running this within the container:

$ cat /etc/resolv.conf | grep nameserver

If the output displays nameserver 127.0.0.11, it indicates that Docker is using an embedded DNS server, which in turn forwards external DNS queries to the DNS servers configured on the host. In such cases, you should modify the nginx configuration to use Docker's embedded DNS server by doing:

--- a/etc/nginx/imagesweserv.conf
+++ b/etc/nginx/imagesweserv.conf
@@ -68,7 +68,7 @@ server {
     deny all;
 
     location / {
-        resolver 8.8.8.8; # Use Google's open DNS server
+        resolver 127.0.0.11; # Use Docker's embedded DNS server
         weserv proxy;
 
         add_header X-Upstream-Response-Length $weserv_response_length;

(see also: https://stackoverflow.com/a/37656784)