fix: Improve handling of unknown hosts and missing certs #2186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for improved readability.
for improved readability.
Before, if neither the vhost-specific cert nor `default.crt` existed, nginx-proxy would not create the https vhost. This resulted in nginx either refusing the connection or serving the wrong vhost depending on whether there was another https vhost with a certificate. Now nginx-proxy always creates an https server for a vhost, even if the vhost-specific certificate and the default certificate are both missing. When both certs are missing, nginx is given empty certificate data to make it possible for it to start up without an error. The empty certificate data causes the user to see a TLS error, which is much easier to troubleshoot than a connection refused error or serving the wrong vhost.
Before, a fallback http server was created to handle requests for unknown virtual hosts even when `HTTPS_METHOD=nohttp`. (In this case, all http vhosts would be unknown.) Likewise, a catch-all fallback https server was still created even if `HTTPS_METHOD=nohttps`. Now the fallback servers are created only if needed. This brings the behavior in line with the documentation and user expectation. It will also make it easier to implement a planned feature: different servers on different ports.
buchdag
requested changes
Mar 14, 2023
buchdag
added
status/pr-needs-docs
rhansen
force-pushed
the
fallback
branch
2 times, most recently
from
74d565d
to
4d7632c
Compare
rhansen
removed
the
status/pr-needs-docs
buchdag
requested changes
Mar 14, 2023
buchdag
approved these changes
Mar 24, 2023
|
Thank you for rebasing and merging, @buchdag! |
|
@rhansen I am testing this with And SSLLabs reports: Session resumption (caching): No (IDs assigned but not accepted) Or testing with openssl directly: I commented line 415 in the nginx.tmpl and that fixed it for me. |
|
@SchoNie line 415 is a listener on the HTTP port, I don't get how it could influence something TLS related 🤯 Could you show us the rendered nginx configuration that work and the one that doesn't (you can get it by executing |
|
Ofcourse, sorry for not doing that right away. I created a small whoami (my prod setup is huge) setup and was able to reproduce. Let me know if any more is needed. Failing nginx.conf:# nginx-proxy version : 1.3.0-17-g7dd32e3
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
# test_default
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $scheme;
'' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
default $http_host;
'' $http_host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $server_port;
'' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
default upgrade;
'' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
# Preserve nginx's default behavior (send "Connection: close").
default close;
# Use an empty string to cancel nginx's default behavior.
true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined. This is necessary because:
# - The $proxy_connection variable is indirectly derived from
# $upstream_keepalive, so $upstream_keepalive must be defined whenever
# $proxy_connection is resolved.
# - The $proxy_connection variable is used in a proxy_set_header directive in
# the http block, so it is always fully resolved for every request -- even
# those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
# The value here should not matter because it should always be overridden in
# a location block (see the "location" template) for all requests where the
# value actually matters.
default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
default off;
https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr" "$request_time" $ssl_protocol/$ssl_cipher';
access_log off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
server_tokens off;
listen 80;
listen 443 ssl http2;
access_log /var/log/nginx/access.log vhost;
# No default.crt certificate found for this vhost, so force nginx to emit a
# TLS error if the client connects via https.
ssl_ciphers aNULL;
set $empty "";
ssl_certificate data:$empty;
ssl_certificate_key data:$empty;
if ($https) {
return 444;
}
return 503;
}
# test.domain.com/
upstream test.domain.com-42099b4af021e53fd8fd4e056c2568d7c2e3ffa8 {
# Container: test-whoami-1
# networks:
# test_default (reachable)
# IP address: 172.22.0.2
# exposed ports: 80/tcp
# default port: 80
# using port: 8000
server 172.22.0.2:8000;
# Container: test-whoami-1
# networks:
# test_default (reachable)
# IP address: 172.22.0.2
# exposed ports: 80/tcp
# default port: 80
# using port: 8000
server 172.22.0.2:8000;
keepalive 32;
}
server {
server_name test.domain.com;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
# Do not HTTPS redirect Let's Encrypt ACME challenge
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
server_name test.domain.com;
access_log /var/log/nginx/access.log vhost;
listen 443 ssl http2 ;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/domain.com.crt;
ssl_certificate_key /etc/nginx/certs/domain.com.key;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/certs/domain.com.chain.pem;
set $sts_header "";
if ($https) {
set $sts_header "max-age=31536000; includeSubDomains; preload";
}
add_header Strict-Transport-Security $sts_header always;
location / {
proxy_pass http://test.domain.com-42099b4af021e53fd8fd4e056c2568d7c2e3ffa8;
set $upstream_keepalive true;
}
}Then I edit the first listener, and do a Working nginx.conf:# nginx-proxy version : 1.3.0-17-g7dd32e3
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
# test_default
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $scheme;
'' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
default $http_host;
'' $http_host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $server_port;
'' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
default upgrade;
'' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
# Preserve nginx's default behavior (send "Connection: close").
default close;
# Use an empty string to cancel nginx's default behavior.
true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined. This is necessary because:
# - The $proxy_connection variable is indirectly derived from
# $upstream_keepalive, so $upstream_keepalive must be defined whenever
# $proxy_connection is resolved.
# - The $proxy_connection variable is used in a proxy_set_header directive in
# the http block, so it is always fully resolved for every request -- even
# those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
# The value here should not matter because it should always be overridden in
# a location block (see the "location" template) for all requests where the
# value actually matters.
default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
default off;
https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr" "$request_time" $ssl_protocol/$ssl_cipher';
access_log off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
server_tokens off;
listen 80;
# listen 443 ssl http2; <<this line is commented
access_log /var/log/nginx/access.log vhost;
# No default.crt certificate found for this vhost, so force nginx to emit a
# TLS error if the client connects via https.
ssl_ciphers aNULL;
set $empty "";
ssl_certificate data:$empty;
ssl_certificate_key data:$empty;
if ($https) {
return 444;
}
return 503;
}
# test.domain.com/
upstream test.domain.com-42099b4af021e53fd8fd4e056c2568d7c2e3ffa8 {
# Container: test-whoami-1
# networks:
# test_default (reachable)
# IP address: 172.22.0.2
# exposed ports: 80/tcp
# default port: 80
# using port: 8000
server 172.22.0.2:8000;
# Container: test-whoami-1
# networks:
# test_default (reachable)
# IP address: 172.22.0.2
# exposed ports: 80/tcp
# default port: 80
# using port: 8000
server 172.22.0.2:8000;
keepalive 32;
}
server {
server_name test.domain.com;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
# Do not HTTPS redirect Let's Encrypt ACME challenge
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
server_name test.domain.com;
access_log /var/log/nginx/access.log vhost;
listen 443 ssl http2 ;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/domain.com.crt;
ssl_certificate_key /etc/nginx/certs/domain.com.key;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/certs/domain.com.chain.pem;
set $sts_header "";
if ($https) {
set $sts_header "max-age=31536000; includeSubDomains; preload";
}
add_header Strict-Transport-Security $sts_header always;
location / {
proxy_pass http://test.domain.com-42099b4af021e53fd8fd4e056c2568d7c2e3ffa8;
set $upstream_keepalive true;
}
}The docker-compose.yml I use for testing:version: '3.9'
services:
nginx-proxy:
image: nginxproxy/nginx-proxy:alpine
container_name: nginx-proxy
restart: unless-stopped
environment:
- TZ=Europe/Amsterdam
- SSL_POLICY=Mozilla-Intermediate
- HSTS=max-age=31536000; includeSubDomains; preload
- TRUST_DOWNSTREAM_PROXY=false
- LOG_FORMAT=$$host $$remote_addr - $$remote_user [$$time_local] "$$request" $$status $$body_bytes_sent "$$http_referer" "$$http_user_agent" "$$upstream_addr" "$$request_time" $$ssl_protocol/$$ssl_cipher
ports:
- 80:80
- 443:443
volumes:
- "/var/run/docker.sock:/tmp/docker.sock:ro"
- "./nginx/conf.d/:/etc/nginx/conf.d/"
- "./nginx/vhost.d/:/etc/nginx/vhost.d/"
- "./nginx/certs/:/etc/nginx/certs/"
- "./accesslogs/:/var/log/nginx/"
- "./templates/nginx_custom.tmpl:/app/nginx.tmpl:ro"
whoami:
image: traefik/whoami:latest
restart: unless-stopped
environment:
- WHOAMI_PORT_NUMBER=8000
- VIRTUAL_HOST=test.domain.com
- VIRTUAL_PORT=8000
- VIRTUAL_PATH=/
labels:
com.github.nginx-proxy.nginx-proxy.keepalive: "32"I think the issue is Name-based HTTPS servers |
|
@SchoNie you are right about the cause but that should not happen if the tests correctly handle SNI, I'm beginning to suspect that those tests don't and aren't reflecting the reality. Could you try again with: openssl s_client -connect test.domain.com:443 -servername test.domain.com -reconnect -CAfile cacert.pemIf you don't use edit: or not ... did some additional test on my end, even with SNI enabled test the renegotiation seems to fail. |
This was referenced May 10, 2023
Closed
Closed
Closed
|
After using this feature, got below error. in my case, the /etc/nginx/certs/default.key & default.crt is missing, and the generated conf like this: |
|
@terryzwt what feature are you talking about ? (this PR was more about fixes and refactor) |
|
@buchdag I mean this fixed cause error message like below, it should be easy to reproduce. |
|
@terryzwt could you open an issue and follow the repo's issue template ? |
This was referenced Dec 19, 2023
|
From what I understand in this test, with this fix it is no longer possible with a self-signed certificate to access a site with an unknown domain name. So, if ever we have an architecture that needs this, it will no longer work. Two things, it seems to me that consequently
I will open an issue for this, as it seems I have the same error as @terryzwt . |
|
Hi. Provided I understood correctly the issue you're facing: accessing a service with an unknow domain name and without additional config was unfortunately never an advertised nor supported feature, it only worked as a side effect of the proxy incorrectly redirecting request for unknown hostname to a proxied service, which was a real issue for a long time. I believe your use case (again if I understood it correctly) can still work by using |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
plus a couple of cleanups