Inconsistent requests SourceIP #17878
-
|
Hi, First of all thank you for minio 🙏. I'm trying to use Minio can't seem to get the right source ip reliably (see below) and thus, I can't use the Expected BehaviorMinio gets the right source Ip every time no matter the node of the cluster and the "order of first sight" of the clients. Current BehaviorRight now, the source IP is correct (seen from the audit logs and the trace) only for the first client ( If another client ( Restarting the minio node "resets" the behaviour and the first client to connect can "set" a new IP. This behavior works across users in the sense that the user loggued doesn't matter, only the reached minio node. Steps to Reproduce (for bugs)My setup isn't really straight forward so I'm not sure it can be replicated easily. Here are sample trace logs:
{
"http": {
"method": "GET",
"status_code": 200,
"content_type": "",
"host": "minio.console",
"referrer": "",
"x_forwarded_for": "<client-1 ip>",
"x_real_ip": "<client-1 ip>",
"url": "/api/v1/buckets/test-bucket",
"version": "HTTP/1.1",
"connection": "4422",
"connection_requests": "1"
},
"network": {
"bytes_written": 919,
"bytes_read": 2400,
"client": { "ip": "127.0.0.1", "port": 60440 },
"destination": { "ip": "127.0.0.1", "port": 9991 },
"nginx": {
"request_time": "0.088",
"upstream_connect_time": "0.001",
"upstream_response_time": "0.088",
"upstream_header_time": "0.088"
}
}
}
{
"http": {
"method": "GET",
"status_code": 200,
"host": "minio.console",
"referrer": "",
"x_forwarded_for": "<client-1 ip>",
"x_real_ip": "<client-1 ip>",
"url": "/api/v1/buckets/test-bucket",
"version": "HTTP/1.1",
"connection": "1864",
"connection_requests": "1"
},
"network": {
"bytes_written": 919,
"bytes_read": 2400,
"client": { "ip": "127.0.0.1", "port": 49764 },
"destination": { "ip": "127.0.0.1", "port": 9991 },
"nginx": {
"request_time": "0.114",
"upstream_connect_time": "0.013",
"upstream_response_time": "0.114",
"upstream_header_time": "0.114"
}
}
}Nginx configuration: user root root;
events {}
http {
log_format json_detailed escape=json
'{'
'"http":{'
'"method":"$request_method",'
'"request_id":"$request_id",'
'"status_code":$status,'
'"content_type":"$content_type",'
'"useragent":"$http_user_agent",'
'"referrer":"$http_referer",'
'"x_forwarded_for":"$http_x_forwarded_for",'
'"url":"$request_uri",'
'"version":"$server_protocol",'
'"connection":"$connection",'
'"connection_requests":"$connection_requests"'
'},'
'"network":{'
'"bytes_written":$bytes_sent,'
'"bytes_read":$request_length,'
'"client":{'
'"ip":"$remote_addr",'
'"port":$remote_port'
'},'
'"destination":{'
'"ip":"$server_addr",'
'"port":$server_port'
'},'
'"proxy_protocol":{'
'"proxy_protocol_addr":"$proxy_protocol_addr ",'
'"proxy_protocol_server_addr":$proxy_protocol_server_addr'
'},'
'"nginx":{'
'"request_time":"$request_time",'
'"upstream_connect_time":"$upstream_connect_time",'
'"upstream_response_time":"$upstream_response_time",'
'"upstream_header_time":"$upstream_header_time"'
'}'
'}'
'}';
access_log /dev/stdout json_detailed;
error_log /dev/stderr info;
server {
listen 9900 ssl proxy_protocol;
server_name {{ inventory_hostname }}.priv;
set_real_ip_from {{ scw__lb01_ip_addr }};
real_ip_header proxy_protocol;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/key.pem;
ssl_dhparam /etc/nginx/ssl/dhparams.pem;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve secp521r1:secp384r1;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_buffer_size 4k;
# Allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# Disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass https://minio-server:9000;
}
}
server {
listen 9901 ssl proxy_protocol;
server_name {{ inventory_hostname }}.priv;
set_real_ip_from {{ scw__lb01_ip_addr }};
real_ip_header proxy_protocol;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/key.pem;
ssl_dhparam /etc/nginx/ssl/dhparams.pem;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ecdh_curve secp521r1:secp384r1;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_buffer_size 4k;
# Allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# Disable buffering
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
# real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass https://minio-server:9001;
}
}
}ContextI have a minio cluster with 4 nodes that sits behind a public load balancer. There is one nginx server for each minio node. The cluster lives on virtual machines hosted in the cloud. RegressionNo Environment
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 2 replies
-
|
Do you have trace output such as |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your answer. 🙏 Yes looking at the verbose logs, the Request It's weird the I did some tests to rule out podman and nginx before opening this issue, but I might have missed something. |
Beta Was this translation helpful? Give feedback.
-
|
yeah so the primary problem we have is knowing the original SourceIp we need to be able to see it somehow, if we can't then it's impossible for MinIO to apply sourceIp based restrictions. IMO you shouldn't rely on these things that are extremely dynamic and have confusing end results. Instead, move to a more "tag" based or some form of namespace-based compartmentalization to apply the right policies. Don't think there is any issue here that we can see. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your answer 🙏. I've been doing more tests today and I think I might have narrowed it down a bit. It seems the problem only happens when I interact with minio through the console, when I talk to minio directly all IPs are correct AFAICT. Could it be possible that the console overwrites the original request headers ? |
Beta Was this translation helpful? Give feedback.
-
|
I think I have found a potential culprit in the console code here: https://github.com/dvaldivia/console/blob/master/restapi/client-admin.go#L539C38-L539C38 It seems there is a client adress cache that would match the behaviour I'm observing. This was introduced with https://github.com/minio/console/pull/2864. |
Beta Was this translation helpful? Give feedback.
-
|
I opened a separate issue in the console: https://github.com/minio/console/issues/3010 |
Beta Was this translation helpful? Give feedback.
yeah so the primary problem we have is knowing the original SourceIp we need to be able to see it somehow, if we can't then it's impossible for MinIO to apply sourceIp based restrictions.
IMO you shouldn't rely on these things that are extremely dynamic and have confusing end results. Instead, move to a more "tag" based or some form of namespace-based compartmentalization to apply the right policies.
Don't think there is any issue here that we can see.