API Key with IP address restriction block all IP including whitelisted

[volodymyrd]

I've created ESPv2 Cloud Endpoint for my API, everything works fine except when I enabling IP address restriction in API Key, I get a message "PERMISSION_DENIED: IP address blocked." for all requests including whitelisted IP addresses.

What you expected to happen:
Block all requests except whitelisted.

Steps to reproduce:
Create IP restriction for API Key for Endpoint base on ESPv2.

I've checked the endpoint logs and see the internal google IP address instead of my IP address from which I send a request, a corresponding screenshot is attached.

[nareddyt]

Thanks for reporting. We are aware of this issue and have a workaround, but I agree it's not the best solution.

Background: ESPv2 uses the standard X-Forwarded-For header to determine the client IP. This contains the client IP address and also a list of any proxies the request was forwarded through (Google Front End, Cloud Run's, etc). It's difficult for ESPv2 to know which index to use in the list; different platforms / deployments have a different number of proxies. ESPv2 doesn't blindly just take the first index because it can be spoofed.

We have a ESPv2 flag that tells ESP which index to use. By default, it uses the 2nd index from the end of the list. If you're seeing a Google internal IP, then you can increase the value of the --envoy_xff_num_trusted_hops ESPv2 flag. Maybe 3 or 4 would work to use an earlier index? I never tested this out.

Reference: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-forwarded-for

[qiwzhang]

If you have --enable_debug flag on for ESPv2, you can check its debug log, look at the request header "x-forwarded-for" to see which one is your client_ip. and adjust --envoy_xff_num_trusted_hops flag accordingly.

…

On Mon, Jun 15, 2020 at 9:30 AM Teju Nareddy ***@***.***> wrote: Thanks for reporting. We are aware of this issue and have a workaround, but it's not fixed. Background: ESPv2 uses the standard X-Forwarded-For header <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For> to determine the client IP. This contains the client IP address and also a list of any proxies the request was forwarded through (Google Front End, Cloud Run's, etc). It's difficult for ESPv2 to know which index to use in the list; different platforms / deployments have a different number of proxies. We have a ESPv2 flag that tells ESP which index to use. If you're seeing a Google internal IP, then you can increase the value of the --envoy_xff_num_trusted_hops ESPv2 flag <https://github.com/GoogleCloudPlatform/esp-v2/blob/2f523997feb29a95639db856865c9404c5d38730/docker/generic/start_proxy.py#L314>. Maybe 3 or 4 would work? I never tested this out. Reference: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-forwarded-for — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#72 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZ6TB7YULIMGWJX2PDADJLRWZEBZANCNFSM4N6KSRYQ> .

[volodymyrd]

Thanks, for me, works:
--envoy_xff_num_trusted_hops=1
--envoy_use_remote_address