Http proxy fix

[ashmeenkaur]

Description

GCSFuse was not respecting https_proxy and http_proxy environment variables. This was pointed out in bug #1541
Setting http.ProxyFromEnvironment in the underlying http client fixes the issue. (Ref: Stackoverflow)

Link to the issue in case of a bug fix.

#1541

Testing details

1. Manual - I confirmed through manual checks that requests are successfully flowing through the proxy server, regardless of whetherGCSFuse is running in foreground or background mode.
   Proxy server access logs:

   1703535566.729  20150 127.0.0.1 TCP_TUNNEL/200 5507 CONNECT storage.googleapis.com:443
   1703535566.729  20226 127.0.0.1 TCP_TUNNEL/200 6334 CONNECT oauth2.googleapis.com:443
   

   Also verified that requests were not routed via proxy server before this change.

2. Unit tests - Added

3. Integration tests - Passed successfully

4. Perf tests - No regression

[gargnitingoogle]

@ashmeenkaur please make sure that this goes into master by Jan 8, 2024 to go into the January 2024 release.

[ethan-crabb]

I'm trying to proxy requests through http://127.0.0.1:8082, however it still appears as though gcsfuse isn't respecting my http_proxy environment variables.

I'm using miymproxy as my proxy server, and starting both processes as follows:

./mitmdump -p 8082 --mode reverse:https://storage.googleapis.com &
mount -t gcsfuse -o allow_other,file_mode=777,dir_mode=777,key_file=/service-account-key.json bucket_name /tmp/ls_media

I have not fully configured the proxy server yet, so requests should return 502, however requests go through perfectly fine.

I've tested this with the LSR and the release linked to this PR, both produce the same result.

I've also tested uploading a file via gcloud storage cp. This produces the expected result and fails after a few 502 responses.

I'm using Debian, and installing gcsfuse from the .deb file using docker.

I set all 4 of these environment variables during the build:

ENV http_proxy http://127.0.0.1:8082
ENV HTTPS_PROXY http://127.0.0.1:8082
ENV https_proxy http://127.0.0.1:8082
ENV HTTP_PROXY http://127.0.0.1:8082

Please let me know if I'm missing something. Many thanks.

[ashmeenkaur]

Re: #1562 (comment)

Hi @ethan-crabb,

Seems like mount -t gcsfuse -o allow_other,file_mode=777,dir_mode=777,key_file=/service-account-key.json bucket_name /tmp/ls_media command is not respecting the environment variable you are setting.

Can you try setting the environment variables inline like the following?

http_proxy=http://127.0.0.1:8082 https_proxy=http://127.0.0.1:8082 mount -t gcsfuse -o allow_other,file_mode=777,dir_mode=777,key_file=/service-account-key.json bucket_name /tmp/ls_media

On the console logs, you will see the following when environment variable is set properly:

.
.
Added environment https_proxy: http://127.0.0.1:8082
{"time":"26/03/2024 12:22:44.106906","severity":"INFO","message":"File system has been successfully mounted."}

Let me know if it works! If not, we can open a GitHub issue to investigate further.

PS: I tried this on latest GCSFuse version (2.0.0).

Thanks,
Ashmeen