/grpc

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC doesn't respect the no_proxy environment variable #9989

Closed
shabzy opened this Issue Mar 6, 2017 · 11 comments

Comments

Assignees

@yashykt yashykt

Labels
None yet
Projects
None yet
Milestone
No milestone
@shabzy

shabzy commented Mar 6, 2017

Issue found using the helloworld greeter server/client example, but should apply to everyone. Using git clone from today.

If on a linux machine, the environment has the usual "http_proxy" environment variable configured, gRPC will take that into account when trying to connect, however, will then proceed to ignore the companion no_proxy setting:

For example:
$ env
http_proxy=http://106.1.216.121:8080
no_proxy=localhost,127.0.0.1

$ ./greeter_client
D0306 16:00:11.419586349 1897 combiner.c:351] C:0x25a9290 finish old_state=3
D0306 16:00:11.420527744 1896 tcp_client_posix.c:179] CLIENT_CONNECT: ipv4:106.1.216.121:8080: on_writable: error="No Error"
D0306 16:00:11.420567382 1896 combiner.c:145] C:0x25a69a0 create
D0306 16:00:11.420581887 1896 tcp_client_posix.c:119] CLIENT_CONNECT: ipv4:106.1.216.121:8080: on_alarm: error="Cancelled"
I0306 16:00:11.420617663 1896 http_connect_handshaker.c:319] Connecting to server 127.0.0.1:50051 via HTTP proxy ipv4:106.1.216.121:8080

Basically, it's using the http_proxy url to connect even though localhost is in the no_proxy list. Since the default for no_proxy includes localhost on most linux machines; the end result is that any user with an http_proxy configured will never be able to connect to localhost.

@ejona86 ejona86 assigned markdroth Mar 6, 2017

@grahamc

This comment has been minimized.

Show comment
Hide comment
@grahamc

grahamc Apr 11, 2017

Using the ruby client (1.2.2, ruby 2.3.0) the error output is a much more sparse than the C++ logging, making it difficult to identify the issues is with a proxy:

[root]# /opt/ruby/2.3/bin/bundle exec ./greeter_client.rb
bundler: failed to load command: ./greeter_client.rb (./greeter_client.rb)
GRPC::Unavailable: 14:Connect Failed
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:46:in `check_status'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:178:in `attach_status_results_and_complete_call'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:340:in `request_response'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:185:in `block (3 levels) in rpc_stub_class'
  /root/grpc/examples/ruby/greeter_client.rb:46:in `main'
/root/grpc/examples/ruby/greeter_client.rb:50:in `<top (required)>'

Adding the environment variables GRPC_VERBOSITY=DEBUG GRPC_TRACE=all produced more information, which was very helpful in debugging:

[
  {
    "created": "@1491939664.942593641",
    "description": "HTTP proxy returned response code 502",
    "file": "src/core/ext/client_channel/http_connect_handshaker.c",
    "file_line": 229
  }
]

In python, similar case:

[root]# python3 greeter_client.py
Traceback (most recent call last):
  File "greeter_client.py", line 48, in <module>
    run()
  File "greeter_client.py", line 43, in run
    response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
  File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>```

grahamc commented Apr 11, 2017 •
edited

Using the ruby client (1.2.2, ruby 2.3.0) the error output is a much more sparse than the C++ logging, making it difficult to identify the issues is with a proxy:

[root]# /opt/ruby/2.3/bin/bundle exec ./greeter_client.rb
bundler: failed to load command: ./greeter_client.rb (./greeter_client.rb)
GRPC::Unavailable: 14:Connect Failed
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:46:in `check_status'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:178:in `attach_status_results_and_complete_call'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:340:in `request_response'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:167:in `request_response'
  /opt/ruby/2.3/lib/ruby/gems/2.3.0/gems/grpc-1.2.2-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:185:in `block (3 levels) in rpc_stub_class'
  /root/grpc/examples/ruby/greeter_client.rb:46:in `main'
/root/grpc/examples/ruby/greeter_client.rb:50:in `<top (required)>'

Adding the environment variables GRPC_VERBOSITY=DEBUG GRPC_TRACE=all produced more information, which was very helpful in debugging:

[
  {
    "created": "@1491939664.942593641",
    "description": "HTTP proxy returned response code 502",
    "file": "src/core/ext/client_channel/http_connect_handshaker.c",
    "file_line": 229
  }
]

In python, similar case:

[root]# python3 greeter_client.py
Traceback (most recent call last):
  File "greeter_client.py", line 48, in <module>
    run()
  File "greeter_client.py", line 43, in run
    response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
  File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.5/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, Connect Failed)>```
@ejona86

This comment has been minimized.

Show comment
Hide comment
@ejona86

ejona86 Apr 11, 2017

Member

It's unclear if "connect" in "connect failed" is HTTP CONNECT. If the status was more expressive, say "Proxy failed to connect to host" or some other details, this would have been more easily debuggable. I expect this would be equally hard to debug in C#.
Member

ejona86 commented Apr 11, 2017

It's unclear if "connect" in "connect failed" is HTTP CONNECT. If the status was more expressive, say "Proxy failed to connect to host" or some other details, this would have been more easily debuggable. I expect this would be equally hard to debug in C#.
@Retik

This comment has been minimized.

Show comment
Hide comment
@Retik

Retik Apr 19, 2017

I just encountered this same problem using gRPC v1.2.2 C#. I had HTTP_PROXY and HTTPS_PROXY set on my windows system and the greeter example in C# was failing to connect. Removing the env variables fixed the issue, but that's not a viable solution.

I tried adding a NO_PROXY env variable but it had no effect. What's further strange is that the client doesn't fail to connect, the connect itself succeeds. However, the first call to an API endpoint fails with StatusCode=Unavailable and Detail="Endpoint read failed".

Another interesting data point, this behavior does not happen using gRPC v1.0.1. This may because gRPC did not support proxy env variables in that version, hard to say for sure.

Retik commented Apr 19, 2017 •
edited

I just encountered this same problem using gRPC v1.2.2 C#. I had HTTP_PROXY and HTTPS_PROXY set on my windows system and the greeter example in C# was failing to connect. Removing the env variables fixed the issue, but that's not a viable solution.

I tried adding a NO_PROXY env variable but it had no effect. What's further strange is that the client doesn't fail to connect, the connect itself succeeds. However, the first call to an API endpoint fails with StatusCode=Unavailable and Detail="Endpoint read failed".

Another interesting data point, this behavior does not happen using gRPC v1.0.1. This may because gRPC did not support proxy env variables in that version, hard to say for sure.
@tundratim

This comment has been minimized.

Show comment
Hide comment
@tundratim

tundratim Jul 10, 2017

Confirmed that this is still a problem today.

tundratim commented Jul 10, 2017

Confirmed that this is still a problem today.

bsyk added a commit to bsyk/grpc that referenced this issue Jul 11, 2017

@bsyk
Check for no_proxy environment variable and skip proxy if host is listed 
This will fix #9989, #11603, #11751.
no_proxy is typically comma separated list of hosts and or domains.  Matching must be from the end of the uri to accomodate whole domain and subdomain whitelisting.
e9f929f

bsyk added a commit to bsyk/grpc that referenced this issue Jul 11, 2017

@bsyk
Check for no_proxy environment variable and skip proxy if host is listed 
This will fix #9989, #11603, #11751.
no_proxy is typically comma separated list of hosts and or domains. Matching must be from the end of the uri to accomodate whole domain and subdomain whitelisting.
a3dac0e

@bsyk bsyk referenced this issue Jul 11, 2017

Merged

Check for no_proxy environment variable and skip proxy if host is listed #11755

bsyk added a commit to bsyk/grpc that referenced this issue Jul 13, 2017

@bsyk
Check for no_proxy environment variable and skip proxy if host is listed 
This will fix #9989, #11603, #11751.
no_proxy is typically comma separated list of hosts and or domains. Matching must be from the end of the uri to accomodate whole domain and subdomain whitelisting.
83f83fc

bsyk added a commit to bsyk/grpc that referenced this issue Jul 17, 2017

@bsyk
Check for no_proxy environment variable and skip proxy if host is listed 
This will fix #9989, #11603, #11751.
no_proxy is typically comma separated list of hosts and or domains. Matching must be from the end of the uri to accomodate whole domain and subdomain whitelisting.

Split strings and compare
Address review comment.  Rather than iterate through the string, split it first then compare each entry.
This will work for well-formed no_proxy strings, but may fail if there are additional spaces.

Break early, cleanup newlines
799af62

bsyk added a commit to bsyk/grpc that referenced this issue Jul 19, 2017

@bsyk
Check for no_proxy environment variable and skip proxy if host is listed 
This will fix #9989, #11603, #11751.
no_proxy is typically comma separated list of hosts and or domains. Matching must be from the end of the uri to accomodate whole domain and subdomain whitelisting.

This will work for well-formed no_proxy strings, but may fail if there are additional spaces.
02d426e

@markdroth markdroth closed this in #11755 Jul 19, 2017

@stefanofiorentino

This comment has been minimized.

Show comment
Hide comment
@stefanofiorentino

stefanofiorentino Jul 24, 2017

in case you still experience the issue please run your program with setting to empty string the http_proxy string:
HTTP_PROXY= http_proxy= ./greeter_client

stefanofiorentino commented Jul 24, 2017

in case you still experience the issue please run your program with setting to empty string the http_proxy string:
HTTP_PROXY= http_proxy= ./greeter_client
@bsyk

This comment has been minimized.

Show comment
Hide comment
@bsyk

bsyk Jul 24, 2017

Contributor

@stefanofiorentino do you still have this issue with the latest master branch code? Do you have localhost in your no_proxy env var?
Contributor

bsyk commented Jul 24, 2017

@stefanofiorentino do you still have this issue with the latest master branch code? Do you have localhost in your no_proxy env var?

@GenuineJyn GenuineJyn referenced this issue Aug 10, 2017

Closed

"Greeter recerived : RPC failed" in cpp example #6862

@RafaARV

This comment has been minimized.

Show comment
Hide comment
@RafaARV

RafaARV Aug 24, 2017

Not fixed yet:

Windows 10 x64
grpcio 1.4.0
python 3.6.2
http_proxy=http://some_proxy:111
https_proxy=https://other_proxy:111
no_proxy=localhost,127.0.0.1

Same error above.
Thanks!.

RafaARV commented Aug 24, 2017

Not fixed yet:

Windows 10 x64
grpcio 1.4.0
python 3.6.2
http_proxy=http://some_proxy:111
https_proxy=https://other_proxy:111
no_proxy=localhost,127.0.0.1

Same error above.
Thanks!.
@markdroth

This comment has been minimized.

Show comment
Hide comment
@markdroth

markdroth Aug 24, 2017

Contributor

The change hasn't made its way into a release yet. It will be in the upcoming 1.6 release.
Contributor

markdroth commented Aug 24, 2017

The change hasn't made its way into a release yet. It will be in the upcoming 1.6 release.

@fhoek fhoek referenced this issue Oct 18, 2017

Closed

using grpc with a proxy server #12815

@mrfortynine mrfortynine referenced this issue Jan 5, 2018

Open

Get UnavailableError when running object detection training on CloudML #3071

@shenqixiaojiang shenqixiaojiang referenced this issue May 27, 2018

Open

Worker waiting for ps server #2

@dhrsharma

This comment has been minimized.

Show comment
Hide comment
@dhrsharma

dhrsharma Jul 26, 2018

no_proxy variable is still being ignored in version v1.13.0.

dhrsharma commented Jul 26, 2018

no_proxy variable is still being ignored in version v1.13.0.

@markdroth markdroth assigned yashykt and unassigned markdroth Jul 26, 2018

@bsyk

This comment has been minimized.

Show comment
Hide comment
@bsyk

bsyk Jul 26, 2018

Contributor

@dhrsharma I just checked with v1.13.0, and it seems to be working. Can you share your no_proxy, http_proxy environment variables and the service url you're trying to use?
Contributor

bsyk commented Jul 26, 2018

@dhrsharma I just checked with v1.13.0, and it seems to be working. Can you share your no_proxy, http_proxy environment variables and the service url you're trying to use?
@dhrsharma

This comment has been minimized.

Show comment
Hide comment
@dhrsharma

dhrsharma Jul 29, 2018

@bsyk my mistake. Yes, it works with v1.13.0. Actually, I was not creating a new client after changing the no_proxy at runtime. Sorry for the trouble.

dhrsharma commented Jul 29, 2018

@bsyk my mistake. Yes, it works with v1.13.0. Actually, I was not creating a new client after changing the no_proxy at runtime. Sorry for the trouble.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment