Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Net::HTTP can't ignore http.proxyHost of JVM System Property #2983

Closed
yousuketto opened this Issue May 25, 2015 · 12 comments

Comments

Projects
None yet
2 participants
@yousuketto
Copy link
Contributor

yousuketto commented May 25, 2015

JRuby ignores http.nonProxyHosts of JVM System Property.
Therefore I think that there are no way to access without http proxy, when there are http.proxyHost.

my env

  • I set http proxy in mac system settings
    • this probably set http.proxyHost, http.proxyPort and http.nonProxyHosts.
$ jruby -e "puts ENV_JAVA.values_at *%w(http.proxyHost http.proxyPort http.nonProxyHosts)"
xxx.example.com
8080
localhost|*.localhost
  • oracle jdk8
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
  • jruby 1.7.20

example code & exec result

#example.rb
require 'net/http'
http = Net::HTTP.new("localhost", 9292, nil, nil)
http.set_debug_output($stderr)
res = http.get("/")
p res.body
$ jruby example.rb 
opening connection to xxx.example.com...
opened
<- "GET http://localhost:9292/ HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: localhost:9292\r\n\r\n"
Conn close because of error Timeout::Error
Timeout::Error: Timeout::Error
          rbuf_fill at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/protocol.rb:146
          readuntil at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/protocol.rb:122
           readline at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/protocol.rb:132
   read_status_line at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:2571
           read_new at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:2560
  transport_request at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1328
              catch at org/jruby/RubyKernel.java:1274
  transport_request at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1325
            request at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1302
            request at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1295
              start at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:746
            request at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1293
                get at /Users/user/.rvm/rubies/jruby-1.7.20/lib/ruby/1.9/net/http.rb:1035
             (root) at example.rb:5
@headius

This comment has been minimized.

Copy link
Member

headius commented May 26, 2015

I thought we had added logic to net/http to honor this. Perhaps it was lost?

@yousuketto

This comment has been minimized.

Copy link
Contributor Author

yousuketto commented May 27, 2015

Are you taking about this? #195 or d04857c#diff-49

I confirmed that Net::HTTP can ignore http.proxyHost at 2.0 mode.
However, I hope that Net::HTTP change operation in response to http.nonProxyHosts, because I think that http.proxyHost, http.proxyPort, and http.nonProxyHosts are one set.
I have no idea...

@headius

This comment has been minimized.

Copy link
Member

headius commented May 27, 2015

I do see the same logic (using ENV_JAVA['http.proxyHost'] etc) in 2.0 mode's net/http. How are you initiating an HTTP connection?

...
    #
    # In JRuby, this will default to the JSE proxy settings provided in the
    # 'http.proxyHost' and 'http.proxyPort' Java system properties, if they
    # are set and no alternative proxy has been provided.
    #
    def HTTP.new(address, port = nil, p_addr = ENV_JAVA['http.proxyHost'], p_port = ENV_JAVA['http.proxyPort'], p_user = nil, p_pass = nil)
      http = super address, port

      if proxy_class? then # from Net::HTTP::Proxy()
...
@yousuketto

This comment has been minimized.

Copy link
Contributor Author

yousuketto commented May 28, 2015

I'm using ActiveResource.
ActiveResoruce create Net::HTTP instance by Net::HTTP.new(@site.host, @site.port), when I use without proxy.
https://github.com/rails/activeresource/blob/v4.0.0/lib/active_resource/connection.rb#L172

ex)

#main.rb
require 'bundler'
Bundler.require
require 'active_resource'

# for debug
class ActiveResource::Connection
  alias_method :new_http_without_debug, :new_http
  def new_http(*args)
    new_http_without_debug(*args).tap{|h| h.set_debug_output($stderr)}
  end
end

# my model
class MyModel < ActiveResource::Base
  self.site = 'http://localhost:9292'
  self.proxy = nil
end

p RUBY_VERSION
if defined?(ENV_JAVA)
  p ENV_JAVA.values_at *%w(http.proxyHost http.proxyPort http.nonProxyHosts)
end
p MyModel.all
$ jruby main.rb 
"1.9.3"
["xxx.example.com", "8080", "localhost|*.localhost]
opening connection to xxx.example.com...
opened
<- "GET http://localhost:9292/my_models.json HTTP/1.1\r\nAccept: application/json\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: localhost:9292\r\n\r\n"

2.0 mode has the same result.

@headius

This comment has been minimized.

Copy link
Member

headius commented May 28, 2015

Ok, so if I understand right, the proxy is being used, but it's not being ignored based on your ignore spec...correct?

@headius

This comment has been minimized.

Copy link
Member

headius commented May 28, 2015

So here's what I think we need to do.

  1. Net::HTTP.new needs to gain a parameter for non proxy hosts. Currently it does support them (via find_proxy in generic.rb) but only from the no_proxy ENV var. It seems a reasonable additional parameter, since the last four params are all setting up a proxy.
  2. Modify find_proxy to also check the JVM properties. I have a patch in progress now that tries http_proxy and no_proxy from ENV, falling back on the ENV_JAVA values.

The first change will take some time to get into Ruby versions, and will need to be filed with ruby-core at http://bugs.ruby-lang.org. Luckily, I'm also a ruby-core committer so I should be able to help usher it through.

The second change can be done in any version of JRuby, but I'm working on doing it in such a way that it doesn't break the current logic for defaulting to ENV_JAVA values.

@headius

This comment has been minimized.

Copy link
Member

headius commented May 28, 2015

I believe I have (2)[https://github.com//issues/2983#issuecomment-106601849] above implemented. Here's the output from your script before and after my change, with http.proxyHost set (note that http.nonProxyHosts defaults to all loopback and private network addresses):

Before:

[] ~/projects/jruby-1.7 $ jruby --2.0 -J-Dhttp.proxyHost=foobar.com blah.rb
"2.0.0"
["foobar.com", nil, "local|*.local|169.254/16|*.169.254/16|127.0.0.1|localhost|*.localhost"]
opening connection to foobar.com:80...
opened

After:

[] ~/projects/jruby-1.7 $ jruby --2.0 -J-Dhttp.proxyHost=foobar.com blah.rb
"2.0.0"
["foobar.com", nil, "local|*.local|169.254/16|*.169.254/16|127.0.0.1|localhost|*.localhost"]
opening connection to localhost:9292...

Here's the diff. We'd apply this to net/http and uri/generic for all versions: https://gist.github.com/headius/6272d168cf165ddf675f

@headius

This comment has been minimized.

Copy link
Member

headius commented May 28, 2015

I have filed a bug with MRI to get my changes incorporated, in whole or in part, back into MRI stdlib: https://bugs.ruby-lang.org/issues/11194

I filed a separate issue to add a no_proxy param to HTTP.new: https://bugs.ruby-lang.org/issues/11195

@yousuketto

This comment has been minimized.

Copy link
Contributor Author

yousuketto commented May 29, 2015

It's the behavior that I wanted.
Until it is fixed, I use code of applying your diff.
Thank you so much. 😄

@headius

This comment has been minimized.

Copy link
Member

headius commented Jun 2, 2015

I will apply the diff to JRuby 1.7 and 9k for next releases, and work separately with MRI to get the changes upstream.

@headius

This comment has been minimized.

Copy link
Member

headius commented Jun 2, 2015

Ok, so the problem I run into is that the ENV support in net/http did not arrive until 2.0, so the 1.9 net/http.rb and uri/generic.rb do not have the code needed for this.

We can apply to the 2.0 stdlib, but we have generally not backported code that ruby-core has not opted to backport themselves. JRuby 1.7.21 will support it in the (experimental) 2.0 mode and JRuby 9k will support it normally.

headius added a commit that referenced this issue Jun 2, 2015

Add support for http.nonProxyHosts and rework ENV_JAVA support.
Fixes #2983.

Note that Ruby 1.9 has no support for environment-driven proxy
settings, so this change is only available in 2.0 mode on JRuby
1.7.

@headius headius closed this in dad2061 Jun 2, 2015

@headius headius added this to the JRuby 9.0.0.0.rc1 milestone Jun 2, 2015

@headius headius added the stdlib label Jun 2, 2015

@yousuketto

This comment has been minimized.

Copy link
Contributor Author

yousuketto commented Jun 4, 2015

thank you. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.