Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Https urls can't be opened #16

Merged
merged 8 commits into from

2 participants

@samphippen

I just tried opening a https url in 0.3.0 (the current version on rubygems) and it totally failed. I pulled the version from master and it doesn't work there, so I've written this spec to expose the behaviour. I'm now going to try and fix it.

samphippen added some commits
@samphippen samphippen Add a spec that exposes the failing behaviour
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
17559ca
@samphippen samphippen Fix the https test
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
3bd40c3
@samphippen

this looks more interesting than I had initially thought, the github examples from the readme all return 500...

@samphippen

Done some investigation with wireshark, looks like we might need to add user agent and host headers by default.

samphippen added some commits
@samphippen samphippen Fix openssl context and host header
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
f0ac101
@samphippen samphippen Add a user agent if one hasn't been specified
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
28237a1
@samphippen samphippen Update specs for new user agent header
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
894f2ec
@samphippen samphippen Now that we have a working library, we can have a working test
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
1b3b500
@samphippen samphippen Add some real world sites to the specs
Signed-off-by: Sam Phippen <samphippen@googlemail.com>
945ee28
@samphippen samphippen Fix some more real world connection issues
specifically this didn't deal with the case where the server
used a local redirect instead of a global one

Signed-off-by: Sam Phippen <samphippen@googlemail.com>
b0a1a6c
@samphippen

In 945ee28 and b0a1a6c I've added some real world sites and some fixes for getting them to the test suite. This seems suitable given that the gem is used for opening real world sites.

@tarcieri tarcieri merged commit dbc5492 into from
@tarcieri
Owner

Awesome, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 6, 2012
  1. @samphippen

    Add a spec that exposes the failing behaviour

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  2. @samphippen

    Fix the https test

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  3. @samphippen

    Fix openssl context and host header

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  4. @samphippen

    Add a user agent if one hasn't been specified

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  5. @samphippen

    Update specs for new user agent header

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  6. @samphippen

    Now that we have a working library, we can have a working test

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  7. @samphippen

    Add some real world sites to the specs

    samphippen authored
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
  8. @samphippen

    Fix some more real world connection issues

    samphippen authored
    specifically this didn't deal with the case where the server
    used a local redirect instead of a global one
    
    Signed-off-by: Sam Phippen <samphippen@googlemail.com>
This page is out of date. Refresh to see the latest.
View
17 lib/http/client.rb
@@ -25,17 +25,23 @@ def body(opts,headers)
# Make an HTTP request
def request(method, uri, options = {})
opts = @default_options.merge(options)
+ host = URI.parse(uri).host
+ opts.headers["Host"] = host
headers = opts.headers
proxy = opts.proxy
method_body = body(opts, headers)
- puts method_body
request = Request.new method, uri, headers, proxy, method_body
if opts.follow
code = 302
while code == 302 or code == 301
- puts uri
+ # if the uri isn't fully formed complete it
+ if not uri.match /\./
+ uri = "http://#{host}#{uri}"
+ end
+ host = URI.parse(uri).host
+ opts.headers["Host"] = host
method_body = body(opts, headers)
request = Request.new method, uri, headers, proxy, method_body
response = perform request, opts
@@ -57,7 +63,12 @@ def perform(request, options)
socket = options[:socket_class].open(uri.host, uri.port) # TODO: proxy support
if uri.is_a?(URI::HTTPS)
- socket = options[:ssl_socket_class].open(socket, options[:ssl_context])
+ if options[:ssl_context] == nil
+ context = OpenSSL::SSL::SSLContext.new
+ else
+ context = options[:ssl_context]
+ end
+ socket = options[:ssl_socket_class].new(socket, context)
socket.connect
end
View
3  lib/http/options.rb
@@ -57,6 +57,9 @@ def initialize(options = {})
@socket_class = options[:socket_class] || self.class.default_socket_class
@ssl_socket_class = options[:ssl_socket_class] || self.class.default_ssl_socket_class
@ssl_context = options[:ssl_context]
+ if not @headers["User-Agent"]
+ @headers["User-Agent"] = "HTTP Gem"
+ end
end
def with_response(response)
View
8 spec/http/options/headers_spec.rb
@@ -4,14 +4,14 @@
let(:opts){ Http::Options.new }
- it 'defaults to {}' do
- opts.headers.should eq({})
+ it 'defaults to just the user agent' do
+ opts.headers.should eq({"User-Agent" => "HTTP Gem"})
end
it 'may be specified with with_headers' do
opts2 = opts.with_headers("accept" => "json")
- opts.headers.should eq({})
- opts2.headers.should eq("accept" => "json")
+ opts.headers.should eq({"User-Agent"=>"HTTP Gem"})
+ opts2.headers.should eq("accept" => "json", "User-Agent"=>"HTTP Gem")
end
it 'accepts any object that respond to :to_hash' do
View
2  spec/http/options/merge_spec.rb
@@ -34,7 +34,7 @@
:response => :parsed_body,
:form => {:bar => 'bar'},
:body => "body-bar",
- :headers => {:accept => "xml", :foo => "foo", :bar => 'bar'},
+ :headers => {:accept => "xml", :foo => "foo", :bar => 'bar', "User-Agent"=>"HTTP Gem"},
:proxy => {:proxy_address => "127.0.0.1", :proxy_port => 8080},
:callbacks => {:request => ["common"], :response => ["foo", "bar"]},
:follow => nil
View
2  spec/http/options/new_spec.rb
@@ -16,7 +16,7 @@
it 'coerces :headers correctly' do
opts = Http::Options.new(:headers => {:accept => "json"})
- opts.headers.should eq(:accept => "json")
+ opts.headers.should eq(:accept => "json", "User-Agent"=>"HTTP Gem")
end
it 'coerces :proxy correctly' do
View
13 spec/http_spec.rb
@@ -11,6 +11,19 @@
response.should match(/<!doctype html>/)
end
+ it "should be easy to get a https resource" do
+ response = Http.with_headers(:accept => 'application/json').get "https://api.github.com/users/samphippen"
+ response["type"].should == "User"
+ end
+
+ it "can get some real world sites, following redirects if necessary" do
+ sites = ["http://github.com/", "http://xkcd.com/", "http://www.spotify.com/"]
+ sites.each do |site|
+ resp = Http.with_response(:object).with_follow(true).get site
+ resp.status.should == 200
+ end
+ end
+
context "with_response" do
it 'allows specifying :object' do
res = Http.with_response(:object).get test_endpoint
Something went wrong with that request. Please try again.