postmodern / spidr

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. — Read more

Sort by: Priority | Votes | Last Updated

Loading…

2.

0 votes

SSL and network, in general
4 comments Created 3 months ago by justfalter

Title

Body
A couple of things 1. From everything I can tell about the net/https , you need to set 'use_ssl' to true when you want things to run over SSL. You'd also want to turn off certificate validation, as since we are spidering, we really don't care about the validity, just connectivity. Ex: http = Net::HTTP.new(host,port) http.use_ssl = true http.verify_mode = OpenSSL::SSL::VERIFY_NONE 2. You really, really, really need to re-use your HTTP session objects once they are created. Right now, Agent will create and destroy a full connection for each request. With SSL is involed, the overhead for negotiating an SSL connection will kill a CPU. You can easily share your sessions if you create a hash that is keyed off of [host,port]. If you run into an error while making an HTTP request, destroy the session and recreate it. You'll be able to run hundreds, if not thousands of HTTP requests over a single TCP connection.
or cancel
A couple of things
1. From everything I can tell about the net/https , you need to set 'use_ssl' to true when you want things to run over SSL. You'd also want to turn off certificate validation, as since we are spidering, we really don't care about the validity, just connectivity. Ex:
  http = Net::HTTP.new(host,port)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE
2. You really, really, really need to re-use your HTTP session objects once they are created. Right now, Agent will create and destroy a full connection for each request. With SSL is involed, the overhead for negotiating an SSL connection will kill a CPU.
  You can easily share your sessions if you create a hash that is keyed off of [host,port]. If you run into an error while making an HTTP request, destroy the session and recreate it. You'll be able to run hundreds, if not thousands of HTTP requests over a single TCP connection.
Comments

postmodern Wed Sep 23 18:01:17 -0700 2009 | link

Alright, I've just added a basic HTTP session Hash to Spidr::Agent and the Agent#get_session method. Now Agent#get_page will set use_ssl and verify_mode of the HTTP session, only if the scheme of the requested URL is 'https'.

Comment
Alright, I've just added a basic HTTP session Hash to Spidr::Agent and the Agent#get_session method. Now Agent#get_page will set use_ssl and verify_mode of the HTTP session, only if the scheme of the requested URL is 'https'.
or cancel

postmodern Mon Oct 05 14:23:33 -0700 2009 | link

Would you consider this properly fixed and optimized now? If so, I'll close this issue.

Comment
Would you consider this properly fixed and optimized now? If so, I'll close this issue.
or cancel

justfalter Thu Oct 15 11:31:17 -0700 2009 | link

I'm a terrible person for not responding to this 9 days ago when you sent your message. Sorry, yes, I would consider this closed. My bad!

~Mike

Comment
I'm a terrible person for not responding to this 9 days ago when you sent your message. Sorry, yes, I would consider this closed. My bad! ~Mike
or cancel

justfalter Thu Oct 15 11:31:29 -0700 2009 | link

er, comment, AND close.

Comment
er, comment, AND close.
or cancel

Parsed with GitHub Flavored Markdown
3.

0 votes

static course specs
0 comments Created 2 days ago by zapnap

Title

Body
I'm working on a patch, and trying to augment the course-related specs with some other real-world examples. However, when I copy the static/course directory to a public webserver and change the COURSE_URL in spec/helpers/course.rb to reflect that, I get a number of spec failures (16 to be exact). Is this expected? Are there particular rewrite rules or something that aren't codified in the static directory I'd need in order to replicate this?
or cancel

I'm working on a patch, and trying to augment the course-related specs with some other real-world examples. However, when I copy the static/course directory to a public webserver and change the COURSE_URL in spec/helpers/course.rb to reflect that, I get a number of spec failures (16 to be exact).

Is this expected? Are there particular rewrite rules or something that aren't codified in the static directory I'd need in order to replicate this?

Comments

Parsed with GitHub Flavored Markdown

Labels

postmodern / spidr

Pledgie Donations

SSL and network, in general

Comments

static course specs

Comments