This repository has been archived by the owner on Feb 13, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 2
/
README
118 lines (100 loc) · 3.44 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
== Retrieve
Homepage:: retrieve.rubyforge.org[http://retrieve.rubyforge.org/]
Author:: Bob Aman (mailto:bob@sporkmonger.com)
Copyright:: Copyright © 2008 Bob Aman
License:: MIT
== Description
Retrieve is an extensible library for retrieving resources identified by a
URI. Retrieval methods are registered to URI schemes.
== Features
* Access any resource with a client defined for its scheme.
* Read and write to anything that can be addressed by a URI.
* Easily implement additional clients for other URI schemes.
* Supports "http" and "file" schemes by default.
* The HTTP client is faster and more efficient than Net::HTTP.
== Example Usage
require "retrieve/clients/file"
res = Retrieve.open("file:///home/sporkmonger/todo.txt", :mode => :read)
#=> #<Retrieve::Resource:0x847c2c URI:file:///home/sporkmonger/todo.txt>
res.read
#=> "TODO: Write some code."
res.close
require "retrieve/clients/http"
status, headers, body = nil
Retrieve.open("http://www.google.com/", :method => :get) do |resource|
body = resource.read
status = resource.metadata[:status]
reason = resource.metadata[:reason]
headers = resource.metadata[:headers]
end
status
#=> "200" # NOTE: This is a String
reason
#=> "OK"
headers
#=> {"Content-Type" => "text/html; charset=ISO-8859-1", ...}
body[0..50] + " ..."
#=> "<html><head><meta http-equiv=\"content-type\" content ..."
# Usage for persistent connections
connections = {}
status, headers, body = nil
Retrieve.open(
"http://google.com/",
:redirect => true,
:connections => connections
) do |resource|
body = resource.read
end
body[0..50] + " ..."
#=> "<html><head><meta http-equiv=\"content-type\" content ..."
# Advanced example using ranges
connections = {}
uri = "http://www.example.com/"
File.open("example.html", "w") do |file|
file.write(File.open("http.log", "a") do |log|
accept_ranges = nil
content_length = nil
bytes_received = 0
Retrieve.open(
uri, :method => :head, :connections => connections, :log => log
) do |r|
accept_ranges = r.metadata[:headers]["Accept-Ranges"]
content_length = r.metadata[:headers]["Content-Length"]
end
buffer = StringIO.new
if accept_ranges == "bytes"
log.write("* Server claims to support partial content.\n")
while content_length.to_i > bytes_received
if (131072 + bytes_received) > content_length.to_i
range = bytes_received..-1
else
range = bytes_received..131072 + bytes_received
end
Retrieve.open(
uri, :connections => connections, :range => range, :log => log
) do |r|
buffer.write(r.read)
bytes_received += r.metadata[:headers]["Content-Length"].to_i
if r.metadata[:status] == "200"
log.write("* Server did not return partial content.\n")
end
end
end
else
log.write("* Server does NOT claim to support partial content.\n")
Retrieve.open(uri, :connections => connections, :log => log) do |r|
buffer.write(r.read)
end
end
for pair, connection in connections
log.write("* Closing connection to #{pair[0]} port #{pair[1]}\n")
connection.close if connection
connections.delete(pair)
end
buffer.string
end)
end
== Requirements
* addressable
== Install
* sudo gem install retrieve