Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

renamed from http-client to clj-web-crawler

  • Loading branch information...
commit e705d710962928c1234139f95e385911ae0ca166 1 parent 7f161df
@heyZeus authored
Showing with 24 additions and 22 deletions.
  1. +11 −9 README
  2. +3 −3 http_client.clj → clj_web_crawler.clj
  3. +10 −10 test/main.clj
View
20 README
@@ -1,18 +1,20 @@
-http-client is a wrapper around Apache commons-client for the Clojure programming language.
+clj-web-crawler allows you to easily crawl and scrape the web from the comfort of
+your own computer. This Clojure script is a wrapper around the Apache
+commons-client Java library.
USAGE
; Prints the HTML of the clojure.org website
-(let [server (client "http://www.clojure.org")
- home (method "/")]
- (println (scrape server home)))
+(let [clj-ws (client "http://www.clojure.org")
+ home (method "/")]
+ (println (scrape clj-ws home)))
; If you don't care about the HTML from the query you should just call
; send-method. In this example you are posting the login form and need
; to make sure a cookie is set to validate the login was successful.
-(let [server (client "http://www.example.com")
+(let [site (client "http://www.example.com")
login (method "/accounts/login" :post {:login "mr_cool" :password "clojurerox"})]
- (send-method server login)
- (if (assert-cookie-names server "username")
+ (send-method site login)
+ (if (assert-cookie-names site "username")
(println "yeah, I'm in")
(println "i can't remember my password again!")))
@@ -20,9 +22,9 @@ USAGE
; like check the response status code. Note you can't check the response
; code outside of the send-method call since all associated resources are
; released at that point.
-(let [server (client "http://www.clojure.org")
+(let [clj-ws (client "http://www.clojure.org")
login (method "/")]
- (send-method server login
+ (send-method clj-ws login
(println (.getStatusCode login))))
I've only implemented some basic functionality to make the commons-client
View
6 http_client.clj → clj_web_crawler.clj
@@ -1,4 +1,4 @@
-(ns http-client
+(ns clj-web-crawler
(:import (org.apache.commons.httpclient HttpClient NameValuePair URI HttpStatus)
(org.apache.commons.httpclient.cookie CookiePolicy CookieSpec)
(org.apache.commons.httpclient.methods GetMethod PostMethod DeleteMethod
@@ -43,9 +43,9 @@
(defn client
"Creates a HttpClient for the given server."
- [server]
+ [host]
(let [c (HttpClient.)]
- (.. c (getHostConfiguration) (setHost (URI. server true)))
+ (.. c (getHostConfiguration) (setHost (URI. host true)))
c))
(defn method
View
20 test/main.clj
@@ -1,19 +1,19 @@
-(ns http-client.test.main
- (:require [http-client :as hc])
+(ns clj-web-crawler.test.main
+ (:require [clj-web-crawler :as wc])
(:use clojure.contrib.test-is))
-(def clj-client (hc/client "http://www.clojure.org"))
-(def home (hc/method "/"))
+(def clj-ws (wc/client "http://www.clojure.org"))
+(def home (wc/method "/"))
(deftest client
(let [host "www.clojure.org"
- client (hc/client (str "http://" host))]
- (is (= (.. client (getHostConfiguration) getHost) host))))
+ clj-ws (wc/client (str "http://" host))]
+ (is (= (.. clj-ws (getHostConfiguration) getHost) host))))
(deftest method
- (let [default-get-method (hc/method "/")
- path-post-method (hc/method "/api" :post)
- params-method (hc/method "/" :get {:language "clojure" :happy "yes"})]
+ (let [default-get-method (wc/method "/")
+ path-post-method (wc/method "/api" :post)
+ params-method (wc/method "/" :get {:language "clojure" :happy "yes"})]
(is (= "GET" (.getName default-get-method)))
(is (= "/" (.getPath default-get-method)))
@@ -25,7 +25,7 @@
(is (= "yes" (.. params-method (getParameter "happy") (getValue))))))
(deftest scrape
- (let [html (hc/scrape clj-client home)]
+ (let [html (wc/scrape clj-ws home)]
(is (= (.contains html "Clojure")))))
(run-tests)
Please sign in to comment.
Something went wrong with that request. Please try again.