Permalink
Browse files

more correct handling of files

better sample galleries! :P
  • Loading branch information...
2 parents b9fe097 + b342628 commit f736c9cf56c48be8c161a3c620b24e82b8dba357 @pistacchio committed Oct 21, 2011
Showing with 34 additions and 15 deletions.
  1. +16 −3 README.md
  2. +1 −1 resources/data/data.dat
  3. +17 −11 src/org/github/pistacchio/deviantchecker/core.clj
View
@@ -87,7 +87,8 @@ All web-related code is contained within `/src/org/github/pistacchio/deviantchec
[clojure.contrib.json :only (json-str)]
net.cgrand.enlive-html)
(:require [compojure.route :as route]
- [compojure.handler :as handler])
+ [compojure.handler :as handler]
+ [clojure.java.io :as io])
(:gen-class))
Apart from `compojure` and Ring Jetty adapter we've talked about, we are going to need:
@@ -96,6 +97,7 @@ Apart from `compojure` and Ring Jetty adapter we've talked about, we are going t
* Enlive
* `compojure.route` for defining the way calls to the server are handled
* `compojure.handler` that has some utility functions we'll use.
+* `compojure.java.io` for accessing files on the file system
Finally, we use `org.github.pistacchio.deviantchecker.scraper` (our functions for scraping Deviantart) and `(:gen-class)` to make a Java class out of our namespace. This, together with the definition of the main method at the end of the file `(defn -main [& args] (run-jetty app {:port 3000}))` makes possibile to specify this as the principal class for our project (remember `:main org.github.pistacchio.deviantchecker.core` in `project.clj`?)
@@ -161,8 +163,17 @@ We're almost done! `(route/not-found "Page not found")` will return "Page not fo
We are storing all the information about Deviantart galleries in a plain text file: `/resources/data/data.dat`. In a real world application, of course, a database backend would be compulsory, but for our example this is enough.
+Note that “n the code I define the name of our file as `data/data.day` because _resources_ is already the default path that Leiningen uses for storing additional resources. If you pack the project as a .jar or .war, resources files would be taken out of `/resources` and deployed in the root directory of the archive.
+
The dynamic nature of Clojure makes serialization really easy. To load our data we use [`(load-file)`](http://clojuredocs.org/clojure_core/1.2.0/clojure.core/load-file), to store it to a file once we've modified it, we just use [`(print-dup)`](http://clojuredocs.org/clojure_core/clojure.core/print-dup).
+I've written a utility function to feed those two functions with a correct path to the data file. I use the same function later on when having to retrieve a template file.
+
+ (defn get-file
+ "return the absolute path of relative-path"
+ [relative-path]
+ (.getPath (io/resource relative-path)))
+
### Data structure in deviantSCRAPER
I won't go into details about our data layer because you can easily read about it through the code. Breafly, data about galleries is stored (in memory and on the file) as a sequence of maps. Each map represents a gallery. For each gallery we store:
@@ -188,7 +199,7 @@ Since we are doing some page scraping and Enlive is both a template engine and a
If you look at `/resources/tpl/home.html`, you'll find out that it's just plain html. We'll take a look at `(get-home)` to see how we use Enlive. The key function call is
- (html-resource (java.io.File. "resources/tpl/home.html")
+ (html-resource (java.io.File. (get-file "tpl/home.html"))
This gives us back a parsed version of our html file. Basically `net.cgrand.enlive-html/html-resource` converts the html file into a Clojure structure made of sequences and maps. For example, if we parse a file like
@@ -281,7 +292,9 @@ and run the application like you would do with any java .jar file.
You can also use `lein jar` that produces a smaller .jar by not including every dependency library, but you'll then have to tinker with the `CLASSPATH`.
-Leiningen can also pack it all into a _.war_ file with `lein ring war` and `lein ring uberwar` so that you can deploy it under [Tomcat](http://tomcat.apache.org/) or any other Java server supporting wars. I haven't tested this already, but when I'll do I'll update the tutorial.
+Leiningen can also pack it all into a _.war_ file with `lein ring war` and `lein ring uberwar` so that you can deploy it under [Tomcat](http://tomcat.apache.org/) or any other Java server supporting wars.
+
+I tried this under Tomcat and it worked _almost_ well. The application runs and works out of the box but with a glitch I haven't been able to overcome. When you visit `http://localhost:8080/deviantscraper` (assuming we've deployed `deviantscraper.war`), Tomcat `should` perform a 302 redirect to `http://localhost:8080/deviantscraper/` (<- note the trailing slash!). This didn't work in my environment causing the .css and .gif file not to load.
## Contact
View
@@ -1 +1 @@
-[{:href "http://dragol.deviantart.com/gallery", :last-page "http://dragol.deviantart.com/gallery/?offset=96", :num-images "10"} {:href "http://Minish-Mae.deviantart.com/gallery", :last-page "http://Minish-Mae.deviantart.com/gallery/?offset=72", :num-images "5"} {:href "http://HotTorchic .deviantart.com/gallery", :last-page "http://HotTorchic .deviantart.com/gallery/?offset=24", :num-images "1"} {:href "http://honeysempailover.deviantart.com/gallery", :last-page "http://honeysempailover.deviantart.com/gallery/?offset=24", :num-images "18"} {:href "http://HotTorchic.deviantart.com/gallery", :last-page "http://HotTorchic.deviantart.com/gallery/?offset=24", :num-images "1"} {:href "http://tacticiandin .deviantart.com/gallery", :last-page "http://tacticiandin .deviantart.com/gallery/", :num-images "4"} {:href "http://123qweasd.deviantart.com/gallery", :last-page "http://123qweasd.deviantart.com/gallery/", :num-images "1"} {:href "http://e-lite.deviantart.com/gallery", :last-page "http://e-lite.deviantart.com/gallery/", :num-images "18"}]
+[{:href "http://zerg118.deviantart.com/gallery", :last-page "http://zerg118.deviantart.com/gallery/?offset=24", :num-images "4"} {:href "http://acrylicdreams.deviantart.com/gallery", :last-page "http://acrylicdreams.deviantart.com/gallery/?offset=96", :num-images "5"} {:href "http://mrshot.deviantart.com/gallery", :last-page "http://mrshot.deviantart.com/gallery/?offset=72", :num-images "7"} {:href "http://guthrieartwork.deviantart.com/gallery", :last-page "http://guthrieartwork.deviantart.com/gallery/?offset=24", :num-images "3"} {:href "http://njay.deviantart.com/gallery", :last-page "http://njay.deviantart.com/gallery/?offset=144", :num-images "20"} {:href "http://tsuji.deviantart.com/gallery", :last-page "http://tsuji.deviantart.com/gallery/?offset=168", :num-images "10"} {:href "http://fyreant.deviantart.com/gallery", :last-page "http://fyreant.deviantart.com/gallery/?offset=168", :num-images "14"}]
@@ -5,21 +5,33 @@
[clojure.contrib.json :only (json-str)]
net.cgrand.enlive-html)
(:require [compojure.route :as route]
- [compojure.handler :as handler])
+ [compojure.handler :as handler]
+ [clojure.java.io :as io])
(:gen-class))
-(def *data-file* "resources/data/data.dat")
+(def *data-file* "data/data.dat")
+
+;; ** utilities ** ;;
+(defn emit**
+ "given an enlive form, returns a string"
+ [tpl]
+ (apply str (emit* tpl)))
+
+(defn get-file
+ "return the absolute path of relative-path"
+ [relative-path]
+ (.getPath (io/resource relative-path)))
;; ** data management ** ;;
(defn load-data
"returns a gallery database stored in *data-file*"
[]
- (load-file *data-file*))
+ (load-file (get-file *data-file*)))
(defn save-data
"serializes data to *data-file* like"
[data]
- (with-open [f (java.io.FileWriter. *data-file*)]
+ (with-open [f (java.io.FileWriter. (get-file *data-file*))]
(print-dup (into [] data) f)))
(defn get-gallery
@@ -35,18 +47,12 @@
(remove #(= (% :href) gallery-url) data)
gallery))))
-;; ** utilities ** ;;
-(defn emit**
- "given an enlive form, returns a string"
- [tpl]
- (apply str (emit* tpl)))
-
;; ** views ** ;;
(defn get-home
"GET /
error-message may be a vector with a selector and an error message to display within it."
[& error-message]
- (let [tpl (html-resource (java.io.File. "resources/tpl/home.html"))
+ (let [tpl (html-resource (java.io.File. (get-file "tpl/home.html")))
data (load-data)
main-transform #(transform % [:li.gallery]
(clone-for [g data]

0 comments on commit f736c9c

Please sign in to comment.