Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Assert bug (duplicates in :imagetags); remove :path from :image_id; spit

most tables successfully
  • Loading branch information...
commit 013c097e82683b8473d876b379257c3bf69abab2 1 parent ef358fd
Tim McCormack authored

Showing 2 changed files with 11 additions and 5 deletions. Show diff stats Hide diff stats

  1. +3 0  clj/src/kpawebgen/munge.clj
  2. +8 5 clj/src/kpawebgen/spit.clj
3  clj/src/kpawebgen/munge.clj
@@ -230,6 +230,7 @@ new table named by 'merged' will be added."
230 230 * :tagstoimages gains :implicit column and is renamed to :imagetags
231 231 * :metadata is removed
232 232 * :changelog is removed
  233 +* :image_id loses :path
233 234 * :image_id, :image_int, and :image_ext are removed and joined into :images"
234 235 [conf db]
235 236 (let [conf (pre-process-conf conf db)
@@ -243,5 +244,7 @@ new table named by 'merged' will be added."
243 244 ;; TODO: remove orphaned tags
244 245 ;; tag filtering comes after image filtering in case a private tag
245 246 ;; is used for marking private images
  247 + ;; post-processing
  248 + (update-in [:image_id] (partial map #(dissoc % :path)))
246 249 (clojure.set/rename-keys {:tagstoimages :imagetags})
247 250 (join-on :imageID [:image_id :image_int :image_ext] :images))))
13 clj/src/kpawebgen/spit.clj
@@ -51,16 +51,19 @@ data structures and photo store."
51 51 ;; write schema
52 52 (run-script (slurp (io/as-file (io/resource "kpawebgen/gallery.sql"))))
53 53 ;; write data
54   - #_(into {}
55   - (for [tbl tables]
56   - (sql/with-query-results r
57   - [(str "SELECT * FROM " (name tbl))]
58   - [tbl (doall r)]))))))
  54 + (doseq [[tbl records] db]
  55 + (println "Spitting table" tbl)
  56 + (doseq [chunk (partition-all 300 records)]
  57 + (println " Spitting chunk" (count chunk))
  58 + (apply sql/insert-records tbl chunk))))))
59 59
60 60 (defn write
61 61 "Write website data out to disk and generate all files.
62 62 Does not yet optimize based on diff to last localweb generation."
63 63 [config db]
  64 + {:pre [(= (count (:imagetags db))
  65 + (count (clojure.set/project (set (:imagetags db))
  66 + [:imageID :catID :tagID])))]}
64 67 (let [db-f (ensure-blank-db config)]
65 68 (write-db db db-f)
66 69 (println "Tables:" (into {} (for [[tn tbl] db] [tn (keys (first tbl))])))))

0 comments on commit 013c097

Please sign in to comment.
Something went wrong with that request. Please try again.