Skip to content
Browse files

convert objets to byte arrays since not all stores can stream

I would prefer to use the InputStream Payload but am having issues
converting an OutputStream to an InputStream.  The big downside to the
byte-array approach is that the object needs to be stored in memory
twice now.  Which isn't possible in some cases with large objects...
  • Loading branch information...
1 parent d976b10 commit 664df92dce63583caf042d02e230d021f4813869 @bmabey committed Jun 3, 2011
Showing with 22 additions and 8 deletions.
  1. +22 −8 blobstore/src/clj_cache/blobstore.clj
View
30 blobstore/src/clj_cache/blobstore.clj
@@ -1,15 +1,29 @@
(ns clj-cache.blobstore
(use org.jclouds.blobstore2)
- (import [java.io File ObjectOutputStream ObjectInputStream Serializable]
+ (import [java.io File ObjectOutputStream ObjectInputStream Serializable ByteArrayOutputStream]
[java.util.zip GZIPOutputStream GZIPInputStream]))
-(defn- obj->stream-fn
+(defn- write-object [^Serializable obj output-stream & [options]]
+ (let [os (if (:gzip options) (GZIPOutputStream. output-stream) output-stream)]
+ (with-open [out (ObjectOutputStream. os)]
+ (.writeObject out obj)))
+ obj)
+
+;; TODO: use this fn instead of obj->input-stream when possible
+;; (Not all BlobStores support streaming uploads yet, so not using this by default)
+(defn obj->stream-fn
"Takes an object returning a function that will write (using Java serialization) the object to the given OutputStream."
- [^Serializable obj & [options]]
+ [obj & [options]]
(fn [output-stream]
- (let [os (if (:gzip options) (GZIPOutputStream. output-stream) output-stream)]
- (with-open [out (ObjectOutputStream. os)]
- (.writeObject out obj)))))
+ (write-object obj output-stream options)))
+
+(defn obj->byte-array
+ "Serializes an object storing it into a byte array that is returned.
+ NOTE: Both the object and the byte array will be kept in memory."
+ [obj & [options]]
+ (let [out (ByteArrayOutputStream.)]
+ (write-object obj out options)
+ (.toByteArray out)))
(defn stream->obj
"Takes an InputStream and deserializes it into an Object."
@@ -22,12 +36,12 @@
"Looks up the map by name and assocs the k/v pair only if the key is absent."
[{:keys [blobstore container path serialization-options]} key value]
(when-not (blob-exists? blobstore container (path key))
- (put-blob blobstore container (blob (path key) :payload (obj->stream-fn value serialization-options))))
+ (put-blob blobstore container (blob (path key) :payload (obj->byte-array value serialization-options))))
value)
(defn lookup
"Looks up an item in the given cache. Returns a vector:
-[element-exists? value]"
+ [element-exists? value]"
[{:keys [blobstore container path serialization-options]} key]
(if (blob-exists? blobstore container (path key))
[true (-> (get-blob-stream blobstore container (path key))

0 comments on commit 664df92

Please sign in to comment.
Something went wrong with that request. Please try again.