Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

No easy option for creating lazy ring responses #35

Open
gfredericks opened this Issue · 8 comments

5 participants

@gfredericks

When returning JSON from a ring server, generate-string works fine but requires setting up the whole response in memory at once. generate-stream is undoubtedly useful for other cases, but for this one requires figuring out how to hook a BufferedWriter up to an InputStream and presumably to execute generate-stream on another thread.

If there were a third option that either returned an InputStream or a lazy seq of strings (in the same manner that enlive does), this would make the ring use case much easier.

@dakrone
Owner

Okay, let me make sure I have this correct in my understanding.

Would it be helpful to have something like:

(def mydata {:foo "bar" :etc "...etc..."})

;; of InputStream type
(def my-inputstream (ring/get-response-inputstream))

;; with "generate-to-stream" or whatever it's named to be added:
(json/generate-to-stream mydata my-inputstream)

? (this is heavily pseudo-coded since I don't know exactly what it looks like from the ring side)

@dakrone
Owner

Also, I'm not quite sure what you mean by a "lazy seq of strings", since the usual output from generate-string is a single string; can you elaborate? (I'm not familiar with what enlive does in this case)

@gfredericks

The simplest case would be

(json/generate-to-stream my-data) ;; returns an input stream

Then the input stream could be returned as the response body to the ring adapter which would read from it and shoot the bytes down the wire as needed.

The lazy seq of strings tactic would be something like

(json/generate-strings {:foo [1 2 3]})
;; => ("{" "\"foo\"" ":" "[" "1" "," "2" "," "3" "]" "}")

how exactly the string is broken up isn't as important as the fact that it's lazy, and so the whole thing doesn't have to be assembled into a single spot in memory -- it can be consumed by the ring adapter lazily, in a manner analogous to using an InputStream.

@dakrone
Owner

Okay, that clarifies it; thanks!

@boxxxie

just incase anyone is interested in a work around for this issue

(ring.util.io/piped-input-stream
   (fn [out] (->> out
                 (OutputStreamWriter.)
                 (BufferedWriter.)
                 (cheshire.core/generate-stream data))))
@dakrone
Owner

@fredericksgary You may also want to check out the cheshire.experimental namespace, as we've been experimenting with stream parsing in there.

@aviflax

+1 on this, and I don’t understand how to employ @boxxxie’s suggested workaround.

@paul-lshift

@aviflax Use this function to turn data into an output stream:

(defn generate-stream
  ([data] (generate-stream data nil))
  ([data options]
    (ring.util.io/piped-input-stream
      (fn [out] (cheshire.core/generate-stream data 
                                               (-> out
                                                 (OutputStreamWriter.)
                                                 (BufferedWriter.))
                                               options)))))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.