No easy option for creating lazy ring responses #35

Open
gfredericks opened this Issue Feb 1, 2013 · 8 comments

Comments

Projects
None yet
5 participants
Collaborator

gfredericks commented Feb 1, 2013

When returning JSON from a ring server, generate-string works fine but requires setting up the whole response in memory at once. generate-stream is undoubtedly useful for other cases, but for this one requires figuring out how to hook a BufferedWriter up to an InputStream and presumably to execute generate-stream on another thread.

If there were a third option that either returned an InputStream or a lazy seq of strings (in the same manner that enlive does), this would make the ring use case much easier.

Owner

dakrone commented Feb 1, 2013

Okay, let me make sure I have this correct in my understanding.

Would it be helpful to have something like:

(def mydata {:foo "bar" :etc "...etc..."})

;; of InputStream type
(def my-inputstream (ring/get-response-inputstream))

;; with "generate-to-stream" or whatever it's named to be added:
(json/generate-to-stream mydata my-inputstream)

? (this is heavily pseudo-coded since I don't know exactly what it looks like from the ring side)

Owner

dakrone commented Feb 1, 2013

Also, I'm not quite sure what you mean by a "lazy seq of strings", since the usual output from generate-string is a single string; can you elaborate? (I'm not familiar with what enlive does in this case)

Collaborator

gfredericks commented Feb 1, 2013

The simplest case would be

(json/generate-to-stream my-data) ;; returns an input stream

Then the input stream could be returned as the response body to the ring adapter which would read from it and shoot the bytes down the wire as needed.

The lazy seq of strings tactic would be something like

(json/generate-strings {:foo [1 2 3]})
;; => ("{" "\"foo\"" ":" "[" "1" "," "2" "," "3" "]" "}")

how exactly the string is broken up isn't as important as the fact that it's lazy, and so the whole thing doesn't have to be assembled into a single spot in memory -- it can be consumed by the ring adapter lazily, in a manner analogous to using an InputStream.

Owner

dakrone commented Feb 1, 2013

Okay, that clarifies it; thanks!

boxxxie commented Apr 30, 2013

just incase anyone is interested in a work around for this issue

(ring.util.io/piped-input-stream
   (fn [out] (->> out
                 (OutputStreamWriter.)
                 (BufferedWriter.)
                 (cheshire.core/generate-stream data))))
Owner

dakrone commented May 22, 2013

@fredericksgary You may also want to check out the cheshire.experimental namespace, as we've been experimenting with stream parsing in there.

aviflax commented Jun 4, 2013

+1 on this, and I don’t understand how to employ @boxxxie’s suggested workaround.

@aviflax Use this function to turn data into an output stream:

(defn generate-stream
  ([data] (generate-stream data nil))
  ([data options]
    (ring.util.io/piped-input-stream
      (fn [out] (cheshire.core/generate-stream data 
                                               (-> out
                                                 (OutputStreamWriter.)
                                                 (BufferedWriter.))
                                               options)))))

@jokimaki jokimaki pushed a commit to lupapiste/lupapiste that referenced this issue Jun 22, 2016

@Snurppa Snurppa return json data as stream for /data-api/json, provides faster data t…
…ransfer start times

piped-input-stream streams data in separate thread, might have some overhead...

https://nelsonmorris.net/2015/04/22/streaming-responses-using-ring.html
dakrone/cheshire#35
195f396
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment