No easy option for creating lazy ring responses

[gfredericks]

When returning JSON from a ring server, generate-string works fine but requires setting up the whole response in memory at once. generate-stream is undoubtedly useful for other cases, but for this one requires figuring out how to hook a BufferedWriter up to an InputStream and presumably to execute generate-stream on another thread.

If there were a third option that either returned an InputStream or a lazy seq of strings (in the same manner that enlive does), this would make the ring use case much easier.

[dakrone]

Okay, let me make sure I have this correct in my understanding.

Would it be helpful to have something like:

(def mydata {:foo "bar" :etc "...etc..."})

;; of InputStream type
(def my-inputstream (ring/get-response-inputstream))

;; with "generate-to-stream" or whatever it's named to be added:
(json/generate-to-stream mydata my-inputstream)

? (this is heavily pseudo-coded since I don't know exactly what it looks like from the ring side)

[dakrone]

Also, I'm not quite sure what you mean by a "lazy seq of strings", since the usual output from generate-string is a single string; can you elaborate? (I'm not familiar with what enlive does in this case)

[gfredericks]

The simplest case would be

(json/generate-to-stream my-data) ;; returns an input stream

Then the input stream could be returned as the response body to the ring adapter which would read from it and shoot the bytes down the wire as needed.

The lazy seq of strings tactic would be something like

(json/generate-strings {:foo [1 2 3]})
;; => ("{" "\"foo\"" ":" "[" "1" "," "2" "," "3" "]" "}")

how exactly the string is broken up isn't as important as the fact that it's lazy, and so the whole thing doesn't have to be assembled into a single spot in memory -- it can be consumed by the ring adapter lazily, in a manner analogous to using an InputStream.

[dakrone]

Okay, that clarifies it; thanks!

[boxxxie]

just incase anyone is interested in a work around for this issue

(ring.util.io/piped-input-stream
   (fn [out] (->> out
                 (OutputStreamWriter.)
                 (BufferedWriter.)
                 (cheshire.core/generate-stream data))))

[dakrone]

@fredericksgary You may also want to check out the cheshire.experimental namespace, as we've been experimenting with stream parsing in there.

[aviflax]

+1 on this, and I don’t understand how to employ @boxxxie’s suggested workaround.

[paul-lshift]

@aviflax Use this function to turn data into an output stream:

(defn generate-stream
  ([data] (generate-stream data nil))
  ([data options]
    (ring.util.io/piped-input-stream
      (fn [out] (cheshire.core/generate-stream data 
                                               (-> out
                                                 (OutputStreamWriter.)
                                                 (BufferedWriter.))
                                               options)))))

[jconti]

Endpoints from systems like ElasticSearch package results as multiple top level objects, which facilitates this.