Long response got cut off when wrap-base
#192
Comments
|
The related codes are here: (defn- build-url [host path query-string]
(let [url (str host path)]
(if (not-empty query-string)
(str url "?" query-string)
url)))
(defn prepare-cookies
"Removes the :domain and :secure keys and converts the :expires key (a Date)
to a string in the ring response map resp. Returns resp with cookies properly
munged."
[resp]
(let [prepare #(-> (update-in % [1 :expires] str)
(update-in [1] dissoc :domain :secure))]
(assoc resp :cookies (into {} (map prepare (:cookies resp))))))
(defn slurp-binary
"Reads len bytes from InputStream is and returns a byte array."
[^java.io.InputStream is len]
(when (and is
(> len 0))
(with-open [rdr is]
(let [buf (byte-array len)]
(.read rdr buf)
buf))))
(defn create-proxy [handler fns]
(let [identifier-fn (get fns :identifier-fn identity)
host-fn (get fns :host-fn {})
path-fn (get fns :path-fn identity)]
(fn [request]
(let [request-key (identifier-fn request)
host (host-fn request-key)
stripped-headers (dissoc (:headers request) "content-length" "host")
path (path-fn (:uri request))
]
(wrap-cookies
(if host
(->
{:url (build-url host (path-fn (:uri request)) (:query-string request))
:method (:request-method request)
:body (if-let [len (get-in request [:headers "content-length"])]
(slurp-binary
(:body request)
(Integer/parseInt len)))
:headers stripped-headers
:follow-redirects true
:throw-exceptions false
:as :stream
:insecure? true
}
client/request
(update-in [:headers] dissoc "Transfer-Encoding")
prepare-cookies)
(handler request)))))))
(defn wrap-proxy [handler]
(-> handler
(create-proxy
{:identifier-fn :uri
:host-fn (fn [^String uri]
(cond
;; (.startsWith uri "/www") "http://localhost:6787/p/www/stockinfogate"
(.startsWith uri "/www") "https://m.joudou.com/p/www/stockinfogate"
(.startsWith uri "/beta") "https://m.joudou.com/p/beta/stockinfogate"
:else nil))
:path-fn (fn [^String uri]
(subs uri (inc (.indexOf (rest uri) \/)))
#_(s/join "/" (cons "" (drop 2 (s/split uri #"/")))))}))) |
|
I can't think of anything that should cut off the response in the middleware off top of my head. However, Jetty might have a default response size limit. One thing to try would be to swap the http server to immutant or http-kit if you're not relying on any jetty specific functionality. |
|
Immutant docs have an example of creating chunked responses under the |
|
@yogthos Ahh.. Thank you for you reply. The server of the original url is developed by my team, too. It is also served by jetty. Is it possible that the jetty of my original server can serve the large response (maybe 500KB) while the Luminus jetty cannot? |
|
You might have to tune it to server larger responses. Take a look here for the available options. Increasing the buffer size might do the trick. |
|
I found that sometimes it served one chunk (64kb) and sometimes two. Maybe always the last one chunk will get lost. I'll try and comment here later. Thank you for your kind. |
|
No problem, hopefully this does the trick. |
|
I tried to tune jetty but nothing changed. |
|
Hmm one thing I just noticed is you're doing (defn create-proxy [handler fns]
(let [identifier-fn (get fns :identifier-fn identity)
host-fn (get fns :host-fn {})
path-fn (get fns :path-fn identity)]
(fn [request]
(let [request-key (identifier-fn request)
host (host-fn request-key)
stripped-headers (dissoc (:headers request) "content-length" "host")
path (path-fn (:uri request))
]
(wrap-cookies
...)))))This might be causing the odd behavior. Internally, (fn
([request]
(-> request
(cookies-request options)
handler
(cookies-response options)))So you probably want: (defn create-proxy [handler fns]
(let [identifier-fn (get fns :identifier-fn identity)
host-fn (get fns :host-fn {})
path-fn (get fns :path-fn identity)]
(fn [request]
(let [request-key (identifier-fn request)
host (host-fn request-key)
stripped-headers (dissoc (:headers request) "content-length" "host")
path (path-fn (:uri request))
]
(-> (if host ... )
(cookies-request options)
handler
(cookies-response options)))))) |
|
Ahh.. Thank you very much! |
The url get response data cut:
http://localhost:3000/www/stock/klinedata?stockid=300370.SZ&period=D
the original url:
https://www.joudou.com/stockinfogate/stock/klinedata?stockid=300370.SZ&period=D
The error message are:
I use the following handler:
I add the
wrap-proxyto get response from another server.I tried to get the raw req and
client/requestthe req, then I got the full response in REPL. So I am sure that the response got by the function returned bywrap-proxyis the complete. Is this because of something in thewrap-basewho cause the response to be cut off?Thanks.
The text was updated successfully, but these errors were encountered: