Skip to content
Permalink
Browse files

Add options for how to flatten/encode form and query parameters (#433)

* Add options for how to flatten/encode form and query parameters

This adds the following new parameters:

- `:ignore-nested-query-string` :: Do not handle nested query parameters specially, treat them as
     the exact text they come in as. Defaults to *false*.
- `:flatten-nested-form-params` :: Flatten nested (map within a map) `:form-params` before encoding
     it as the body. Defaults to *false*, meaning form params are encoded only
     `x-www-form-urlencoded`.
- `:flatten-nested-keys` :: An advanced way of specifying which keys having nested maps should be
     flattened. A middleware function checks the previous two options
     (`:ignore-nested-query-string` and `:flatten-nested-form-params`) and modifies this to be the
     list that will be flattened.

Resolves #427

* Add a test for the new middleware

* Throw IllegalArgumentException when multiple options are specified
  • Loading branch information
dakrone committed Feb 13, 2018
1 parent 29116c5 commit 47a7762ed42e1d772e51a6f5bdaa61b436b54cb3
Showing with 129 additions and 22 deletions.
  1. +13 −0 README.org
  2. +5 −2 changelog.org
  3. +31 −4 src/clj_http/client.clj
  4. +80 −16 test/clj_http/test/client_test.clj
@@ -339,6 +339,19 @@ content encodings.
(if (> try-count 4) false true))})
#+END_SRC

A word about flattening nested =:query-params= and =:form-params= maps. There are essentially three
different ways to handle flattening them:

- =:ignore-nested-query-string= :: Do not handle nested query parameters specially, treat them as
the exact text they come in as. Defaults to *false*.
- =:flatten-nested-form-params= :: Flatten nested (map within a map) =:form-params= before encoding
it as the body. Defaults to *false*, meaning form params are encoded only
=x-www-form-urlencoded=.
- =:flatten-nested-keys= :: An advanced way of specifying which keys having nested maps should be
flattened. A middleware function checks the previous two options
(=:ignore-nested-query-string= and =:flatten-nested-form-params=) and modifies this to be the
list that will be flattened.

** DELETE
:PROPERTIES:
:CUSTOM_ID: h-c7165d6b-232a-439d-9390-8c05e6ef1e6f
@@ -13,14 +13,17 @@ List of user-visible changes that have gone into each release
** 4.0.0 (Unreleased)
- Removed slingshot dependency. clj-http exceptions are now regular exceptions
- Removed deprecated :follow-redirects & :force-redirects options
- Wrap nested querystring params before form params, fixing
https://github.com/dakrone/clj-http/issues/427
- +Wrap nested querystring params before form params, fixing
https://github.com/dakrone/clj-http/issues/427+ Reverted, see further below
- Merged https://github.com/dakrone/clj-http/pull/426 to allow an empty SSLGenericSocketFactory
context
- Merged https://github.com/dakrone/clj-http/pull/424 to add :mime-subtype request parameter to
override mime subtype
- create-multipart-entity with three arguments arity lets the selection of =HttpMultipartMode=
- new request key :http-multipart-mode which is HttpMultipartMode/STRICT by default
- Added =:ignore-nested-query-string=, =:flatten-nested-form-params=, and =:flatten-nested-keys=
options for finer-grained control over which nested parts of the request are flattened. Fixes
https://github.com/dakrone/clj-http/issues/427

** 3.8.0
- Reintroduce the =:save-request= and =:debug-body= options
@@ -863,13 +863,12 @@
request))

(defn- nest-params-request
[{:keys [content-type] :as req}]
(if (or (nil? content-type)
(= content-type :x-www-form-urlencoded))
[{:keys [flatten-nested-keys] :as req}]
(if (seq flatten-nested-keys)
(reduce
nest-params
req
[:query-params :form-params])
flatten-nested-keys)
req))

(defn wrap-nested-params
@@ -881,6 +880,33 @@
([req respond raise]
(client (nest-params-request req) respond raise))))

(defn- nested-keys-to-flatten
[{:keys [flatten-nested-keys] :as req}]
(when (and (not (nil? (opt req :ignore-nested-query-string)))

This comment has been minimized.

Copy link
@kachayev

kachayev Feb 13, 2018

Contributor

@dakrone Shouldn't this be something like

(when (and (some? flatten-nested-keys)
           (or (some? (opt req :ignore-nested-query-string))
               (some? (opt req :flatten-nested-form-params))))

Because right now it throws only when both :ignore-nested-query-string and :flatten-nested-form-params specified along side with :flatten-nested-keys.

This comment has been minimized.

Copy link
@dakrone

dakrone Feb 14, 2018

Author Owner

Good catch, I've pushed a commit to fix this, thanks!

This comment has been minimized.

Copy link
@kachayev
(not (nil? (opt req :flatten-nested-form-params)))
flatten-nested-keys)
(throw (IllegalArgumentException.
(str "only :flatten-nested-keys or :ignore-nested-query-string/"
":flatten-nested-keys may be specified, not both"))))
(let [iqs-key (when-not (opt req :ignore-nested-query-string) :query-params)
ifp-key (when (opt req :flatten-nested-form-params) :form-params)]
(or flatten-nested-keys
(remove nil? (list iqs-key ifp-key)))))

(defn wrap-flatten-nested-params
"Middleware wrapping options for whether or not to flatten `:query-params` and
`:form-params`. Modifies the request by adding a `:flatten-nested-keys`
sequence of the nested keys that will be flattened."
[client]
(fn
([req]
(client
(assoc req :flatten-nested-keys (nested-keys-to-flatten req))))
([req respond raise]
(client
(assoc req :flatten-nested-keys (nested-keys-to-flatten req))
respond raise))))

(defn- url-request
[req]
(if-let [url (:url req)]
@@ -1014,6 +1040,7 @@
wrap-content-type
wrap-form-params
wrap-nested-params
wrap-flatten-nested-params
wrap-method
wrap-cookies
wrap-links
@@ -52,7 +52,7 @@
(is (= 200 (:status resp)))
(is (= "close" (get-in resp [:headers "connection"])))
(is (= "get" (:body resp))))
(let [params {:a "1" :b {:c "2"}}]
(let [params {:a "1" :b "2"}]
(doseq [[content-type read-fn]
[[nil (comp parse-form-params slurp)]
[:x-www-form-urlencoded (comp parse-form-params slurp)]
@@ -66,7 +66,8 @@
:form-params params})]
(is (= 200 (:status resp)))
(is (= "close" (get-in resp [:headers "connection"])))
(is (= params (read-fn (:body resp))))))))
(is (= params (read-fn (:body resp)))
(str "failed with content-type [" content-type "]"))))))

(deftest ^:integration roundtrip-async
(run-server)
@@ -90,7 +91,7 @@
(is (= "get" (:body @resp)))
(is (not (realized? exception))))

(let [params {:a "1" :b {:c "2"}}]
(let [params {:a "1" :b "2"}]
(doseq [[content-type read-fn]
[[nil (comp parse-form-params slurp)]
[:x-www-form-urlencoded (comp parse-form-params slurp)]
@@ -103,6 +104,7 @@
:as :stream
:method :post
:content-type content-type
:flatten-nested-keys []
:form-params params
:async? true} resp exception)]
(is (= 200 (:status @resp)))
@@ -163,11 +165,11 @@
(let [client (fn [req] {:status 500})
e-client (client/wrap-exceptions client)]
(try
(e-client {})
(catch Exception e
(if (= :clj-http.client/unexceptional-status (:type (ex-data e)))
(is true)
(is false ":type selector was not caught."))))))
(e-client {})
(catch Exception e
(if (= :clj-http.client/unexceptional-status (:type (ex-data e)))
(is true)
(is false ":type selector was not caught."))))))

(deftest throw-on-exceptional-async
(let [client (fn [req respond raise]
@@ -814,13 +816,38 @@

(deftest apply-on-nested-params
(testing "nested parameter maps"
(are [in out] (is-applied client/wrap-nested-params
{:query-params in :form-params in}
{:query-params out :form-params out})
{"foo" "bar"} {"foo" "bar"}
{"x" {"y" "z"}} {"x[y]" "z"}
{"a" {"b" {"c" "d"}}} {"a[b][c]" "d"}
{"a" "b", "c" "d"} {"a" "b", "c" "d"}))
(is-applied (comp client/wrap-form-params
client/wrap-nested-params)
{:query-params {"foo" "bar"}
:form-params {"foo" "bar"}
:flatten-nested-keys [:query-params :form-params]}
{:query-params {"foo" "bar"}
:form-params {"foo" "bar"}
:flatten-nested-keys [:query-params :form-params]})
(is-applied (comp client/wrap-form-params
client/wrap-nested-params)
{:query-params {"x" {"y" "z"}}
:form-params {"x" {"y" "z"}}
:flatten-nested-keys [:query-params]}
{:query-params {"x[y]" "z"}
:form-params {"x" {"y" "z"}}
:flatten-nested-keys [:query-params]})
(is-applied (comp client/wrap-form-params
client/wrap-nested-params)
{:query-params {"a" {"b" {"c" "d"}}}
:form-params {"a" {"b" {"c" "d"}}}
:flatten-nested-keys [:form-params]}
{:query-params {"a" {"b" {"c" "d"}}}
:form-params {"a[b][c]" "d"}
:flatten-nested-keys [:form-params]})
(is-applied (comp client/wrap-form-params
client/wrap-nested-params)
{:query-params {"a" {"b" {"c" "d"}}}
:form-params {"a" {"b" {"c" "d"}}}
:flatten-nested-keys [:query-params :form-params]}
{:query-params {"a[b][c]" "d"}
:form-params {"a[b][c]" "d"}
:flatten-nested-keys [:query-params :form-params]}))

(testing "not creating empty param maps"
(is-applied client/wrap-query-params {} {})))
@@ -894,7 +921,7 @@
(fn [req] {:body nil})) {:decode-body-headers true})
resp4 ((client/wrap-additional-header-parsing
(fn [req] {:headers {"content-type" "application/pdf"}
:body (.getBytes text)}))
:body (.getBytes text)}))
{:decode-body-headers true})]
(is (= {"content-type" "text/html; charset=Shift_JIS"
"content-style-type" "text/css"
@@ -1234,3 +1261,40 @@
(is (= 200 (:status resp)))
(is (.contains query-string "a[]=1&a[]=2&a[]=3") query-string)
(is (.contains query-string "b[]=x&b[]=y&b[]=z") query-string))))

(deftest t-wrap-flatten-nested-params
(is-applied client/wrap-flatten-nested-params
{}
{:flatten-nested-keys [:query-params]})
(is-applied client/wrap-flatten-nested-params
{:flatten-nested-keys []}
{:flatten-nested-keys []})
(is-applied client/wrap-flatten-nested-params
{:flatten-nested-keys [:foo]}
{:flatten-nested-keys [:foo]})
(is-applied client/wrap-flatten-nested-params
{:ignore-nested-query-string true}
{:ignore-nested-query-string true
:flatten-nested-keys []})
(is-applied client/wrap-flatten-nested-params
{}
{:flatten-nested-keys '(:query-params)})
(is-applied client/wrap-flatten-nested-params
{:flatten-nested-form-params true}
{:flatten-nested-form-params true
:flatten-nested-keys '(:query-params :form-params)})
(is-applied client/wrap-flatten-nested-params
{:flatten-nested-form-params true
:ignore-nested-query-string true}
{:ignore-nested-query-string true
:flatten-nested-form-params true
:flatten-nested-keys '(:form-params)})
(try
((client/wrap-flatten-nested-params identity)
{:flatten-nested-form-params true
:ignore-nested-query-string true
:flatten-nested-keys [:thing :bar]})
(catch IllegalArgumentException e
(is (= (.getMessage e)
(str "only :flatten-nested-keys or :ignore-nested-query-string/"
":flatten-nested-keys may be specified, not both"))))))

0 comments on commit 47a7762

Please sign in to comment.
You can’t perform that action at this time.