Some UTF-8 chars seem to break articles

[martinklepsch]

This article contains some UTF-8 characters that don't seem to go well with our code.

Need to look into it some more but likely the issue is somewhere here:

cljdoc/src/cljdoc/renderers/html.clj

Lines 97 to 104 in 422f463

	(let [doc-slug-path (:doc-slug-path route-params)
	doc-tree (doctree/add-slug-path (-> cache-contents :version :doc))
	doc-p (->> doc-tree
	doctree/flatten*
	(filter #(= doc-slug-path (:slug-path (:attrs %))))
	first)
	doc-html (or (some-> doc-p :attrs :cljdoc/markdown rich-text/markdown-to-html)
	(some-> doc-p :attrs :cljdoc/asciidoc rich-text/asciidoc-to-html))]

(presumably when trying to find the matching article inside the doctree)

[martinklepsch]

Seems that this is something with Pedestal. Pedestal reports the following URL for the GET request:

io.pedestal.http {:msg "GET /d/lambdaisland/kaocha/0.0-118/doc/01-kaocha-%E8%80%83%E5%AF%9F", :line 80}

but to correctly resolve the article we'd need something like this:

/d/lambdaisland/kaocha/0.0-118/doc/01-kaocha-考察

Seems there is a thing ServletRequest#setCharacterEncoding() to set the used character encoding for processing requests. Haven't yet confirmed this is the issue but at least Pedestal doesn't seem to be calling this setCharacterEncoding.

[plexus]

Seems like a regular case of URL percent encoding/decoding

(require '[lambdaisland.uri.normalize :as n])

(n/percent-decode "/d/lambdaisland/kaocha/0.0-118/doc/01-kaocha-%E8%80%83%E5%AF%9F")
;;=> "/d/lambdaisland/kaocha/0.0-118/doc/01-kaocha-考察"

lambdaisland.uri.normalize is unreleased though but the same functionality should be available in ring/pedestal and I think even in the Java standard libs.

Update:

java.net.URLDecoder.decode(url, "UTF-8");

[martinklepsch]

Thanks, I was kind of thinking this should be done by Pedestal but regardless of that we probably don't want to wait for an upstream release to get it fixed.
42c01c3 introduces a change that manually url-decodes the article slugs.

[martinklepsch]

Deployed and working now: https://cljdoc.xyz/d/lambdaisland/kaocha/0.0-118/doc/01-kaocha-考察

[martinklepsch]

Closing this issue, might bug Pedestal people about this but it's working for now 🚀

[martinklepsch]

Seems it is a known issue in Pedestal, introduced in a recent version: pedestal/pedestal#588