Lint in parallel

[borkdude]

This can speed up linting when providing a large classpath.

We borrowed some code from ClojureScript parallel-build:

https://github.com/clojure/clojurescript/blob/a15247a743d4d1c5d73224038f7289c447b38ca8/src/main/clojure/cljs/closure.clj#L1072

Without parallel:

clj-kondo --lint $(lein classpath)
linting took 59392ms, errors: 2176, warnings: 3995

With --parallel:

clj-kondo --lint $(lein classpath) --parallel
linting took 35919ms, errors: 2176, warnings: 3995

• Mac binary: https://10632-176829714-gh.circle-artifacts.com/0/release/clj-kondo-2020.07.30-SNAPSHOT-macos-amd64.zip
• Linux binary: https://10633-176829714-gh.circle-artifacts.com/0/release/clj-kondo-2020.07.30-SNAPSHOT-linux-amd64.zip
• Windows binary: https://ci.appveyor.com/api/buildjobs/6kpaoq5rl8y3vxrj/artifacts/clj-kondo-2020.07.30-SNAPSHOT-windows-amd64.zip
• Maven version: "2020.07.30-20200828.181423-13"

TODO:

• improve contention on namespaces atom by using local atom in analyze-input and merging the contents with the main atom
• get rid of nested swap! where possible (e.g. calling reg-finding! inside clj-kondo.impl.namespace/reg-var!).

[sanel]

This can be already done via find and xargs or parallel, but clj-kondo needs to be compiled as binary for fast startup.

Assuming standard leiningen project with src and test folders:

# run 10 parallel processes
$ find src test -name "*.clj*" | xargs -n 1 -P 10 clj-kondo --lint

# or
$ find src test -name "*.clj*" | parallel -j 10 clj-kondo --lint

However, this requires --cache false to be used or it will start throwing exception:

java.lang.Exception: Clj-kondo cache is locked by other process.
	at clj_kondo.impl.cache$sync_cache.invokeStatic(cache.clj:124)
	at clj_kondo.core$run_BANG_.invokeStatic(core.clj:97)
	at clj_kondo.main$main.invokeStatic(main.clj:85)
	at clj_kondo.main$main.doInvoke(main.clj:78)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.core$apply.invokeStatic(core.clj:665)
	at clj_kondo.main$_main$fn__4926.invoke(main.clj:104)
	at clj_kondo.main$_main.invokeStatic(main.clj:104)
	at clj_kondo.main$_main.doInvoke(main.clj:102)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clj_kondo.main.main(Unknown Source)

Not sure if caching per process will make sense. After I disabled caching, speed was noticeable in multiple processes.