babel plugins

[kurtharriger]

I have some js files that require transform-class-properties
I added the plugin to .babelrc and babel will transpile them it, but shadow-cljs will not.

I updated the local-js example to demonstrate the issue:
https://github.com/thheller/shadow-cljs-examples/compare/master...kurtharriger:babelrc

This works:

babel src/demo/foo.js

But

shadow-cljs compile app

emits:

shadow-cljs - config: /Users/kharriger/projects/shadow-cljs-examples/local-js/shadow-cljs.edn version: 2.0.150
shadow-cljs - starting ...
[:app] Compiling ...
errors in file: /Users/kharriger/projects/shadow-cljs-examples/local-js/src/demo/foo.js
{:js-requires [], :js-imports [], :js-invalid-requires [], :js-language "es3", :js-str-offsets [], :js-errors [{:line 26, :column 11, :message "'(' expected"}], :js-warnings [], :tag :shadow.build.npm/errors}
ExceptionInfo: errors in file: /Users/kharriger/projects/shadow-cljs-examples/local-js/src/demo/foo.js
	clojure.core/ex-info (core.clj:4739)
	clojure.core/ex-info (core.clj:4739)
	shadow.build.npm/get-file-info* (npm.clj:527)
	shadow.build.npm/get-file-info* (npm.clj:435)
	shadow.build.npm/get-file-info (npm.clj:560)
	shadow.build.npm/get-file-info (npm.clj:557)
	shadow.build.npm/find-resource* (npm.clj:654)
	shadow.build.npm/find-resource* (npm.clj:614)

[thheller]

The problem is that currently files are first read by the Closure Comiler via JSInspector to extract metadata about them (eg. their require or import calls). If that inspector identifies the file as ES6 they will be piped through babel.

The Inspector can only read files that are "valid" "ECMASCRIPT_NEXT" according to the closure compiler.

I need to add support for processing the files FIRST anyways to support .jsx and other dialect but I haven't gotten around to doing that yet.

I'm not yet sure how that will work but it'll probably be similar to https://clojurescript.org/news/2017-07-20-js-preprocessing-improvements. Ideally I want to keep the post processing out of the compilation itself since it might not be that easy to integrate other tools like typescript, coffeescript, flow, etc.

[kurtharriger]

This isn't the only plugin I we use just the easiest to demo.

Fully qualified symbols seems reasonable for custom preprocessing steps, but I'm wondering if perhaps it could be done now with Custom Builds. I'm also thinking instead of custom builds it might be even better to provide a middleware pipeline around the target steps instead of redefining and delegating to the target. Maybe this middleware does preprocessing, or maybe something different. In either case, preprocessing js would make for a much more interesting custom build example.

I currently transpile the js first and import from the transpiled code. I was thinking perhaps it would be possible to transpile and then update the source path from a custom build so that I don't need to reference the transpiled path, but this might be less trivial then just calling out to babel before each rebuild. I haven't had time to dig to deeply into the current implementations, but the example doesn't give much insight into the structure of the state object. Is there any existing docs around its structure and/or a spec for it?

[thheller]

No Custom Builds would not be a solution. They only allow you to customize how the output is organized and can only control which resources get into a build not how.

The issue I have with fully qualified symbols for custom processing is that they don't capture lifecycle. Take the cljsjs/babel-standalone preprocess as an example. It starts a nashorn JS engine via delay and will keep that running indefinitely once started. The only way to talk to it is by giving it a file to process. It is using a fairly limited version of babel since it is not running via node, so you can't do any plugins or control which babel version is used. It is fine to keep the engine running in this case but when dealing with external node processes that is not ok.

The rough outline for my current plan looks like this:

JS resources are currently always resolved via files and relative paths between each other because this is how npm/node work. They do not have the concept of a classpath and cannot follow into .jar files.

CLJS resources are always addressed via the classpath. They cannot "break out" of the classpath. Names must be unique.

So we create a second kind of JS resource that follows the classpath rules and can use relative and absolute addressing. (:require ["/my-company/components/foo.js"]) always works on the classpath. In node this wouldn't work since it would look at the FS. But node people never use absolute paths anyways so thats fine. Relative requires work as well with the limitation that you can't go past the root (eg. ../../../../../something.js) which is desirable anyways. Similary to npm resolve rules the way to break out of the classpath is to use the ambiguous require form (eg. import ... from "react").

So technically each file then has a unique name. You could then create a :source-path for pre-processed JS sources. Say you have src/js/my-company/components/foo.jsx which we can't read directly. You run an external processor like babel to generate the src/gen/my-company/components/foo.js where src/gen is part of the classpath. All namespaces can access this by either the absolute path or relative between each other. When publishing as a jar you include the pre-processed file so "clients" don't need to do any pre-processing themselves. In essence the same deal you get with npm.

I think this would provide the best of both worlds where JS can come from either npm for ambiguous requires or the classpath (ie. jar) for everything else.

With those resolve rules in place shadow-cljs wouldn't need to do anything for pre-processing but could at a later date provide automatic integration with things like babel so you don't have to run them manually. Until then a simple npx babel src/js --watch --out-dir src/gen would do.

The real hurdle would be when the pre-process stuff would attempt to resolve require/import itself but those tools probably has some kind of plugin mechanism to allow hooking into resolve.

[kurtharriger]

I don't think I'm following on the lifecycle bit, I thought it was just using namespaced keywords to reference functions without them necessarily being loaded yet. That doesn't mean they need to have the same signature as those in cljsjs/babel-standalone. The function could be invoked multiple times with a state object to participate in multiple lifecycle stages in the same way custom targets do.

To be more specific I thought I would try to hack something up in the example project that injects arbitrary middleware around the target function to preform pre/post processing at each stage. This is what came up with: https://github.com/thheller/shadow-cljs-examples/compare/master...kurtharriger:middlware

I created my own custom target that reads a list of middleware and the original desired shadow-target from the config. I then wrap a couple middleware methods around it, one that dumps the state for each stage to an edn file before calling the target function and another middleware function that attempts to babelize a source directory.

(After writing resolve-keyword I realized edn had no issue using symbol for target so why not just use symbols instead of keywords, unless the goal is to decouple from implementation, but then maybe they shouldn't be namespaced... but I digress).

My main goal was to see if I could create a custom target that creates a middleware chain to do pre/post processing in each stage in a target agnostic way and and then use the middleware to transpile babel code.

I was able to create the middleware chain as I wanted and I was able to call babel during the :configure stage (note docs say :init not :configure) to transpile the src folder... but I haven't quite digested where/what state needs to be updated when to consume it from the new location.

Maybe there isn't a good way of doing it yet , but I also noticed in the state there is a babel service and I'm seriously wondering if I could hijack the in and out channel from the middleware during the configure stage and startup a new babel service that uses a different config?

Hopefully this implementation is useful to you and gets some ideas brewing.

[thheller]

I wish it was this simple.

There are a few flaws with your approach.

• Multiple builds would each do the babel work although once would be enough
• Modifying the source files (babel input files) does not trigger a re-compile in watch.
• :compile-prepare is not the correct stage as you already noticed
• shelling out is too slow

:target really is not meant to provide new input forms. Resolving the correct sources with reliable caching is already hard enough without targets messing around in there.

By lifecycle I was referring to the lifecycle of the individual server components. Right now the internal babel process is started once per server and shared among all builds. It is properly stopped on server shutdown which is important for server/stop! in embedded mode. Don't want to leak node processes. In addition to the JVM lifecycle itself there is a lifecycle per worker, ie. when a worker is started and stopped. None of this is currently available to :target fns (on purpose).

I have a plan for plugins but nothing concrete yet.

BTW the babel stuff you could use via (shadow.build.babel/convert-source babel-from-state state-itself "some JS source text" "the/source/name.js"). It'll return the processed code. No need accessing the channels manually.

Don't worry though the plan I outlined above should take care of everything easily.

[kurtharriger]

My thinking was that I needed to intercept and replace the current babel service because that plugins are hardcoded here https://github.com/thheller/shadow-cljs/blob/master/src/main/shadow/cljs/npm/transform.cljs#L25.
Thus if you want to use different configurations for different files you can't use a shared service (I think), so I'm not sure how using convert-source is useful unless the configuration can also be specified.

I guess in part I don't understand the role babel currently plays in shadow-cljs and perhaps more so the closure compiler. You mentioned earlier that the transforms need to be done before closure compiler because of a limitation in JSInspector, but if closure then is handling the transpile then when does shadow-cljs use babel or if shadow-cljs is running babel then why not run it sooner?

I agree that shelling out isn't the most performant approach, but I would think that the middlware could start a service and add it to the state in a namespaced keyword for reuse in future phases (there probably should also be a cleanup stage in the pipeline to allow for middleware to properly shutdown).

Webpack uses this concept of loaders that are chained together to form a pipeline for all files that match a given rule. I think webpack just passes the contents of the file as a strings through each loader to parse and rewrite rather than structured data, but it seems that it is sufficient and loaders share resources and accumulate state via plugins.

Unlike shadow-cljs however I don't think webpack ever needs to flush everything back to disk when running a loader as they are provided the file contents. The problem with writing back to disk to run another step such as the closure compiler is that unless you overwrite the source files the paths change and all the import/requires need to be rewritten as well.

I'm wondering if perhaps it would be easier to copy everything into a virtual filesystem so that transpiled code can be written to the exact same path as the source file. I think gulp does something like this.

[thheller]

babel plugins are hardcoded but .babelrc is supported and you can specfiy additional plugins there.

shadow-cljs uses babel internally to compile ES6 -> CJS for node_modules files. Can't use the Closure Compiler for that since it was too unreliable to rewriting too aggressively.

shadow-cljs basically already has a virtual filesystem and the classpath is one too. CLJS style pre-processing with a symbol are basically like webpack loaders.

The problem I have with loaders is the .jar situation. Say you write a CLJS library but want to write some React Components in JSX for whatever reason. You publish the library with the .jsx files. Now everyone wanting to consume this library must configure the loader to be able to use your library. That is nonsense and the .jar should only contain the file after JSX processing (just like npm).

If we don't do the loader but instead do the JSX conversion entirely independently it is much easier to integrate and has to deal with way less moving parts.

I'll have the solution ready soon, then we'll see how it compares.

[thheller]

Just pushed the implementation of my plan above. No release yet since it needs more testing. clone and lein install if you want to test.

Relative and absolute requires in CLJS are now resolved on the classpath which means (:require ["/demo/myComponent"]) or (ns demo.thing (:require ["./myComponent"])) now work properly even if the files are in a .jar. Previously that did not work at all.

Classpath JS is also transformed by Closure so you get the full DCE experience.

Currently only the loading part is implemented which means you need to run external tools manually but a simple babel watch src/js -d src/gen worked perfectly fine during testing.

Running those tools via shadow-cljs will be enabled by plugins once I can decide on a proper plugin API.

[thheller]

I pushed 2.1.0 earlier today which has all the changes I described.

Documented here: https://shadow-cljs.github.io/docs/UsersGuide.html#_dealing_with_js_files

The next step will be to do proper integration with babel etc so you don't have to run them manually, Not sure when I'll get to that though.

[alex-dixon]

babel example here using the babel-cli command to watch a directory