Tasks composition without gulp

[Reinmar]

A followup of #66.

Down the rabbit hole...

I'm a framework/library author (namely, CKEditor). We need to provide a set of dev tools such as compilers for different types of assets, watchers, test runners, bundlers, scaffolders, etc.

For us, the library is the thing we work on, but for the community it's just a small piece of their projects. This changes quite a lot.

• We're using gulp as a task runner, they may not.
• We need 20 tasks for all the processes we use, they may need 5 of those and 3 different ones which are a different composition of the granular sub-tasks which we expose.
• We can pass some args to the tasks on CLI, they may want to hardcode them.
• And last but not the least – we're perfectionists. We want to have a control e.g. over what gets logged – to give an example – I don't want to register every sub-task to gulp, because a longer process would log a lot of unnecessary things.

After a few failed attempts to organise this code we decided to simply expose a set of functions. Those functions represents granular sub-tasks, accept options as parameters and, depending on what they do, they can be synchronous or return a stream or a promise. Sounds super obvious, I know :D.

What happens now? We use gulp, so our gulpfile.js looked like this:

const compiler = require( '@ckeditor/ckeditor5-dev-compiler' );

gulp.task( 'compile:js', () => compiler.compileJS( { format: 'esnext' } ) ); // Let's say this is a stream.
gulp.task( 'compile:sass', () => compiler.compileSass( sassOpts ) ); // This returns a promise.
gulp.task( 'compile:icons', () => compiler.compileIcons( iconsOpts ) ); // And this is synchronous

gulp.task( 'compile', [ 'compile:js', 'compile:sass', 'compile:icons' ] );

Now, there are couple of things which may happen (actually, they all happened :D).

Our gulpfile grew so we wanted to combine some sub-tasks into functions representing bigger chunks (e.g. compiler.compile() should run internally compiler.compileJS(), compiler.compileSass() and compiler.compileIcons()).

Exposing the popular tasks as bigger chunks would also simplify the community's gulpfiles which is an important thing if you want a good DX. Instead of the 4 tasks above, you get now:

const compiler = require( '@ckeditor/ckeditor5-dev-compiler' );

// Mind also the changed name – for the community it's not anymore "compile".
// Compiling CKEditor is just a sub-step of "compile my project".
gulp.task( 'compile:ckeditor', compiler.compile( { all options here } ) );

What also happened was that a compiler for tests may need different options than a compiler for documentation builder or for a specific release. So at some point we had something like:

gulp.task( 'compile:js', () => compiler.compileJS( getOptsFromCLI() ) );
gulp.task( 'compile:js:esnext', () => compiler.compileJS( { format: 'esnext'} ) );
gulp.task( 'compile:js:amd', () => compiler.compileJS( { format: 'amd' } ) );

gulp.task( 'test', [ 'compile:js:amd' ], () => tests.run() );
gulp.task( 'test', [ 'compile:js:esnext' ], () => docsBuilder.build() );

Horror... :D

What also happened was that we wanted to postpone certain require() calls, because combination of all possible dev tasks was loading a ridiculously huge number of dependencies. This complicated the gulpfile even more.

And then a developer comes to you and says that they are not using gulp at all in their project and they don't want it as a dependency, so how to combine now the compilation process with linting and testing.

Way out

At some point we realised how we'd like to solve all this. We needed a tool to sequence or run in parallel the sub-tasks so we can easily expose the less granular ones (like compile()) and that we can tell community how to run our code when they are not using gulp. Also, by registering only the end point tasks, and not the intermediate ones, we're able to keep the gulp -T short and prevent from logging too many things on the output.

After some research we found undertaker. "Task registry that allows composition through series/parallel methods." sounds for me exactly like what I wrote above :D Just, you said it shouldn't.

Anyway, this is how we use it now: https://github.com/ckeditor/ckeditor5-dev-compiler/blob/93d521e2e1b0e63fbf62133c1dff9ca9518a51f1/lib/tasks.js#L303-L330.

I hope that this better explains what we want to achieve (and what we don't want to do) and how undertaker could help.

[Reinmar]

Haha, I've just found https://github.com/gulpjs/bach in your pinned repos, @phated :D So is the tl;dr – use bach instead of undertaker? :D

[phated]

@Reinmar I think using bach might be better for your use case because it doesn't seem like you are benefiting at all from your use of undertaker (note: undertaker can take plain functions instead of registered task strings and would act similar to bach). If you just made the "tasks" named functions and passed them to bach.series, you'd end up in pretty much the same place without the undertaker overhead.

[phated]

Btw, thanks for opening this deep-dive. I love learning about other use cases for these libraries.

[phated]

@Reinmar is there anything further to discuss here? Otherwise I'd like to close this.

[phated]

Going to close this due to inactivity.