Multi Threaded Plugins
- Adding and Deleting Files
- Multi-Threaded Plugins
- Accessing All Files
- Automatic Rebuilds
- Other Plugin APIs
CodeEngine uses worker threads to run faster by processing many files simultaneously across multiple threads.
Multi-threading can significantly improve build times by processing multiple files concurrently, but that doesn't mean it should be used for every plugin. There is some overhead involved in synchronizing multiple threads and transferring files back and forth between the main thread and worker threads, so you want to be sure that this overhead doesn't outweigh the benefits of concurrent processing.
Simple plugins (like the one in this tutorial) are usually not good candidates for multi-threading. Our "Append Footer" plugin only performs a single String.replace() call, which takes only a few milliseconds — less than the time it wold take to transfer a file from the main thread to a worker thread — so it wouldn't make sense to use multi-threading for this plugin.
As a general rule, multi-threading is a good fit for any plugin that performs non-trivial CPU bound processing. Examples include:
- Compressing or processing images
- Compiling or transpiling code
- Linting or validating files
Multi-threading is not a good fit for plugins that perform simple processing (less than 50ms) or processing that is primarily I/O bound or memory bound. Examples include:
- Writing files to the hard drive or network
- Loading a large dataset into memory
- Fetching data from a URL
- Calling APIs
To take advantage of multi-threading, just give CodeEngine the path of your processFile() script, rather than a function. You've already seen this technique in the first tutorial when we used a file path in the plugins array of our generator.
my-generator/index.ts
export default {
plugins: [
"./plugins/footer.ts"
]
};In that tutorial, the footer.ts file exported a function that processes files. By using the path of the footer.ts file rather than importing it and referencing its function in the generator, CodeEngine was able to load the plugin on its multiple worker threads and process many files simultaneously.
But what if your plugin exports a Plugin object, not just a file-processing function? We'll show you how to do handle that next.
Let's return to the plugin that we wrote in the previous tutorial and refactor it to support multi-threading. As explained above, this plugin is actually not a good fit for multi-threading, but for the purpose of this tutorial, let's just pretend that it's a more complicated plugin that warrants multi-threading. 😉
my-generator/plugins/process-file.ts
The first step is to move the processFile() method out of the footer.ts file and into its own file. It needs to be in its own file so we can give CodeEngine the path of that file. CodeEngine will then load the file in all of its worker threads.
NOTE: We've hard-coded the year to "2020" for now. We'll show you how to make this dynamic in the next step.
/**
* This is the implementation of the `processFile()` method
* of the Append Footer plugin.
*/
export default function processFile(file) {
file.text = file.text.replace(
"</body>",
`<footer>Copyright © 2020</footer></body>`
);
return file;
}my-generator/plugins/footer.ts
Now that the processFile() method has been moved to a separate file, you can set the processFile property of your plugin to the path of that file.
import path from "path";
/**
* This is a CodeEngine plugin that adds a copyright footer to all HTML pages.
*/
export default function appendFooter(options = {}) {
const year = options.year || new Date().getFullYear();
return {
name: "Append Footer",
filter: options.filter || "*.html",
processFile: path.join(__dirname, "process-file.ts")
};
}We've used path.join() and the Node.js __dirname variable to set our plugin's processFile property to the full path of the process-file.ts file. CodeEngine will automatically import this file into all of its worker threads.
Now run the code-engine command. You'll see that your plugin still works the same as before. The difference is that it's now running on multiple worker threads rather than the main thread.
Our plugin accepts options that allow users to customize the year shown in the copyright. But now that we've refactored the plugin, that option no longer works. The reason is that the appendFooter() function and its options argument exist in the main thread, not in the worker threads where our processFiles() function runs. We need a way to pass options from the main thread to the worker threads.
To pass options (or any data you want) to the worker threads, you need to set the processFile property to a ModuleDefinition object instead of just a string. Like this:
processFile: {
// The path of the JavaScript file
moduleId: "/path/to/my/script.js",
// Any data you want to pass
data: {
foo: "bar",
answer: 42,
}
}By setting the data property, you're telling CodeEngine to pass that data to the worker threads.
my-generator/plugins/footer.ts
Let's modify our plugin to pass the year option to the worker threads.
import { join } from "path";
/**
* This is a CodeEngine plugin that adds a copyright footer to all HTML pages.
*/
export default function appendFooter(options = {}) {
const year = options.year || new Date().getFullYear();
return {
name: "Append Footer",
filter: options.filter || "*.html",
processFile: {
moduleId: join(__dirname, "process-file.ts"),
data: { year }
}
};
}my-generator/plugins/process-file.ts
We also need modify worker thread script to accept the options and use the year option in the copyright.
export default ({ year }) =>
/**
* This is the implementation of the `processFile()` method
* the Append Footer plugin.
*/
function processFile(file) {
file.text = file.text.replace(
"</body>",
`<footer>Copyright © ${year}</footer></body>`
);
return file;
}And now if you run the code-engine command, you'll see that our plugin correctly uses the year option in the copyright footer.