Plugin system

[Drew-S]

I created a plugin system: plugin. I have not done a pull request yet, as the plugin directory location is arbitrary, it is inside the relaxed src directory. I want to set a user configuration directory for things like plugins (~/.relaxed/plugins), global templates, global configurations, etc. But before I do that I would like for some feedback on the plugin system. Plugins are loaded from src/plugins/<plugin>/index.js and are formatted gist. This system was made for my bibliography system so that it does not run during the puppeteer runtime, as per @Zulko's issue with my current system #59. This also provides a system for other plugins in the future. It consists of three parts, pre-pug processing: where the raw pug string is given (main file only), post-pug processing: where the raw compiled html is given (before saving), and mixin: where any plugin mixins are added along with the default mixins.

[Zulko]

Definitely a good direction, and great code to my non-js-expert eyes (is this a common approach to plugins in JS ?). The separation pre pug/post html actions is spot on.

This being said I would be for waiting a bit to be sure to get the best architecture possible for plugins. Right now I would suggest we add important features (and bibliography certainly is one) in the core code, and take our time, until we can look at a few of these features and ask "what is the plugin architecture that can take them all out of the code". Right now I can see the MathJax filter, the bibliography, and custom mixins, but that's just after a few weeks, let's wait a bit.

My main feature requests for plugins at this point would be:

• Plugins should be npm-installable. Like the jstransformer plugins, you just npm i -g some-relaxed-plugin, and boom you can use it in ReLaXed.
• Even when a plugin is installed, it should not be used by default, but be activated, for instance at the beginning of your master.pug:

// use-plugin: mathjax
// use-plugin: bibliography style=someStyle

That's what I am doing with the MathJax plugin (which used to be applied on every document, and considerably slowed down every rendering, even for documents which didn't need any mathjax)

[Drew-S]

I am not entirely sure if my method of JS plugins is common, I have seen it in a few locations before. Your suggestion of // use-plugin: would prevent loading too much at once and allow for faster rendering, I can see how to program this in.

As for the bibliography I do know how I can program it in directly, when I was programming the plugins, I built the bibliography as a plugin, I can easily adapt this a built in system. But, more importantly, I used jsdom to do the bibliography and I think that could be used to do similar things such as a ToC, Figure list, Indexing, etc. It may be possible with jsdom to do page number referencing as well, but I do not think it can be done easily, it probably would require calculating the elements pixel size and comparing that to page size to manually count up. Just did some testing, jsdom does not render a document and so everything is located at (0,0), so we cannot calculate position this way.

[Zulko]

I believe these lines are how Pug looks for filters that are provided by plugins.

It would work for us, relaxed plugins would be npm libraries called relaxed-mathjax, relaxed-bibiography, etc. The user would specify which plugins they want to use

\\ use-plugin: mathjax
\\ use-plugin: bibliography [options]

Then the plugins will be resolved in Node using require.resolve('relaxed-' + pluginName) and they would have a structure like the one you outlined here (but let's wait a bit for that, your latest development showed that some plugins should be "post-page-load-in-chrome", lets wait if other rendering life-cycle hooks should be allowed too).

I am also thinking that instead of using \\ use-plugin: we could simply have a JSON file plugins.json:

{
  mathjax: true,
  bibliography: {
    style: 'apa-citation'
  }

This would make it much easier to have plugins with options.
I am also thinking that my different watchers (for making diagrams etc.) could become plugins too, which would make this chained condition more elegant and take most of the converters.js code out of the main repo.
Instead of JSON we could also use YAML which is more user-friendly.

So we already have several hooks that plugins should consider:

• Mixins to add to pug
• Special file extensions to watch and transform on change
• HTML transformations
• Transformations after page loaded by chrome

Again I am in no hurry to do that before ReLaXed has a good userbase.

[Drew-S]

So, as I worked on the table of contents I was thinking about how @Zulko mentioned it will shift content down and that needs to be taken into account. The same issue would happen for an index list, which would cause the same problem. So I was thinking of the plugin structure and the hooks we would need.

For the hooks I would think we need:

• mixins -- The plugin provides additional mixins
• pre-pug -- The plugin gets the raw pug files and scss files (might not need)
• post-pug -- The plugin gets the raw html string (might not need)
• page-first-pass -- The plugin gets the page to modify
• page-second-pass -- The plugin gets the page to modify

Because the page has the power to modify more than the pugs, I do not think that pre-pug or post-pug would be used all that often, in fact, we may want to omit those altogether, although, they may still be needed for a plugin to capture some data, though we can solve that data capture and provide the data to the plugin somehow.

The reason we would need two page passes, is so that the bibliography can generate in the first pass, the ToC, index, and figure list can generate without page numbers. And the second pass the ToC, index, and figure list can fill in the missing page numbers.

As for the plugin settings and enabling I think we should provide both the option to use a config.json (or yaml, ini, what ever is best) as well as the comments at the top //- use-plugin: x setting and //- set: variable this. Personally, coming from LaTeX I would prefer to use //- comments, providing the option I think is the best. We default to using a config file if provided, and if it is provided we ignore the comments. (save compute time on the regex).

[Drew-S]

I have done a fair bit of work on a plugin system currently, it will have a different structure than what we first did. If you want to check it out here. It has tonnes of work left to do, but it has plenty of hooks, full async, all type checked and error handling present, mostly documented (as I have been going).

Currently, I am working on the way it loads the plugins, either from a json file (or other config type later), and code comments in the master file. There are ways of capturing the plugin names, settings, and dependencies (other plugins). I am stopping for the night. Later on I will need to write up a tree to handle plugin dependencies and control load order of plugins, plugins are still going to load asynchronously.

The hooks are/will be:

• watcher -- File extensions to watch for
• mixin -- Pug mixin code
• prePug -- Raw pug files (before render)
• postPug -- Raw html string
• pageFirstPass -- Page content placeholder generation
• pageSecondPass -- Page content fill in

Plugins, along with using their own npm packages, can depend upon relaxed-plugin packages, set variables, see loaded plugins and communicate between them (not yet implemented).

[Zulko]

EDIT: just edited a bit. sorry for the noise.

Awesome. Some remarks.

• I believe postPug should be called htmlFilter or htmlTransformer or simply html, as it really modifies html, made with to pug or not.

• For prePug, I am not sure (1) what the use cases would be that wouldn't be covered by mixins and in-pug variables, and (2) about the implementation. Maybe it could be left out in a first step. One problem I see is that my master document is generally a very small file which just includes other pug files. The pug renderer is responsible for both rendering pug to HTML and compiling the different pug sources together. So it's not like you can load "all the pug" across all files, apply a filter, and render with PugJS. The best you can do (at least in a straightforward way) is load the master file (and only this one), change its content (like add mixins) and compile.

• For watchers, which are going to replace these lines, there should be a "precedence" system bases on the inclusion of extensions, from most specific to less. For instance a .ext1.ext2 watcher should take precedence and shortcut a .ext2 watcher.

• I'm not sure I understand the sentence "Plugins (...) can depend upon relaxed-plugin packages", what is the difference between plugin and relaxed-plugin ? The rest of the sentence, "set variables, see loaded plugins and communicate between them" seems a bit complex (I'd prefer baby steps when we can) and I am not completely sure what problems it solves (possibly related to what you needed for the bibliography ?)

• Very importantly, plugins should accept variables. Actually, what a npm package will provide in the first place is not a plugin object, but a plugin constructor. The most important thing this will allow is to have several plugins delivered in a single npm package. For instance mathjax({everywhere: true}) and mathjax('local-only') may be two very different plugins, but delivered by a single npm

The implementation I had in mind was this. First the json in which the user defines the plugins (the structure is so it will look much simpler in YAML):

[
 [
    "mathjax",
    {"everywhere": true}
  ],
  ...
]

Then here is how the plugins are loaded. They should be reloaded every time the file "plugins.json" changes:

pluginsList = require('plugins.json')

// registeredPlugins is not a list of plugins, but has separate lists for mixin, html, etc.
registeredPlugins  = { mixins: [], html: [], page1st: [], etc. }
for entry in pluginsList:
    [name, options] = entry
    pluginConstructor = require('relaxed-' + name) // here, relaxed-mathjax.
    // other resolution for local plugins if 
    plugin = pluginConstructor(options) // here options: {"everywhere": true}
    if plugin.mixins:
      registeredPlugins.mixins.push(mixin)
    if plugin.html:
      registeredPlugins.html.push(html)

// now reorder the internal registeredPlugins.mixins, registeredPlugins.html, etc.

Then in the ReLaXed code, it will be as simple as

var html = ... // render html
registeredPlugins.html.each(function (transformer) {
    html = transformer(html)
})

These are just thoughts, I'm open to other ideas. Let me know if it makes sense and if I understood it correctly.

[Drew-S]

Sorry, I was heading off to bed when I wrote that, probably not the best time to explain the plugin dependency. I was imagining that a relaxed-plugin could require another relaxed-plugin in order to operate. Like something requiring the bibliography to render before it renders, they kind of dependency, and this type of plugin could not simply require the npm package cause it would operate differently. This dependency, and how I am designing it, is that variables can be exposed by plugins for other plugins, again, imagine another plugin asking for the bibliography's cite object to get some data.

Ill have to make up an example of how I see the plugins being initialized and their operations run. That will give a better understanding of the structure.

plugin/index.js

// private variables, functions, etc. Not exported
const name = 'plugin_example'
exports.activate = async function(plugin) {
  await plugin.registerMixin({
    plugin: name, // I probably will come up with a solution to omit this
    mixin: path.join(__dirname, 'mixins.pug')
  })

  await plugin.registerWatcher({
    plugin: name,
    ext: ['.test', '.test2'],
    handler: async function(file) { ... }
  })

  await plugin.expose({
    some_option: 'thing',
    some_other_option: 'thing 2'
  })
}

exports.deactivate = async function(plugin) {
  // In case some sort of tear down is needed after pdf is rendered
}

// Optionally, instead of plugin.expose we could do this, I prefer the plugin.register method myself
exports.options = {}
exports.mixin = path.join(__dirname, 'mixins.pug')

[Drew-S]

I just updated the plugin system, it should be operational now, It is integrated, and has gone through 1 very very limited test. I need to create an actual npm module for relaxed to test it out fully.

[Zulko]

Sure, what about the relaxed-bibliography plugin for a start ?

[DanielRuf]

What about npm link? And npm can use local packages.

[Drew-S]

@Zulko I was actually going to make a relaxed-example plugin that does an example of all the features, this way it can also act as a boiler-plate for developers. As I make that plugin I also will be working on a way to test the plugins as well.

I do not know about anyone else, but, I have been thinking that the pre pug hook, where the plugin hands over the raw pug files, is not needed. I cannot think of a use case that cannot be covered by the html hook or page hooks. I am going to remove that unless someone needs it.

@DanielRuf, Yea, when I meant an actual npm package I did not mean a published one, just local testing to ensure everything is working, we will publish real ones later on.

[Zulko]

@Drew-S An "example plugin" makes sense. Can we call it relaxed-example-plugin so that people who come across it won't believe it is a usage example for ReLaXed. Also, can we host it on this organization's Github page ? Do you have the permissions to create a new repo here ?

As I said before I share your feeling about prePug filters. As I understand it, it should be easy to add new "hooks" (like prePug) to plugins later, without breaking the compatibility of existing plugins (they'll just have "undefined" for this hook). Is that correct ?

[Drew-S]

@Zulk Yes, I do have permission to create a new repo. All the hooks are isolated so adding or removing them will not cause harm, instead of just removing the code we can have the disabled function return an error or statement that it is not available.

[Zulko]

Here is some inspiration on what the yaml file could look like:

https://github.com/svg/svgo/blob/master/docs/how-it-works/en.md

[Drew-S]

I added yaml support as well as json to the plugin system, I also tested it with normal relaxed, it does not affect anything when plugins are omitted. I think we can integrate the plugin system with the master branch without issue, I still want to get the example plugin made though. I am currently working on that.

Yaml structure:

someSetting: 'thing'
plugins:
  - example-plugin:
    - dependencies:
      - test2-plugin
    - someVariable: 4

JSON Structure:

{
  "someSetting": "thing",
  "plugins": [
    {
      "example-plugin: {
        "dependencies": ["test2-plugin"],
        "someVariable": 4
      }
    }
  ]
}

[Zulko]

what is "dependencies" and does it need to be set by the user ? Is that some kind of "run after this other plugin ?"

[Drew-S]

Its optional. Essentially yes, it is a run after some other plugin thing, I do not think there will be much in the way of dependencies, but it is a real possibility to consider.

[Zulko]

I'm nitpicking, but can we call it depends_on like docker does ?

[Drew-S]

It honestly does not matter to me what we call it, I just kept to the npm packages.json naming convention. Either works for me. I just completed testing and have a full example plugin ready to go, Ill go ahead a create a new repo and upload the example there. Take a look: example-plugin

[Zulko]

I just pushed a giant commit which does a lot of reorganizing, and rewrites in some extent the way plugins are loaded (I'll document it a bit more later), it comes with some features (again, to be documented) but I may also have cut some of the plugin possibilities you wanted (notably dependencies) when I thought it was too complicated for the moment. Bibliography etc. work, as shown by the tests.

I have also renamed some things, for instance mixins becomes pugHeaders because mixins made it unclear that it was pug-related, and they don't have to be mixins in particular anyways.

Edit: this is a halfway-commit, in the sense that some features will very soon get their own plugins (mermaid, flowcharts, vegalite), which will lead to the removal of converters.js, the templates/ directory, and probably the utils.js module. the code will look much, much simpler when this is done.

[Zulko]

The latest commit ends the path to "a" plugin system. It is certainly not definitive, there are some hooks I want to add:

• css hooks (to add CSS in the header of the final document)
• in-pug transformation filters (like :markdown-it))
• post-pdf hooks.

[Zulko]

I will spend some time documenting it later, and updating the relaxed-examples and relaxed-example-plugin repos. In the meantime, you can have a look at:

• The plugin implementations
• An example of use (bibliography)
• An example of a local, user-defined plugin