Support for custom templates

[Russman12]

I was surprised to find that there doesn't seem to be any way to provide a custom template for the generated markdown. I know there are post processing hooks and ways to update the frontmatter, but I could not find any way to customize the templates themselves.

While I generally really like this project, I think the templates are meh. It certainly gets the job done, but I think it looks too cluttered. Personally, I like how salesforce's apex documentation has a bulleted list of methods and descriptions, with links to more detailed info about the methods. In the sprit of making the project "non opinionated"

I really think there should be a way for consumers to provide their own templates. This would greatly increase the customizability of package.

Some ways this can be implemented:

1. Provide a new config, for pointing to a custom templates directory. When provided, any templates matching expected names will override the default templates provided by the project. In the future, this could be expanded on to allow consumers to specify any template under the directory, and have some syntax to link it or something that could be very robust.
2. On install, create a template directory within the consumer project with default templates. This would be the new exclusive source for apexdoc's templates. If a consumer doesn't like the template, they can just change it. This would require no config changes for the consumer, but would also allow the consumer to break things.
3. Have multiple config entries, one for each individual template. This might be verbose if a caller wants to override multiple templates, but it's the most explicit of the options as it allows a consumer to structure their project how they want, and only specify the overrides they need.

@cesarParra, I'd like to hear your thoughts on this.

[Russman12]

I've created a PR of a PoC demonstrating a potential solution.

#287

[cesarParra]

Hi @Russman12. Sorry for the radio silence this past week, I had some time off.

Thanks so much for the PR! I fully agree this is needed and something I've been meaning to get to.

One of the concerns I've had with implementing this is that the way Handlebars is used internally by the tool at the moment is a bit complex, and also requires knowledge of the shape of all of the Renderable types. That's fine the way it works at the moment because its internal to the tool and can be refactored at any point, but by exposing the override flag then that implementation becomes part of the public API, which limits the refactoring ability and makes Handlebars a permanent dependency.

Now, I don't have any immediate plans to refactor that, so I think its fine, so we'd just need to document how to override the templates (how to name the files, and what is accessible to the handlebars template)

[Russman12]

I'm glad to hear you agree and want this! I think you make a good point about coupling the public API to handlebars. I'm not sure how we could get away from depending on a specific template engine, while giving complete template control to the caller.

Here's my idea for how a complete solution could look:
There could be a separate module that is responsible for rendering the template. This module accepts 2 elements, the JSON data, and a config. The config would include a portion which represents a "blueprint" for how various rendered files are generated. Elements in this section could map to properties from the JSON data to a template file, and a name function. For example, the following shows how the apexClasses property is selected and each element would generate a distinct file, all rendered using a classTemplate.md template.

{
    blueprints: [ //this list represents each of the data properties which will result in a distinct output file.
        {
            propertyName: "apexClasses", //this is the name of the property from the JSON data. uses dot notation to traverse multiple levels.
            fileName: (propData, data) => { ... }, //this function defines how to derive the file name from the data for the selected prop.
            template: 'classTemplate.md' // the template for which the selected data will be injected
        }
    ]
}

This solution is still coupled to handlebars and is a pretty big change. It also might be a big design shift. The shape of the config might need refinement as well. However, I think this is a more ideal state as it should give near complete control of template design to the caller. I think this could also make some functionality unnecessary, for example, the transformation stuff might not be necessary.

@cesarParra, what are your thoughts on this approach?

[Russman12]

@cesarParra, I owe you an apology. I just reread my original issue message and the tone was all wrong. I think I may have mistakenly submitted a rough draft. I can tell there was a lot of effort put into the templates as are they are today, and I want to acknowledge that. All I was trying to say is that consumers may want a way to customize the template. But I think you already understood that and my careless words were met with your grace and patience. Thanks for that!

[cesarParra]

@Russman12 I really appreciate it, but no worries at all! I didn't take it in any negative way.

I agree that there's really no (easy) way of decoupling this functionality entirely from a templating engine, so I'm good moving ahead with it being Handlebars. But I don't fully understand the proposed solution and why that could remove the need from the transformation. Could you expand a bit on how that could look like (pseudocode is fine, no need to go into implementation details)?

I'm not opposed to a big shift if it makes sense, or even introducing potentially breaking changes to the config, but keeping in mind that this feature should be optional (i.e. if no custom templates are configured, everything works as it currently does).

[Russman12]

@cesarParra, those are great questions. I'll try to concisely describe my idea. I apologize ahead of time, I tried to discover where everything was in the project and implement a PoC, but unfortunately, my TS skills have failed me. Here's a summary of the flow:

1. Files are read and compiled into a complete data structure including all data pertinent to templates.
2. The data is passed to a renderer engine
3. The renderer engine is then responsible for processing all configured renderers, writing files to the local file system.

Prerequisites

First, let's assume the compiled top level data structure looks like this (here, each apex element is same as what gets returned by @cparra/apex-reflection):

{
  "apex": [
    {
      "name": "BaseClass",
      "type_name": "class",
      ...
    },
    {
      "name": "SampleClass",
      "type_name": "class",
      ...
    },
    ...
  ],
  ...
}

Renderer config

A renderer, is a class which represents a file to be created, along with all the instructions necessary to produce it. For example, I could configure a renderer like this:

new Renderer(
  {
    viewData: (data) => data, // this defines what data will be passed to the view relative to the full data structure, in this case, all data
    fileName: (propData, data) => 'docs/data.json', // determines how the resulting file is named, in this case it's static
    template: 'data.hbr', // determines which handlebars template is to be used
  }
);

When this renderer is processed, the handlebars template data.hbr will accept the full top level data structure (including classes, trigger, sobjects, etc), and produce a file, docs/data.json with all that data injected to it.

Here's another more complex example:

new Renderer({
    viewData: (data) => data.apex.filter((element) => element.type_name === 'class'), //data passed to the view is only apex elements that are 'classes'. When this function resolves to an array, a distinct file will be created for each element
    fileName: (propData, data) => `docs/types/${propData['name']}.md`, // propData is the same type as the elements from viewData. We name the file based on the "name" property from our individual "apex" element
    template: 'class.hbr',
});

When this is executed, each of the apex elements where type_name === 'class', will be passed to the template and an output file is created at docs/types/{name}.md for each.

Note

Templates, are dynamically, lazy loaded and cached. Only templates which are referenced in a configured renderer are actually used. They are cached to minimize file reads. When the renderer is executed, the data is selected from the complete data object based on the viewData method. If the function returns an array, then one file is written for each element.

How the config file looks

To configure all this, there could just be a new method within the config file:

export default defineMarkdownConfig({
  renderers: (data: ViewData) => {
    return [
      new Renderer({
        viewData: (data) => data,
        fileName: (propData, data) => 'docs/data.json',
        template: 'data.hbr', 
      }),
    ];
  },
});

Additional related elements

Transformer/decorator: When the reflector returns data for an apex class, it would be nice to have the structure decorated or entirely transformed to present the data in a way more applicable to the view. The same is true for when metadata is returned from things like sobjects.

Helpers: There's several helpers that would need to be implemented so that templates have the necessary tools to build what they want. A bunch of these already exist and probably need to be included. Perhaps there's a way to configure custom helpers as well...

Partials: Partials can be implemented in a similar way as the templates, dynamic, lazy loaded, and cached, but, to my knowledge, need to pass through a method in order to attain that. Here's how that would look from the template:

```handlebars
{{> (partial "methodPartial.hbr") .}}
```

*edit:

default renderers and templates: the project would have built in renderers and templates so that a caller doesn't have to define anything custom if they don't want to. It's also possible for a renderer bundle feature, where users can submit their own template/renderer bundle to include with the project. Then a caller could select an alternative bundle from the config without the complex configuration of a renderers and templates.

Limitations

This solution isn't perfect and does have some limitations. Here's a few I can think of:

• Arrays: It's not possible to pass a list to pass a list to a single template. Instead implementors are forced to pass objects one level above the list they want to access. Lists can be passed to partials though.
• Performance: I'm not sure if there's a performance penalty with providing a large data structure to a template. All data will be loaded to memory at once so there could be issues with huge repositories. I have no idea how legitimate either of these concerns are.
• As stated previously, this is likely a huge refactor. I think this will require some rework to have the data is processed. I think right now, files are written as they are read, reducing memory the load, but this solution would require that the data all exist together in memory in order for it to be injected to the template.

I've created a working PoC of this design, but I'm not sure what the best way is to share it with you.

[Russman12]

To more directly answer your question, transforming the generated files in it's current form may not be necessary, because the caller can define a renderer which performs the necessary action. For example, the vitepress example shows a transformDocs function which creates a sidebar.json file. The proposed solution means that a caller would instead define a handlebar template file representing the shape of the JSON, with placeholders referencing the data to be injected.

It's possible there exists some circumstance that still requires a transformer, but I'm unaware of this.

[cesarParra]

Ok, I think I'm getting it. Would love to see your POC, feel free to share it as a PR if you want (I'm short on time but should be able to give it a look in the next few days).

The one thing I'm wondering is though, with this amount of freedom, wouldn't that undermine the benefit of the tool itself? At some point, if users are writing all of the filtering, transformations, and templating themselves, wouldn't it make sense to import @cparra/apex-reflection and work on top of that output directly? Maybe for users that want this amount of freedom what makes the most sense is to make a more robust JS/TS library that exports the transformations and the most useful templates, and provide the building blocks for power users to build their own tooling out of that.

[Russman12]

I'll make a PR for it. Since it's not integrated with the main project, I'll place it in a new example directory.

Good question, I've thought about that too. No, I don't think this will detract from the benefits of apexdocs, my idea isn't just to make the config more complicated, it's to provide flexibility while allowing for all the customizations that would be desired, within reason. There's an additional layer that could simplify things for implementors: the "bundle" layer.

• Base layer (existing internal functionality): Handles data retrieval from local files, including non-apex sources. It goes beyond @cparra/apex-reflection, covering changelogs, OpenAPI specs, and constructing the data structure to pass to renderers.
• Template layer: Offers flexible templating with customizable config. This is where template relationships are established.
• Bundle layer: Combines templates and renderers into a complete set with its own config options. Apexdocs would ship with a default bundle replicating current markdown output. Bundles could be extended with custom renderers (e.g., VitePress sidebar). A CLI command could export templates for local editing and config override.

An implementor starts at the bundle layer, where they can make some simple config tweaks. think of the current templating functionality as it's own sort of "bundle". If the implementor needs to extend a bundle with their own template, they can reach into the template layer to produce their own template/renderer. If they want to simply modify the existing template, they can extract the default template with a command, and make changes to the handlebar files directly.

I don’t think this detracts from the tool, rather, it lays a flexible foundation for future growth. I haven't nailed down exactly how this "bundle" layer would look or how configuration would work, but that's not strictly necessary to implement the first part, the renderer/templating.

[Russman12]

@cesarParra, I just updated my PR #287 with a PoC of the proposed solution. Let me know what you think!

[cesarParra]

Thanks @Russman12

I like your "layered" idea. What's most important to me is honestly to keep the tool pretty much as is for users who don't create custom configs (which is the vast majority of users - pretty much everyone uses the CLI commands as is without configuration). From there I'm pretty open to any internal refactoring that don't break the public API, and even some breaking changes within reason.

With your approach there does seem to be a path forward where pretty much no breaking changes need to be introduced, if any. Things can continue to work as they do today but once it is time to run the templating it either routes to the default one (what exists today), or a user-configured one.

The one thing I do think still needs to happen is that the templating engine should receive a different input than the raw output from @cparra-apex-reflection (and I'm simplifying because it is not just apex-reflection that gets used for parsing, there is also Custom Object/Field and CMT parsing that is also supported).

That's because there is also some additional transformation that occurs to enhance that output with relationships between classes. This happens in many ways, for example, a method/property in one class might have another class as a return type, or there could be an @see annotation referencing a different class, etc. Whenever this happens, those references are turned into markdown links (eventually <a> tags) in the finalized template, so that users can navigate between them in the final site.

That's a common enough use case that I'm thinking any template implementor (including the tool internally) would want to have access to that (and other similar "enhancements" to the reflection output). That also prevents exposing the output of apex-reflection directly, allowing that tool to change independently of this project (I know I own both 😅 but reflection gets used independently by some people, including myself, for some CI/CD things).

Today, the "transformed" version that the templates receive are all of the Renderable permutations. So I'm thinking something similar, but way more generic, since today those are tailor-made for the existing templates.

[Russman12]

@cesarParra, thanks for the quick response!

I agree that the input to the engine should be different from the output of @cparra-apex-reflection. In my example I have a loader and a transformer which handle loading the data with the reflector, then transforming it to another format. I expect this transformation logic already exists in the project and likely doesn't have to change much, if at all. The engine just needs a type that includes a superset of all data pertinent to the docs. To change the type, you just need to update the engine data property and the signature of the renderer config viewData and fileName props. I think something like the Renderable permutations could be great.

Links are a monster and I've been thinking about how to solve this problem. The actual link to a file will be determined by the renderer's fileName, which is a method that resolves per file. The initial transformation occurs before the renderers are even instantiated, so it's not possible to transform the data up front to include a link. Furthermore, renderers could be configured to create files in ./classes and in ./interfaces with multiple renderers. For this reason, we can't even add a raw link to the template because in this scenario the template needs to discern if the link is an interface vs a class. I agree we need to add a link property to the data, but I think it needs to happen within the bundle layer.

I think what we want is to decorate the transformed model in the bundle. Here, there is some functionality which allows the bundle to decorate the model. While creating links is likely something that all bundles will need, how exactly that happens might look different depending on the bundles. We could make an abstract function like getSeeLink on the bundle class which returns the function describing how to determine the link.

This idea also plays into what you're describing as the tailor-made portions of your Renderable permutations. Effectively, most of that stuff could go into the default bundle's decorator functionality.

One other thing I hadn't figured out, is how to make the bundle templates internal. I would like to see a bundle contain all of it's templates internally so they can't be changed. Then, if configured, the bundle would instead look at a project directory, where a caller could customize a bundle file. I ran into some issues with this approach though because of the ways partials work. I'll see if I can figure that out.

I'll add some commits to my PR when I get a chance, so you can see my ideas in action.

[cesarParra]

Just wanted to mention, thanks so much for being willing and open to contribute. I do want to be honest and upfront and mention that at the moment I don't have a lot of time to dedicate to this, so I'll be prioritizing #283 whenever I do find some time, but I'm very open to PRs and should be able review those (relatively) quickly.

---

For links, we as you mention we might not be able to pass the final link, but there are a few options that come to mind. For one, the code currently does a couple of passthroughs, where the first one builds a "Reference" all of code/metatada (which eventually becomes the index file of the output docs). That same concept can be applied to build a reference lookup table before templating occurs.

[Russman12]

That's understandable. It looks like the official documentation as some differences in syntax than what is currently in the apexdocs, so I'm sure you have a bunch to work out with that.

I'll investigate some more and see if I can do anything else when I have some free time. Do you want me to leave the PR open so you can review it, or should I close it to keep the project tidy?