Draft: V3.0

[rdmueller]

Hello fellow Documentarians!

We would like to start the work on the next major release of docToolchain and ask for your input and feedback.

The big feature of V2.0 was the docToolchain-Wrapper. It moved all the code out of sight. The main goal was to have only your documents within your repository and all the scipts, HTML templates etc. somewhere else - in a docker container or in an install folder within your home folder.

The idea for V3.0 is to make this approach more modular, flexible and faster. Here are the main thoughts:

Problems identified

slow startup of Gradle

Gradle helped us to get started and it is a great build tool. But docToolchain doesn't use it as build tool anymore. It is mainly used for dependency management for Java based dependencies. The downside of Gradle is that it need quite long to start up.

every thing Java

Through Gradle, docToolchain tasks are mainly Java or Groovy tasks. But we already have lots of tasks which make use of other scripting language like exportEA. It would be great if tasks could be written in whatever technology best fits

Solutions

• The vision is to be able to create tasks in whatever technology you like.
• all tasks should share a common configuration file
• all tasks should be orchestrated thorugh dtcw
• dtcw should take care of the dependencies of those tasks. Even for runtime environments. So if a tasks needs, let's say python, then dtcw should go and fetch a python environment
• tasks should be more like plugins. A basic set of tasks should be shipped with docToolchain, but it should be also easy to discover and add new suitable tasks

[rdmueller]

@mh182, @ascheman the above text is just a draft - a dump of some ideas. What are your thoughts?

[mh182]

I'm not sure if it makes sense to go for a version 3.0. A major version change implies either incompatibility or a big feature/change.

But first, let's discuss the problems you identified.

slow startup of Gradle

Gradle helped us to get started and it is a great build tool. But docToolchain doesn't use it as build tool anymore. It is mainly used for dependency management for Java based dependencies. The downside of Gradle is that it need quite long to start up.

Do you have any numbers? What I mean, what is the Gradle startup time compared to the time to generate the document?
If the runtime for document generation by Asciidoctor takes a minute, saving 5 seconds in the startup is not well invested.

So I would like to gather some data based on the size of the documentation project.

• startup time for Gradle (empty documentation project)
• average documentation project
• Documentation project

every thing Java

Through Gradle, docToolchain tasks are mainly Java or Groovy tasks. But we already have lots of tasks which make use of other scripting language like exportEA. It would be great if tasks could be written in whatever technology best fits

Do we (as a project) really want to support any technology? If after a couple of years we have Lua, Python, Rust or some other script how do we want to provide support if bugs show up. From experience, people show up, if you are lucky they drop some code and then leave (and never to be seen again).

I would rather go with:

• Define a boundary: what technology, what code quality, ...
• Provide a good template as starting point with good documentation.
• Make it easy to use other technology. Those modules are not supported by the docToolchain project, but can easily be integrated (for example as git submodule).

I have to think what other problems we have and add those to this issue soon.

[mh182]

docToolchain acts as glue for a set of software components with the goal to make it easy to generate professional documentation based on Asciidoc format.

So the core values docToolchain provides is:

• An introduction to documentation as a code: source formats, available tooling, references to more information (arc42),
• Easy installation of the tool chain to create the documentation
• Keep the different components up-to-date
• Make it easy to generate the documentation
• Support for different platforms: Linux, macOS, MS Windows

Some problems I see at the moment (which are part of the problems you described above):

• The default configuration is overwhelming and for most people unnecessary (this goes against the design rule: only expose necessary information).
• Installation size:
  • Not using the latest Java LTS: 310MB Java to install Java 11 just for docToolchain
  • Installed components: only materialize components which are needed
  • The docker image size: almost 1 GB, can it be reduced?
• It's quite cumbersome to create a Self-Contained docToolchain.
• dtcw is leaking the underlying assembly system (Gradle): this may be seen as good, but makes it harder to change in the future.
• 235 open issues (whenever I decide to use an open source tool, I go to the project page and check how well maintained it is).

Easy to solve, but nevertheless important:

• The user documentation for the first steps could be improved : there is missing a picture which provide help for how docToolchain is used.
• We should make a triage and close outdated issues. I will start with closing the dtcw related issues when I finished the refactoring I’m currently working on. I would in general close all issues which are older than a certain time period and ask users to reopen them again if they disagree.

Coming back to your solutions.

• dtcw should take care of the dependencies of those tasks. Even for runtime environments. So if a tasks needs, let's say python, then dtcw should go and fetch a python environment

Maybe I misunderstand you here. If you mean dtcw should take care to install dependencies needed by tasks, then I agree with you. But dtcw should not take care to assemble the components needed by a task. That's the main purpose of the underlying assembly tool (Gradle).

The main functionalities of dtcw should be:

• manage (install, upgrade, change) docToolchain base installations (environments)
• forward call to the right docToolchain environment
• manage the used docToolchain "modules" (i.e. "tasks" - here you see that how Gradle term is leaking)
  • list of available (converter) modules (Visio, Excel, ...)
  • add configuration of a module to the project configuration
  • remove configuration of a module from the project
  • Remove/install modules based on the project configuration
• manage dtcw itself: upgrade docToolchain, upgrade dtcw
• provide means to create a self-contained docToolchain installation package which can be deployed on a machine which has not internet access.

Since the core components are Java centric, I would stick to Java assembly (we don't really need a build tool) tools which are:

• Gradle
• Bazel
• Maven
• Ant

To be honest, I do not see much value in changing something there.

One point we should discuss is, if we want to support packages for Linux OS: Debian and RH. But packaging Java components is PITA, especially for fast changing components.

That's all from my side.

[rdmueller]

thanx for the feedback. I think our thoughts do align. Let's push those things forward which can be made better.

[lprimak]

I would like to update to the latest Java 20, Groovy 4, Gradle 8+ etc. ast the first thing

[lprimak]

Also, separate things so they can be invoked outside gradle (i.e. maven)
I think that was mentioned above as well

[mh182]

Maven was just mentioned as a possible candidate. But since we currently use Gradle I don't think we switch to Maven since this would not provide any advantages to the current state.

[lprimak]

I am not talking about a switch to maven. I am talking about separating the docToolchain components so tasks such as generateSite could be called from maven instead of forcing gradle.

[mh182]

Well, the components you mention are 95% Gradle scripts (Groovy, Java, JavaScript): materialize dependencies/plugins, create directories, copy files, call the materialized plugin.

What I mean to explain is, that docToolchain, as it is at the moment, is meant to be a black box which happens to be driven by Gradle. It doesn't make sense to provide components with an interface, so those components can be called from different build systems.

The interface is doctoolchain shell script, respective dtcw. Your build automation system calls those scripts. That's it.

The use of Gradle is implementation detail, which unfortunately leaks too much at the moment.

[lprimak]

Well, the components you mention are 95% Gradle scripts

Yes exactly! These scripts can be pulled out of gradle and be made into a library. None of that stuff depends on gradle.
That way, when "black box" doesn't work (my case) the code can be reused directly.

[ascheman]

Sorry for being late to the discussions but I'd like to bring in my thoughts as well. Thanks for the discussion so far, it has helped me clear my mind.

Separation of tools/tasks

In general, I would appreciate a better separation of the single tools for tasks. I would like to be able to call them stand alone and even from misc. orchestrators:

• Those orchestrators could be Java build systems like the given Gradle, or Maven.
• It could also be (cloud) pipeline engines like Gitlab CI, Github Actions, Tekton.
• Individual shell (or Groovy or whatever) scripts might be calling the tasks if I needed to make up a special sequence of interacting tools for my project.

So I would like to make the tools available as separate libraries with perhaps a thin layer of integration into their orchestrators around them. Think of this thin layer as either an integration with Gradle/Maven/..., or a Groovy/JBang/... script with the library as dependency, or whatever would be needed for, e.g., pipeline engines.

Configuration

The other challenge is the configuration. The current configuration reflects all possible integrations of the currently provided tasks. For example, you can integrate with Atlassian Jira or Confluence as well as you can create a JBake based micro site. This is nice when you want to make use of all aspects of docToolchain. But I suspect, that most users only need a very small part of it. Hence, they are confused by the many parts they never would even imagine to use. Additionally some parts of the configuration depend on each other and are hard to maintain or even configure, e.g., generating html and then performing sanity checks on them.

Orchestrated configurations

Lets talk about the full-blown docToolchain as we know it today. I would rather like to start with a simple configuration (which is perhaps very opinionated behind the scenes) and then allow to stepwise override the default behaviour. One approach could be a yaml file for configuration which could be empty (not given at all) or very small in the beginning to perform the opinionated approach. Execution of steps could even be dependant on the existence of files.

Examples:

If there is an index.adoc in src/docs, perform the following when dtcw is called:

• Create an html file out of it
• Perform a sanity check.

If there is a src/site folder, generate a microsite with jbake.

Open question: how to express dependencies between different tasks (as with html vs. sanity checking), in general, having one task consuming the result of another?

Solution idea: Provide a yaml based configuration. The following is just an outline of a rather simple configuration with small adjustments

// Configuration (excerpt) for multi documentation output (i.e., handbooks for architecture, development, operations)
docToolchain:
  build: generated // generate all documentation artefacts to folder `generated/` instead of `build/` as before
  artefacts: 
  - name: jiratickets
    tool: jiraexport
  - name: developmentdoc
    source: manuals/development // Take the index.adoc from here instead of `src/docs/` as per default)
    dependson: [jiratickets] // Make use of exported Jira tickets
  - name: architecturedoc
    tool: asciidoctor
    source: manuals/arch
    formats: [html, pdf] // Generate HTML and PDF (not only HTML as per default).
  checks:
  - name: htmlsanity
    tool: htmlsanitycheck
    source: generated/*/index.html // File globbing pattern to perform sanity check for all individual docs 

Stand alone usage/configuration

On the other hand, if I'd separate the tasks as libraries and enable stand alone usage, it will be necessary to individually configure them. An (strongly) opinionated approach could help here as well: Provide as little configuration as necessary to make the tool perform a helpful build (step).

Examples:

• asciidoctor is already there to generate a single html/pdf from an .adoc (would we need a wrapper to include typical plugins like asciidoctor-diagram?)
• htmlsanitycheck could be useful (I will try to extend the given aim42 implementation respectively -> @gernotstarke)
• jiraexport could be provided and would need at least a URL and credentials. The latter are usually independent from a generic build configuration anyways and could be provided by environment variables.
• confluenceupload could work in a similar way.

Compositions

If a user needed a customized doc build, they could still compile a (Shell) script to call the respective tasks in their project context.

Perhaps a general entry point like dtcw could still also allow to only execute some of the standalone tools. Consider dtcw jiraexport changelog for example to generate a changelog which can refer to the currently exported Jira tickets.

[lprimak]

All great points!
Except yaml. I would rather have groovy config file instead of yaml. Anything but yaml :)

[ascheman]

All great points! Except yaml. I would rather have groovy config file instead of yaml. Anything but yaml :)

I am also not a big fan of yaml. However, for some use cases it's a little bit like democracy: From all the bad solutions it is the best one (or least worse) ;-). Besides: You can always have a Groovy based configuration with DTC, it is called Gradle ...

[rdmueller]

The groovy config has some advantages, because it is dynamic. It allows me to read environment variables and other nice things. The result however is just a map. We could add a feature to read a yaml or json config as map. This would be 10 lines of code. Do json and yaml have comments? Because this is also an important feature...

…

Message ID: ***@***.*** com>

[ascheman]

In the long run I'd be happy to have schema for yaml as it would allow editor support.
yaml has comments, json usually not (perhaps in the latest version?), therefore I prefer yaml over json.

[mh182]

Separation of tools/tasks

In general, I would appreciate a better separation of the single tools for tasks. I would like to be able to call them stand alone and even from misc. orchestrators:

I think we have a different point of view on this topic. Let's take a step back to discuss what docToolchain is.

Quoting from docToolchain homepage

"docToolchain is a collection of scripts that makes it easy to create and maintain powerful technical documentation."

Currently, the interface supported by docToolchain are shell scripts: dtcw and doctoolchain (respective their counterparts for MS Windows doctoolchain.ps1).

Note: the use of Gradle under the hood is an implementation detail.

docToolchain is vendoring all components, including Gradle.

Unfortunately, this implementation detail leaks to the user and irritates people who already use a build system like Gradle/Maven. And I can understand that an engineer doesn't want to install the same tooling, in different versions, twice.

Now we have to discuss the current implementation of those "set of scripts" (tasks) do:

• they define the dependencies on the tools needed to perform a functionanlity
• they materialize (download) the tools
• they read an user-provided configuration file and process some input, create some output, based on this configuration

Appart from a few tasks most of the functionanlities of a task are based on Gradle features.

If you "rip out" the part of the code of a task which would not depend on Gradle there is noting left, because Gradle is used to "drive the scripts".

But let's assume for a moment we could achieve the goal as you suggested, separating the single tool which could be called by orchestrators.

• which interface (technology) would you use so that different orchestrators could use them?
• would this make it "easy" to create the documentation (one of the main goals of docToolchain)?

To put this to test: pick the core task AsciiDocBasics.gradle which implements generateHTML and generatePDF and refactor it in such a way, so it would provide a new interface which can be called by Gradle, Maven and a bash script.
Maybe we come up with a new interface which we can use for further discussion.

I lack the know-how (and the time) to achieve this.

* Those orchestrators could be Java build systems like the given Gradle, or Maven.

What about build systems other than Java? I haven't worked with Java since 10 years, and I still need documentation for the systems I design.

I can understand that people coming from a Java world are irritated to have "another Gradle" installed by docToolchain.
But as explained before, it is an implementation detail which should not concern you.

* It could also be (cloud) pipeline engines like Gitlab CI, Github Actions, Tekton.

I could not see why the current docToolchain can't be used by CI/CD systems. Use the Docker container and call dtcw <taskname>.

* Individual shell (or Groovy or whatever) scripts might be calling the tasks if I needed to make up a special sequence of interacting tools for my project.

Once again: you already can do that. dtcw <task1> <task2> or chaining the call dtcw task1 && dtcw task2

So I would like to make the tools available as separate libraries with perhaps a thin layer of integration into their orchestrators around them. Think of this thin layer as either an integration with Gradle/Maven/..., or a Groovy/JBang/... script with the library as dependency, or whatever would be needed for, e.g., pipeline engines.

I understand what you mean. Please provide a prototype of this thin layer, so we could discuss this further.

Configuration

The other challenge is the configuration. The current configuration reflects all possible integrations of the currently provided tasks. For example, you can integrate with Atlassian Jira or Confluence as well as you can create a JBake based micro site. This is nice when you want to make use of all aspects of docToolchain. But I suspect, that most users only need a very small part of it. Hence, they are confused by the many parts they never would even imagine to use. Additionally some parts of the configuration depend on each other and are hard to maintain or even configure, e.g., generating html and then performing sanity checks on them.

I fully agree with you and this was already mentioned in my comment

Orchestrated configurations

Lets talk about the full-blown docToolchain as we know it today. I would rather like to start with a simple configuration (which is perhaps very opinionated behind the scenes) and then allow to stepwise override the default behaviour. One approach could be a yaml file for configuration which could be empty (not given at all) or very small in the beginning to perform the opinionated approach. Execution of steps could even be dependant on the existence of files.

Examples:

If there is an index.adoc in src/docs, perform the following when dtcw is called:

• Create an html file out of it
• Perform a sanity check.

If there is a src/site folder, generate a microsite with jbake.

This means docToolchain would provide an opinionated structure (like for example Maven is doing).
I’m more a friend of "explicit is better than implicit". Having a minimal configuration, only providing the configuration for the used tasks, would be a good start.

Open question: how to express dependencies between different tasks (as with html vs. sanity checking), in general, having one task consuming the result of another?

Solution idea: Provide a yaml based configuration. The following is just an outline of a rather simple configuration with small adjustments

// Configuration (excerpt) for multi documentation output (i.e., handbooks for architecture, development, operations)
docToolchain:
  build: generated // generate all documentation artefacts to folder `generated/` instead of `build/` as before
  artefacts: 
  - name: jiratickets
    tool: jiraexport
  - name: developmentdoc
    source: manuals/development // Take the index.adoc from here instead of `src/docs/` as per default)
    dependson: [jiratickets] // Make use of exported Jira tickets
  - name: architecturedoc
    tool: asciidoctor
    source: manuals/arch
    formats: [html, pdf] // Generate HTML and PDF (not only HTML as per default).
  checks:
  - name: htmlsanity
    tool: htmlsanitycheck
    source: generated/*/index.html // File globbing pattern to perform sanity check for all individual docs 

Task dependencies is a different aspect and I would like to separate this from the "simplify configuration" feature.
I agree with you, that it would be nice to have it.

In an ideal world docToolchain should know in which order the tasks have to be executed.
We just have to make sure the user can change the default behavior in case it is needed.

At the moment I see this low prio, since we can do it manually.

Stand alone usage/configuration

This makes only sense to discuss if we have a proto-type with a possible interface (technology) as mentioned above.

Compositions

If a user needed a customized doc build, they could still compile a (Shell) script to call the respective tasks in their project context.

Perhaps a general entry point like dtcw could still also allow to only execute some of the standalone tools. Consider dtcw jiraexport changelog for example to generate a changelog which can refer to the currently exported Jira tickets.

Dito: this makes only sense to discuss if we have a proto-type with a possible interface (technology) as mentioned above.

[lprimak]

Appart from a few tasks most of the functionanlities of a task are based on Gradle features.

I do not believe this is correct. They are based on groovy, not gradle.
Groovy can be perfectly compiled into a JAR so it can be reused by the orchestration tool.
Orchestration tool can be gradle, maven, or something else entirely.

My experience was this:
I am a current https://sdkman.io user. My CI/CD has sdkman, java, gradle installed. It has no docker (as it is too heavy currently on Apple Silicon
I installed doctoolchain via sdkman and it failed miserably because I had gradle 8 and java 20 installed.

While doctoolchain does a lot of things, one thing it doesn't do is integrate with existing landing page and marketing web sites.
I would need to re-skin doctoolchain and integrate it with the current web site. This was impossible, even though existing site already using jbake because the rest of the tools are java 17 minimum fronted by maven.

Bottom line: doctoolchain is a great tool but it requires a unique (obsolete?) dependency and tool configuration (old java, old gradle, old jbake, old asciidoctor) and it doesn't integrate well with the rest of the ecosystem, which, IMHO can (and should) be improved.

[mh182]

Appart from a few tasks most of the functionanlities of a task are based on Gradle features.

I do not believe this is correct. They are based on groovy, not gradle. Groovy can be perfectly compiled into a JAR so it can be reused by the orchestration tool. Orchestration tool can be gradle, maven, or something else entirely.

As I'm not Groovy nor Gradle literate, I have a hard time to distinguish between Groovy and Gradle functionality.

But as I mentioned before, come up with a POC of how such a small layer could look like with https://github.com/docToolchain/docToolchain/blob/ng/scripts/AsciiDocBasics.gradle. I would be glad to discuss a possible solution for such a layer and how we could achieve the goal so that docToolchain supports a more low-level interface which makes it easier to integrate it into build systems "natively".

My main concern is how we, as docToolchain project, are not frustrating users with breaking changes. Since the term "breaking change" is defined by the definition of the interface, I don't see the capacity (for the moment) to support additional interfaces than the existing one: doctoolchain and dtcw.

My experience was this: I am a current https://sdkman.io user. My CI/CD has sdkman, java, gradle installed. It has no docker (as it is too heavy currently on Apple Silicon I installed doctoolchain via sdkman and it failed miserably because I had gradle 8 and java 20 installed.

We are currently trying to upgrade the infrastructure (see #1102) which should reduce your pain point. If you look into the issue, you will see that it is not easy to integrate the different tools so that they work together.

Nevertheless, you are supposed to call dtcw or doctoolchain within your build system. All you have to do is to install an additional Java and Gradle version. Yes, it is not ideal, and it takes is "a bit" (blame Java for that) of hard disk space.

While doctoolchain does a lot of things, one thing it doesn't do is integrate with existing landing page and marketing web sites. I would need to re-skin doctoolchain and integrate it with the current web site. This was impossible, even though existing site already using jbake because the rest of the tools are java 17 minimum fronted by maven.

This is a whole different point of discussion. I use docToolchain to create technical documentation with developers as the primary target audience. Personally, I'm not (yet) convinced that docToolchain is the right tool to make a landing page and marketing page for a product.

Bottom line: doctoolchain is a great tool but it requires a unique (obsolete?) dependency and tool configuration (old java, old gradle, old jbake, old asciidoctor) and it doesn't integrate well with the rest of the ecosystem, which, IMHO can (and should) be improved.

Thanks for the feedback.

[ascheman]

Check out our proposal to make HTML Sanity Checker more modular in order to streamline docToolchain as well (and please vote it up if you like it).

[rdmueller]

Regarding the small layer and difference between groovy and gradle: the two confluence tasks are a good example. They can be used stand-alone when you add some grab/grape annotations to fetch dependencies. I would like to check if we could wrap them in an if-statement.

The asciidocbasic.gradle script currently heavily depends on gradle and the plugins.

[mh182]

Regarding the small layer and difference between groovy and gradle: the two confluence tasks are a good example. They can be used stand-alone when you add some grab/grape annotations to fetch dependencies. I would like to check if we could wrap them in an if-statement.

The asciidocbasic.gradle script currently heavily depends on gradle and the plugins.

Holy sh**. I just skimmed through the 1000 lines of asciidoc2confluence.groovy - that's something you can scare people.

How do you test this code?

The asciidocbasic.gradle script currently heavily depends on gradle and the plugins.

But that script is the heart of docToolchain. If we can't come up with a POC for this script, everything else is moot.

[rdmueller]

1000 lines of code

Yes, it grew a bit but it is better structured than half a year ago. Mea culpa.

Asciidocbasic.gradle

I think a) it wouldn't be a problem to still have tasks which depend on gradle and b) it can be rewritten to use the commandline jbake and asciidoctorj

[ascheman]

Some people created a small Proof-of-Concept how to migrate away from Gradle scripts towards a class based implementation: #1208 .

Thanks to @sparsick @PacoVK and @johthor !

[PacoVK]

While digging into #1226 and having #1208 in mind, i started to investigate a bit further.
Now that i have a running PoC that IMHO is kind of acceptable, here are some of my findings:

• A separation dramatically improves DX, because you can simply checkout docToolchain and the IDE knows which dependency and runtime to link. Just like any other project
• Encapsulating business logic into a dedicated submodule that does not know anything about Gradle or any other buildtool, opens the door for non-Gradle usage, like aforementioned in the discussion. Gradle is still the first class citizen as of now.
• Much easier to test, because you can target the scope of a very test, no need to build logic into any gradle Task and therefore no need to run the Gradle Runner. However, those tests may still be valid as kind of "integration tests"
• Easier to understand, all groovy-written business logic could be centralized and managed. There is no such thing that a part is in a Gradle Task definition and other parts are in scripts.
• Much easier dependency tree management. No need to specify dependencies multiple times in gradle files buildscript the core submodule as dependency is just enough
• Without Gradle Runner, the Unit testsuite will be tremendously faster
• In WIP: Start a POC for separation of tools and tasks #1208 we used Gradles buildSrcmechanisms, but they seem to be very limited and also very Gradle specific. A submodule is IMHO the better solution.

While this list may not be complete, i'd like to continue this way and have a discussion about this.
As @rdmueller already suggested in #1226 this could result in a new ADR for docToolchain, once we came to a decision.
I will draft the ADR in the nearest future, as soon as i have enough infos

[PacoVK]

Maybe i need to rename #1226 at some point, i went quite out of scope...

[rdmueller]

That all sounds great! 🥳

[PacoVK]

First ADR drafted in #1243 Please keep the discussion ongoing and feel free to review :)

[rdmueller]

I would like to take the ADR even one step further.

We do not use the Gradle features anymore. Even the dependency resolving mechanism is a pain for some types of installation. I guess there are better ways to go. In addition, Gradle startup time is slow and Gradle needs lot of resources.

Let's try to avoid Gradle altogether. There is already one small task which shows a different path:

https://github.com/docToolchain/docToolchain/blob/ng/bin/doctoolchain#L39

the Gradle wrapper calls bin/doctoolchain as last action and we can divert tasks away from Gradle at this point. We could even create a configurable switch to enable a soft launch