From 256314a34903b4b9ec70198177ffb9d5bf44aacb Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 11:40:34 +0200 Subject: [PATCH 1/9] Update our README.md and introduce CONTRIBUTING.md --- CONTRIBUTING.MD | 74 ++++++++++++++++++++ README.md | 171 ++++++++++++++--------------------------------- docs-builder.sln | 1 + 3 files changed, 126 insertions(+), 120 deletions(-) create mode 100644 CONTRIBUTING.MD diff --git a/CONTRIBUTING.MD b/CONTRIBUTING.MD new file mode 100644 index 000000000..7b31805c8 --- /dev/null +++ b/CONTRIBUTING.MD @@ -0,0 +1,74 @@ +# Contributing + +## Prerequisites + +- [.NET 9.0 SDK](https://dotnet.microsoft.com/en-us/download/dotnet/9.0) +- [Node.js 22.13.1 (LTS)](https://nodejs.org/en/blog/release/v22.13.1) + - [Aspire 9.4.1](https://learn.microsoft.com/en-us/dotnet/aspire/) + ```bash + dotnet workload install aspire + ``` + +## Validate the fully assembled documentation + +```bash +dotnet run --project aspire +``` + +Will spin up all our services and clone and build all the documentation sets. + +```markdown +dotnet run --project aspire -- --assume-cloned --skip-private-repositories +``` + +`--asume-cloned` will assume a documentation set set is already cloned if available locally. + +`--skip-private-repositories` will skip cloning private repositories. It will also inject our `docs-builder docs into the +navigation. This allows us to validate new features' effect on the assembly process. + +Our [Integration Tests](./tests-integration/Elastic.Assembler.IntegrationTests) use this exact command to validate the +assembler builds. + +## Continuously build all assets during development. + +```shell +./build.sh watch +``` + +This will monitor code, cshtml template files & static files and reload the application +if any changes. + +Web assets are reloaded through `parcel watch` and don't require a recompilation. + +Markdown files are refreshed automatically through livereload + +Code or layout changes will relaunch the server automatically + +# Release Process + +This section outlines the process for releasing a new version of this project. + +## Versioning + +This project uses [Semantic Versioning](https://semver.org/) and its version is +automatically determined by [release-drafter](https://github.com/release-drafter/release-drafter) +based on the labels of the pull requests merged into the `main` branch. + +See the [release-drafter configuration](./.github/release-drafter.yml) for more details. + +## Creating a New Release + +To create a new release trigger the [release](https://github.com/elastic/docs-builder/actions/workflows/release.yml) workflow on the `main` branch. + +Every time a pull request is merged into the `main` branch, release-drafter will +create a draft release or update the existing draft release in the [Releases](https://github.com/elastic/docs-builder/releases) page. + +To create a new release you need to publish the existing draft release created by release-drafter. + +> [!IMPORTANT] +> Make sure the [release-drafter workflow](https://github.com/elastic/docs-builder/actions/workflows/release-drafter.yml) is finished before publishing the release. + +> [!NOTE] +> When a release is published, the [create-major-tag workflow](./.github/workflows/create-major-tag.yml) +> will force push a new major tag in the format `vX` where `X` is the major version of the release. +> For example, if the release is `1.2.3` was published, the workflow will force push a new tag `v1` on the same commit. diff --git a/README.md b/README.md index 76d189a1e..51fe44e9e 100644 --- a/README.md +++ b/README.md @@ -2,84 +2,83 @@ [![ci](https://github.com/elastic/docs-builder/actions/workflows/ci.yml/badge.svg?branch=main&event=push)](https://github.com/elastic/docs-builder/actions/workflows/ci.yml) -You've reached the home of the latest incarnation of the documentation tooling. +Distributed documentation tooling for a distributed company. This repository is host to: -* *`docs-builder`* command line tool to generate single doc-sets (13mb native code, no dependencies) -* *`docs-assembler`* command line tool to assemble all the doc sets. (IN PROGRESS) -* `elastic/docs-builder@main` Github Action to build and validate a repositories documentation +* *`docs-builder`* command line tool to generate single doc-sets (native code, no dependencies) +* *`docs-assembler`* command line tool to assemble all the doc sets. -## Command line interface +## Distributed documentation -```bash -$ docs-builder --help -Usage: [command] [options...] [-h|--help] [--version] - -Converts a source markdown folder or file to an output folder - -Options: - -p|--path Defaults to the`{pwd}/docs` folder (Default: null) - -o|--output Defaults to `.artifacts/html` (Default: null) - --path-prefix Specifies the path prefix for urls (Default: null) - --force Force a full rebuild of the destination folder (Default: null) - -Commands: - generate Converts a source markdown folder or file to an output folder - serve Continuously serve a documentation folder at http://localhost:3000. - diff validate Validates redirect rules have been applied to the current branch. - File systems changes will be reflected without having to restart the server. -``` +The main driver for folks writing documentation is `docs-builder`. +This tool builds each repository in isolation and in addition produces a links.json file. + +Each time a repository successfully builds on its respective main integration branch, the tool will update its links.json file. +Example: [Elasticsearch's links.json](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/elastic/elasticsearch/main/links.json) representing all linkable resources in the Elasticsearch repository. + +The `docs-assembler` tool then assembles all the repositories in the [link-registry](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/link-index.json) using their last known good commit. -In the near term native code will be published by CI for the following platforms +This allows us to: -| OS | Architectures | -|----------|---------------| -| Windows | x64, Arm64 | -| Linux | x64, Arm64 | -| macOS | x64, Arm64 | +* Validate outbound and inbound links ahead of time during `docs-builder` builds. +* Documentation errors in one repository won't affect all the others. +* Since we only build commits that have produced a `links.json` file the assembler is resilient to repositories having build failures on their integration branches. -And we'll invest time in making sure these are easily obtainable (`brew`, `winget`, `apt`) -For now you can run the tool locally through `docker run` +## Installation +### Linux / OSX ```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ - ghcr.io/elastic/docs-builder:edge +curl -sL https://ela.st/docs-builder-install | sh ``` -This ensures `.git`/`docs` and `.artifacts` (the default output directory) are mounted. +### Windows +```ps +iex (New-Object System.Net.WebClient).DownloadString('https://ela.st/docs-builder-install-win') +``` -The tool will default to incremental compilation. -Only the changed files on subsequent runs will be compiled unless you pass `--force` -to force a new compilation. +Installing through the script will download the latest version of the tool and make it available in your `PATH`. -```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ - ghcr.io/elastic/docs-builder:edge --force -``` +If you want to manually install the tool you can download the latest release from the [Releases Page](https://github.com/elastic/docs-builder/releases) -#### Live mode +### Building locally -Through the `serve` command you can continuously and partially compile your documentation. +Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run: ```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ - -p 3000:3000 ghcr.io/elastic/docs-builder:edge serve +./build.sh publishbinaries ``` +After which the locally built binaries will be available at: -Each page is compiled on demand as you browse http://localhost:3000 and is never cached so changes to files and -navigation will always be reflected upon refresh. +* **docs-builder**: `./.artifacts/publish/docs-builder/release/docs-builder` +* **docs-assembler**: `./.artifacts/publish/docs-assembler/release/docs-assembler` + +## Getting Started -Note the docker image is `linux-x86` and will be somewhat slower to invoke on OSX due to virtualization. +Our [Documentation](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/locally) is the best place to learn how to start using the tool locally. +The TLDR, however, is -## Github Action +* Running `docs-builder` from the root of any checkout with a `docs` folder will build the documentation. + * Running `docs-builder` consecutively will only rebuild the documentation that has changed. + * Running `docs-builder` with the `--force` flag will force a full rebuild of the documentation. + * You can target different folders by passing a path to `docs-builder --path ` +* Running `docs-builder serve` will provide a local server with live reloading. + * You can leave this command running while you add/remove/rename files in your `docs` folder. -The `docs-builder` tool is available as github action. -Since it runs from a precompiled distroless image `~25mb` it's able to execute snappy. (no need to wait for building the tool itself) +#### Live mode + +Through the `serve` command you can continuously and partially compile your documentation. + + +## Github Action + +The `docs-builder` tool is available as a GitHub action. + +It runs as native code on a distroless image `~25mb` so there is little overhead invoking the tooling. ```yaml jobs: @@ -129,74 +128,6 @@ If documentation is moved, renamed or deleted, `docs-builder` can verify if chan --- -## Run without docker - -You can use the .NET CLI to publish a self-contained `docs-builder` native code -binary. (On my M2 Pro mac the binary is currently 16mb) - -Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run: - -```bash -dotnet publish "src/docs-builder/docs-builder.csproj" -``` - -The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed. - -# Performance - -To test performance it's best to build the binary and run outside of docker: - -For reference here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to -existing surveyed tools - # Local Development -## Prerequisites - -- [.NET 9.0 SDK](https://dotnet.microsoft.com/en-us/download/dotnet/9.0) -- [Node.js 22.13.1 (LTS)](https://nodejs.org/en/blog/release/v22.13.1) - - -## Continuously build all assets during development. - -```shell -./build.sh watch -``` - -This will monitor code, cshtml template files & static files and reload the application -if any changes. - -Web assets are reloaded through `parcel watch` and don't require a recompilation. - -Markdown files are refreshed automatically through livereload - -Code or layout changes will relaunch the server automatically - -# Release Process - -This section outlines the process for releasing a new version of this project. - -## Versioning - -This project uses [Semantic Versioning](https://semver.org/) and its version is -automatically determined by [release-drafter](https://github.com/release-drafter/release-drafter) -based on the labels of the pull requests merged into the `main` branch. - -See the [release-drafter configuration](./.github/release-drafter.yml) for more details. - -## Creating a New Release - -To create a new release trigger the [release](https://github.com/elastic/docs-builder/actions/workflows/release.yml) workflow on the `main` branch. - -Every time a pull request is merged into the `main` branch, release-drafter will -create a draft release or update the existing draft release in the [Releases](https://github.com/elastic/docs-builder/releases) page. - -To create a new release you need to publish the existing draft release created by release-drafter. - -> [!IMPORTANT] -> Make sure the [release-drafter workflow](https://github.com/elastic/docs-builder/actions/workflows/release-drafter.yml) is finished before publishing the release. - -> [!NOTE] -> When a release is published, the [create-major-tag workflow](./.github/workflows/create-major-tag.yml) -> will force push a new major tag in the format `vX` where `X` is the major version of the release. -> For example, if the release is `1.2.3` was published, the workflow will force push a new tag `v1` on the same commit. +See [CONTRIBUTING.md](CONTRIBUTING.md) for more information on how to develop locally and contribute to the project. diff --git a/docs-builder.sln b/docs-builder.sln index dd6035845..7af8a0bd8 100644 --- a/docs-builder.sln +++ b/docs-builder.sln @@ -13,6 +13,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "root", "root", "{94AC66E5-3 dotnet-tools.json = dotnet-tools.json global.json = global.json Directory.Packages.props = Directory.Packages.props + CONTRIBUTING.MD = CONTRIBUTING.MD EndProjectSection EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{BE6011CC-1200-4957-B01F-FCCA10C5CF5A}" From bede1761c6fd0e3dd6b699a18d2144e283615481 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 11:46:12 +0200 Subject: [PATCH 2/9] touchups --- README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 51fe44e9e..238a31cc6 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,11 @@ Distributed documentation tooling for a distributed company. This repository is host to: -* *`docs-builder`* command line tool to generate single doc-sets (native code, no dependencies) +* *`docs-builder`* command line tool to generate single doc-sets * *`docs-assembler`* command line tool to assemble all the doc sets. +Both get distributed [as native OSX, Linux and Windows binaries for several CPU architectures.](#installation) + ## Distributed documentation The main driver for folks writing documentation is `docs-builder`. @@ -22,11 +24,12 @@ The `docs-assembler` tool then assembles all the repositories in the [link-regis This allows us to: * Validate outbound and inbound links ahead of time during `docs-builder` builds. -* Documentation errors in one repository won't affect all the others. -* Since we only build commits that have produced a `links.json` file the assembler is resilient to repositories having build failures on their integration branches. +* Snapshot builds: only building commits that produced a `links.json` + * Documentation errors in one repository won't affect all the others. + * Resilient to repositories having build failures on their integration branches, we fall back to the last known good commit. -## Installation +## Installation ### Linux / OSX ```bash From dd54b1320e35d4a25445b1fa1b599ca9f8dab766 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 11:59:53 +0200 Subject: [PATCH 3/9] touchups --- README.md | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 238a31cc6..08129ec2f 100644 --- a/README.md +++ b/README.md @@ -11,19 +11,24 @@ This repository is host to: Both get distributed [as native OSX, Linux and Windows binaries for several CPU architectures.](#installation) +The documentation files: +* are written in common Markdown + * [Additional syntax extension](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) are supported to produce a richer writing and reading experience. +* By conventions the documentation lives in `docs` folder but the tooling will look for any folder holding a `docset.yml` file given the current working directory. + ## Distributed documentation The main driver for folks writing documentation is `docs-builder`. This tool builds each repository in isolation and in addition produces a links.json file. -Each time a repository successfully builds on its respective main integration branch, the tool will update its links.json file. -Example: [Elasticsearch's links.json](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/elastic/elasticsearch/main/links.json) representing all linkable resources in the Elasticsearch repository. +Each time a repository successfully builds on its respective main integration branch, our automation will publish its links.json file. +For example, [Elasticsearch's links.json](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/elastic/elasticsearch/main/links.json) representing all linkable resources in the Elasticsearch repository. The `docs-assembler` tool then assembles all the repositories in the [link-registry](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/link-index.json) using their last known good commit. This allows us to: -* Validate outbound and inbound links ahead of time during `docs-builder` builds. +* Validate outbound and inbound links ahead of time, even during local `docs-builder` builds. * Snapshot builds: only building commits that produced a `links.json` * Documentation errors in one repository won't affect all the others. * Resilient to repositories having build failures on their integration branches, we fall back to the last known good commit. @@ -71,10 +76,10 @@ The TLDR, however, is * You can leave this command running while you add/remove/rename files in your `docs` folder. +### Other commands to know: -#### Live mode - -Through the `serve` command you can continuously and partially compile your documentation. +* `docs-builder mv` [Move files and folders](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/move) +* `docs-builder diff validate` [Manage redirects across doc sets](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/redirects#validation) ## Github Action @@ -121,16 +126,6 @@ https://github.com/elastic/{your-repository}/settings/pages --- -## Validating redirection rules - -If documentation is moved, renamed or deleted, `docs-builder` can verify if changes in the working branch in relation to the default branch are reflected in the repository's `redirects.yml`. Verification in the local machine is currently supported. - -`docs-builder diff validate ` - -`` is an optional parameter to customize the documentation folder path. It defaults to `docs`. - ---- - # Local Development See [CONTRIBUTING.md](CONTRIBUTING.md) for more information on how to develop locally and contribute to the project. From e6eb8bc2274bfe085efee708073df90d5c7912e0 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 12:00:59 +0200 Subject: [PATCH 4/9] touchups --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 08129ec2f..ebe0165e0 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,7 @@ This repository is host to: Both get distributed [as native OSX, Linux and Windows binaries for several CPU architectures.](#installation) The documentation files: -* are written in common Markdown - * [Additional syntax extension](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) are supported to produce a richer writing and reading experience. +* are written in common Markdown with [Additional syntax extension](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) to produce a richer writing and reading experience. * By conventions the documentation lives in `docs` folder but the tooling will look for any folder holding a `docset.yml` file given the current working directory. ## Distributed documentation From cc0c479b051880e666e109ac5352ee1cf0304c4f Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 12:02:44 +0200 Subject: [PATCH 5/9] touchups --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ebe0165e0..8d2f4bada 100644 --- a/README.md +++ b/README.md @@ -12,13 +12,13 @@ This repository is host to: Both get distributed [as native OSX, Linux and Windows binaries for several CPU architectures.](#installation) The documentation files: -* are written in common Markdown with [Additional syntax extension](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) to produce a richer writing and reading experience. +* are written in common Markdown with [Additional syntax extensions](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) to produce a richer writing and reading experience. * By conventions the documentation lives in `docs` folder but the tooling will look for any folder holding a `docset.yml` file given the current working directory. ## Distributed documentation The main driver for folks writing documentation is `docs-builder`. -This tool builds each repository in isolation and in addition produces a links.json file. +This tool builds each repository in isolation and in addition produces a full mapping of all the linkable resources it contains in a `links.json` file. Each time a repository successfully builds on its respective main integration branch, our automation will publish its links.json file. For example, [Elasticsearch's links.json](https://elastic-docs-link-index.s3.us-east-2.amazonaws.com/elastic/elasticsearch/main/links.json) representing all linkable resources in the Elasticsearch repository. @@ -32,7 +32,6 @@ This allows us to: * Documentation errors in one repository won't affect all the others. * Resilient to repositories having build failures on their integration branches, we fall back to the last known good commit. - ## Installation ### Linux / OSX From e7d5b4f9efec0a861390fec5198329fde6754823 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 12:05:05 +0200 Subject: [PATCH 6/9] touchups --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8d2f4bada..94eafc18a 100644 --- a/README.md +++ b/README.md @@ -100,8 +100,7 @@ jobs: ### GitHub Pages -To set up the tool to publish to GitHub pages use the following configuration. -**NOTE**: In the near feature we'll make this a dedicated single step GitHub action +To set up the tool to publish to GitHub Pages, use the following configuration. ```yaml environment: @@ -115,6 +114,8 @@ steps: id: deployment ``` +This single action will build and validate the documentation before publishing. + Make sure your repository settings are set up to deploy from GitHub actions see: https://github.com/elastic/{your-repository}/settings/pages From 7aeacfe49ade5ee5705d96c0c65e2acfb06d401a Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 12:10:25 +0200 Subject: [PATCH 7/9] touchups --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 94eafc18a..42db45a6d 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Both get distributed [as native OSX, Linux and Windows binaries for several CPU The documentation files: * are written in common Markdown with [Additional syntax extensions](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/syntax/) to produce a richer writing and reading experience. -* By conventions the documentation lives in `docs` folder but the tooling will look for any folder holding a `docset.yml` file given the current working directory. +* By conventions the documentation lives in `docs` folder but the tooling will look for any folder holding the `docset.yml` configuration file given the current working directory. ## Distributed documentation @@ -78,6 +78,7 @@ The TLDR, however, is * `docs-builder mv` [Move files and folders](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/move) * `docs-builder diff validate` [Manage redirects across doc sets](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/redirects#validation) +* `inbound-links validate-link-reference` can be used after a build to validate the local `links.json` against all published documentation. ## Github Action @@ -96,8 +97,6 @@ jobs: uses: elastic/docs-builder@main ``` - - ### GitHub Pages To set up the tool to publish to GitHub Pages, use the following configuration. From 51224a61b4aca615af0c3a32bd231cbf6635bcff Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 12:16:05 +0200 Subject: [PATCH 8/9] touchups --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 42db45a6d..4abb8c8ef 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ The TLDR, however, is * `docs-builder mv` [Move files and folders](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/move) * `docs-builder diff validate` [Manage redirects across doc sets](https://docs-v3-preview.elastic.dev/elastic/docs-builder/tree/main/contribute/redirects#validation) -* `inbound-links validate-link-reference` can be used after a build to validate the local `links.json` against all published documentation. +* `docs-builder inbound-links validate-link-reference` can be used after a build to validate the local `links.json` against all published documentation. ## Github Action From 3e1ebcdda64ecd1cf46645bfb3afedb3807ce8b7 Mon Sep 17 00:00:00 2001 From: Martijn Laarman Date: Tue, 26 Aug 2025 15:38:55 +0200 Subject: [PATCH 9/9] Update CONTRIBUTING.MD --- CONTRIBUTING.MD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.MD b/CONTRIBUTING.MD index 7b31805c8..f07195b31 100644 --- a/CONTRIBUTING.MD +++ b/CONTRIBUTING.MD @@ -21,7 +21,7 @@ Will spin up all our services and clone and build all the documentation sets. dotnet run --project aspire -- --assume-cloned --skip-private-repositories ``` -`--asume-cloned` will assume a documentation set set is already cloned if available locally. +`--assume-cloned` will assume a documentation set set is already cloned if available locally. `--skip-private-repositories` will skip cloning private repositories. It will also inject our `docs-builder docs into the navigation. This allows us to validate new features' effect on the assembly process.