More docs updates

README.md

@@ -7,11 +7,11 @@ This repository is host to:
 * *`docs-builder`* command line tool to generate single doc-sets (13mb native code, no dependencies)
 * *`docs-assembler`* command line tool to assemble all the doc sets. (IN PROGRESS)
 * `elastic/docs-builder@main` Github Action to build and validate a repositories documentation
-
+* *`docs-generator`* command line tool to deterministically generate a docset and incremental updates to generated content
 
 ## Command line interface
 
-```
+```bash
 $ docs-builder --help
 Usage: [command] [options...] [-h|--help] [--version]
 
@@ -26,7 +26,7 @@ Options:
 Commands:
   generate    Converts a source markdown folder or file to an output folder
   serve       Continuously serve a documentation folder at http://localhost:5000.
-	File systems changes will be reflected without having to restart the server.
+    File systems changes will be reflected without having to restart the server.
 ```
 
 In the near term native code will be published by CI for the following platforms
@@ -132,22 +132,19 @@ https://github.com/elastic/{your-repository}/settings/pages
 ## Run without docker
 
 You can use the .NET CLI to publish a self-contained `docs-builder` native code
-binary. (On my M2 Pro mac the binary is currently 13mb)
+binary. (On my M2 Pro mac the binary is currently 16mb)
 
 Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run:
 
 ```bash
-dotnet publish "src/docs-builder/docs-builder.csproj" -c Release -o .artifacts/publish \
-    --self-contained true /p:PublishTrimmed=true /p:PublishSingleFile=false /p:PublishAot=true -a arm64
+dotnet publish "src/docs-builder/docs-builder.csproj"
 ```
 
-**Note**: `-a` should be the machine's CPU architecture
-
-The resulting binary `./.artifacts/publish/docs-builder` will run on machines without .NET installed
+The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed.
 
 # Performance
 
 To test performance it's best to build the binary and run outside of docker:
 
-For refence here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to
+For reference here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to
 existing surveyed tools

docs/source/configure/content-set/attributes.md

@@ -0,0 +1,5 @@
+---
+title: Attributes
+---
+
+tbd

docs/source/configure/content-set/file-structure.md

@@ -0,0 +1,5 @@
+---
+title: File structure
+---
+
+tbd

docs/source/configure/content-set/index.md

@@ -0,0 +1,11 @@
+---
+title: Content set configuration
+navigation_title: Content set
+---
+
+Now we'll zoom into the AsciiDoc book level, and explore the V3 equivalent: content sets. At the book level, the system consists of:
+
+| System property | Asciidoc | V3 |
+| -------------------- | -------------------- | -------------------- |
+| **Content source files** --> A whole bunch of markup files as well as any other assets used in the docs (for example, images, videos, and diagrams). | **Markup**: AsciiDoc files **Assets**: Images, videos, and diagrams | **Markup**: MD files **Assets**: Images, videos, and diagrams |
+| **Information architecture** --> A way to specify the order in which these text-based files should appear in the information architecture of the book. | `index.asciidoc` file (this can be spread across several AsciiDoc files, but generally starts with the index file specified in the `conf.yaml` file)) | TBD |

docs/source/configure/content-set/navigation.md

@@ -0,0 +1,5 @@
+---
+title: Navigation
+---
+
+tbd

docs/source/configure/index.md

@@ -0,0 +1,4 @@
+---
+title: Configure Elastic Docs
+navigation_title: Configuration reference
+---

docs/source/configure/page.md

@@ -0,0 +1,10 @@
+---
+title: Page configuration
+navigation_title: Page
+---
+
+tbd
+
+go into frontmatter here?
+
+for now, see [frontmatter](../syntax/frontmatter.md).

docs/source/configure/site/content.md

@@ -0,0 +1,43 @@
+---
+title: Content config
+---
+
+In both the AsciiDoctor- and V3-based system, there is site-wide configuration where you list all content sources, where to find those sources, and in what order they should be added to the site.
+
+In the AsciiDoctor system, this all happens in one YAML file in the /docs repo. In the V3 system, this ...
+
+## AsciiDoctor conf.yml
+
+In the AsciiDoctor-powered site, content configuration at the site level is done in the [`conf.yaml`](https://github.com/elastic/docs/blob/master/conf.yaml) file in the elastic/docs repo. In the `conf.yml` file, the configuration information for all books are listed in this one file. Here's the example of what it looks like to configure a single book:
+
+```yaml
+- title:      Machine Learning
+  prefix:     en/machine-learning
+  current:    *stackcurrent
+  index:      docs/en/stack/ml/index.asciidoc
+  branches:   [ {main: master}, 8.9, 8.8, 8.7, 8.6, 8.5, 8.4, 8.3, 8.2, 8.1, 8.0, 7.17, 7.16, 7.15, 7.14, 7.13, 7.12, 7.11, 7.10, 7.9, 7.8, 7.7, 7.6, 7.5, 7.4, 7.3, 7.2, 7.1, 7.0, 6.8, 6.7, 6.6, 6.5, 6.4, 6.3 ]
+  live:       *stacklive
+  chunk:      1
+  tags:       Elastic Stack/Machine Learning
+  subject:    Machine Learning
+  sources:
+    -
+      repo:   stack-docs
+      path:   docs/en/stack
+    -
+      repo:   elasticsearch
+      path:   docs
+    -
+      repo:   docs
+      path:   shared/versions/stack/{version}.asciidoc
+    -
+      repo:   docs
+      path:   shared/attributes.asciidoc
+    -
+      repo:   docs
+      path:   shared/settings.asciidoc
+```
+
+## V3 configuration
+
+TO DO

docs/source/configure/site/index.md

@@ -0,0 +1,24 @@
+---
+title: Site configuration
+navigation_title: Site
+---
+
+Start by understanding how the new V3 system works at the site level compared to how our custom AsciiDoctor system works. The system consists of:
+
+
+| System property | Asciidoc | V3 |
+| -------------------- | -------------------- | -------------------- |
+| **Content sources** --> Collections of markup files containing doc content. These are split up across many docs repos. | _Books_ | _Content sets_ |
+| **Content configuration** --> A way to specify where to find those content sources, and in what order they should be added to the site. | Configuration file ([`conf.yml`](https://github.com/elastic/docs/blob/master/conf.yaml) in elastic/docs) | Config file location TBD |
+| **Cross-site values** --> Key-value pairs that should be substituted across all sources as web pages are built. | Shared attributes ([`shared/`](https://github.com/elastic/docs/tree/master/shared) in elastic/docs) | Shared attrs file TBD |
+| **Docs build tool** --> An engine used to take the markup in the content sources and transform it into web pages. | Customized version of AsciiDoctor (lives in [**elastic/docs**](https://github.com/elastic/docs)) | Customized doc builder using open source tools (lives in [**elastic/docs-builder**](https://github.com/elastic/docs-builder)) |
+
+Where these pieces live and what format they are in varies between the two systems, but they generally achieve the same goal.
+
+## Asciidoc
+
+![site-level config in the asciidoc system](./img/site-level-asciidoctor.png)
+
+## V3
+
+DIAGRAM NEEDED

docs/source/configure/site/landing-page.md

@@ -0,0 +1,5 @@
+---
+title: Landing page
+---
+
+tbd

docs/source/configure/site/redirects.md

@@ -0,0 +1,26 @@
+---
+title: Redirects
+---
+
+## From an elastic.co/guide page
+
+The web team owns redirects for `elastic.co`. Writers do not have direct access to the system where www.elastic.co redirects are managed.
+
+### Permanent redirect
+
+Currently the only way to implement true redirects from a page on `elastic.co/guide` is to open an issue with the web team, add your request to a Google Sheet, and wait for the web team to address your request.
+
+### Stopgap
+
+Writers have developed a stopgap to communicate to users that content has been removed or moved to another location. We create a page at the existing URL that links to the new location of the information. This is better than removing a page without redirecting at all, but is not the best user experience and doesn't align with SEO best practices.
+
+## From an elastic.co/docs page
+
+STRATEGY TBD
+
+## Migration redirects
+
+Docs inventory script: https://github.com/elastic/docs-helpers/tree/main/docs-inventory
+Docs inventory sheet: https://docs.google.com/spreadsheets/d/1LfPI3TZqdpONGxOmL8B8V-Feo1flLwObz9_ibCEMkIQ/edit?gid=605983744#gid=605983744
+
+This sheet will be used to request redirects after migration from 8.current to 9.current.

docs/source/configure/site/shared-attributes.md

@@ -0,0 +1,29 @@
+---
+title: Shared attributes
+---
+
+To promote consistency across documentation, AsciiDoc uses shared attributes for common terms, URLs, and versions.
+
+In the AsciiDoctor-based system, shared attributes live in the [`shared/` directory](https://github.com/elastic/docs/blob/master/shared) in the elastic/docs repo. The most used files in this directory are:
+
+The [`attributes.asciidoc` file](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc), which contains URLs, common words and phrases, and more.
+The files in the [`versions/stack` directory](https://github.com/elastic/docs/tree/master/shared/versions/stack), which contain the latest versions for various products for a given Stack version.
+
+## Versions
+
+### Asciidoc
+
+File: [`shared/versions/stack/8.8.asciidoc`](https://github.com/elastic/docs/blob/master/shared/versions/stack/8.8.asciidoc) in the elastic/docs repo
+
+### V3
+
+See [Product availability](../../syntax/applies.md).
+
+
+## URLS
+
+File: [`shared/attributes.asciidoc`](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc) in the elastic/docs repo
+
+## Common terms
+
+File: [`shared/attributes.asciidoc`](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc) in the elastic/docs repo

docs/source/contribute/change-browser.md

@@ -0,0 +1,11 @@
+---
+title: Update the docs locally
+---
+
+1. Install dependencies
+2. Clone repositories
+3. Make changes
+4. Open a Pull Request
+5. Work with CI
+6. Get approvals and merge
+7. View your changes live on elastic.co

docs/source/contribute/change-local.md

@@ -0,0 +1,3 @@
+---
+title: Update the docs in your web browser
+---

docs/source/contribute/index.md

@@ -0,0 +1,10 @@
+---
+title: Elastic Docs contribution guide
+navigation_title: Contribution guide
+---
+
+Want to contribute to the Elastic documentation? You're in the right spot! Select an option below to get started:
+
+* I just want to suggest a change --> [Open a docs issue](https://github.com/elastic/docs-content/issues/new?template=internal-request.yaml)
+* I want to make a small change to a single page --> [Update the docs in your _web browser_](change-browser.md).
+* I want to make a larger change or a change to multiple pages --> [Update the docs in your _code editor_](change-local.md).

docs/source/docset.yml

@@ -19,6 +19,32 @@ exclude:
   - '_*.md'
 toc:
   - file: index.md
+  - folder: migration
+    children:
+      - file: index.md
+      - file: file-structure.md
+  - folder: contribute
+    children:
+      - file: index.md
+      - file: change-browser.md
+      - file: change-local.md
+  - folder: configure
+    children:
+      - file: index.md
+      - folder: site
+        children:
+          - file: index.md
+          - file: content.md
+          - file: shared-attributes.md
+          - file: landing-page.md
+          - file: redirects.md
+      - folder: content-set
+        children:
+          - file: index.md
+          - file: file-structure.md
+          - file: attributes.md
+          - file: navigation.md
+      - file: page.md
   - folder: syntax
     children:
       - file: index.md

docs/source/index.md

@@ -28,7 +28,7 @@ Options:
 Commands:
   generate    Converts a source markdown folder or file to an output folder
   serve       Continuously serve a documentation folder at http://localhost:5000.
-	File systems changes will be reflected without having to restart the server.
+    File systems changes will be reflected without having to restart the server.
 ```
 
 In the near term native code will be published by CI for the following platforms
@@ -44,7 +44,8 @@ And we'll invest time in making sure these are easily obtainable (`brew`, `winge
 For now you can run the tool locally through `docker run`
 
 ```bash
-docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" ghcr.io/elastic/docs-builder:edge
+docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \
+  ghcr.io/elastic/docs-builder:edge
 ```
 
 This ensures `.git`/`docs` and `.artifacts` (the default output directory) are mounted.
@@ -54,15 +55,17 @@ Only the changed files on subsequent runs will be compiled unless you pass `--fo
 to force a new compilation.
 
 ```bash
-docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" ghcr.io/elastic/docs-builder:edge --force
+docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \
+  ghcr.io/elastic/docs-builder:edge --force
 ```
 
 #### Live mode
 
 Through the `serve` command you can continuously and partially compile your documentation.
 
 ```bash
-docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" --expose 8080 ghcr.io/elastic/docs-builder:edge serve
+docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \
+  -p 8080:8080 ghcr.io/elastic/docs-builder:edge serve
 ```
 
 Each page is compiled on demand as you browse http://localhost:8080 and is never cached so changes to files and
@@ -130,18 +133,20 @@ https://github.com/elastic/{your-repository}/settings/pages
 
 ## Run without docker
 
-If you have dotnet 8 installed you can use its CLI to publish a self-contained `docs-builder` native code
+You can use the .NET CLI to publish a self-contained `docs-builder` native code
 binary. (On my M2 Pro mac the binary is currently 16mb)
 
+Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run:
+
 ```bash
-$ dotnet publish "src/docs-builder/docs-builder.csproj"
+dotnet publish "src/docs-builder/docs-builder.csproj"
 ```
 
-The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed
+The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed.
 
 # Performance
 
 To test performance it's best to build the binary and run outside of docker:
 
-For refence here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to
+For reference here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to
 existing surveyed tools