The Welcome Guide, or: blergh "What's New" back from armbian-next

docs/Developer-Guide_Welcome.md

@@ -0,0 +1,127 @@
+# Welcome to `armbian/build`, the Armbian build system
+
+Welcome to the Armbian build system (`armbian/build`).
+The build system has significantly changed since release 23.02, when the `armbian-next` effort was merged into
+our `main` branch.
+Please expect some rough edges and report them to us.
+
+## Errors, errors everywhere
+
+Previously, when faced with an error, the build system would simply ignore them, hardly log them anywhere, and continue.
+This led to builds completing successfully, but with (sometimes very sneakily) broken packages and images.
+Get used to seeing errors stopping builds now. Inspect the whole log, specially the errors, the last few lines, and the
+stack trace.
+We try, as much as possible given Bash limitations, to show the source of the error, with a "stack trace" of sorts.
+The topmost elements of the stack trace are usually the most relevant. Each line is prefixed with the file and line
+number where the error occurred.
+Open the file and go to the line number to see the source of the error. Most times, this simple inspection is enough to
+understand the error.
+
+### (ANSI) Logging
+
+Logging is a bit more structured now. Logs are output to `output/logs`, in a few different formats.
+We output ANSI color both to the screen and to the logs.
+_Please_ add `SHARE_LOG=yes` to share your logs with us when reporting issues, that allows us to check the logs on a web
+browser and keep to correct formatting.
+
+## Command line syntax has changed
+
+General CLI syntax: `./compile.sh PARAM=value OTHER_PARAM=other_value [<configfile> <configfile> ...] [<command>]`
+
+- where `command` defaults to `build` if you don't specify it; could also be `kernel-config` or `u-boot` etc
+- you _can't_ have a config file with the same name as a possible `<command>` (system will check & bomb if so)
+- also: there's no more `default` config -- you have to be explicit
+- also: there's no more `docker` config -- Docker is fully auto-managed now. The system will complain if you have one.
+- you can specify `PARAM=value`, `<configfile>` or `<command>` in _any order_
+
+## No more `config-default.conf`, you need to specify the config file in the command line
+
+- No "default" config is auto-loaded anymore. Default config lead to unreproducible failing builds and was a source of
+  confusion.
+- The configs still go to the same place, `userpatches/config-xyz.conf` -- but now you gotta tell the build system to
+  use that config, like `./compile.sh BOARD=xxx xyz`; otherwise works the same.
+
+## Artifacts, cache, what the ...?
+
+The `armbian/build` system is currently undergoing refactoring to improve its structure. Previously, the build system
+was a single, complex Bash script that mixed the building of `.deb` packages with the creation of images.
+
+This was reworked into a `1-to-N` image-to-artifact dependency tree; a certain image build will depend on N possible
+"artifacts". Artifacts are either `.deb` packages, a `.tar` of multiple `.deb` packages, or a `rootfs.tar.zstd`. Each
+artifact can be individually built, and has a specific name and a _version_.
+
+Each artifact is also now **cached by default** using OCI storage at ghcr.io (GitHub Container Registry). To achieve
+_consistent caching_, each artifact produces a version that includes _hashes_ of its composing files, variables,
+patches,
+hooks, external git SHA1 references, etc. That way we can consistently check the remote OCI cache for previously-built
+artifacts, and possibly save image builders from having to build heavy packages just to produce an image.
+
+### TL-DR about artifacts and caching:
+
+- `KERNEL_ONLY=yes` and `KERNEL_ONLY=no` are no more. Use the `kernel` CLI command instead.
+- `ARTIFACT_IGNORE_CACHE=yes` can help if you're getting false positives. Please also report the problem, with a
+  complete logfile.
+
+## Automatic Docker/sudo launcher
+
+- `compile.sh` will prefer to use Docker if it detects Docker is installed and working.
+    - This handles Docker Desktop and Rancher Desktop (in Docker emulation mode) under macOS/Darwin, including Apple
+      M1/M2.
+    - You **don't need and actually can't have the old docker config file**.
+- If Docker is not installed, it will try to use `sudo` to run the build as root.
+- If you run directly as root, it will warn you and ask you to run it without sudo.
+
+## Kernel Git Trees: shallow vs full
+
+During the build, depending on which local or remote caches are hit, it might be needed to build the Linux Kernel.
+
+The kernel's git repo is huge. Most build systems resort to fetching "shallow" trees directly from upstream git servers,
+to save bandwidth. Unfortunately that creates immense extra CPU load on the git servers. To avoid this problem,
+Armbian produces daily automated git tree exports cached in ghcr.io OCI repositories, and only uses `git fetch` to
+update the relatively small new changes from the upstream git server.
+
+There are two types of cached Kernel git trees:
+
+- `full` is a complete git tree, including all of Torvald's `master` and all of the currently-supported `stable`
+  branches.
+    - `full` is very large download and requires a lot of disk space.
+    - `full` is more useful over time and when building multiple different kernels on the same machine, like for CI
+      servers or developer workstations.
+- `shallow` is a shallow tree for a specific `stable` branch
+    - `shallow` is a much smaller download and requires less disk space
+    - `shallow` is appropriate for restricted devices like SBCs which will build a single kernel
+
+**TL,DR: ** `KERNEL_GIT=full` or `KERNEL_GIT=shallow` or let the system decide for you.
+
+## Consider Forking before cloning the repo
+
+Before cloning the repo, consider forking it first. This will allow you to make changes and submit pull requests.
+You will need a GitHub account to do this;
+see [GitHub's documentation](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) for more
+information.
+If you fork, make sure to keep your fork up-to-date with the main repo, by rebasing your fork.
+
+## Some really confusing stuff still remains
+
+This is (by far) not a complete list:
+
+- wifi/other kernel drivers are still using pre-armbian-next code, and are very hard to work with. it's not only the
+  contents are a mess, the way the whole thing works leads to more and more compounding work. To make it worse, family
+  patches sometimes need to patch driver code, leading to a cycle of sadness for developers. We're still coming up with
+  a plan to completely replace this lest most of us go insane.
+- although "aggregation" has been rewritten in Python, it still mostly works using the legacy principle, by scanning
+  directories and files in a very complex and error-prone way. This is a source of many bugs and confusion. We plan to
+  replace this with pure extensions eventually.
+- "minimal" images are far from really minimal. It is common to see "extensions" that actually remove stuff put there
+  by the core. Those should be refactored.
+- board-side scripts (armbian-config, firstrun, hwoptimization, etc) haven't really changed with armbian-next, and are
+  in dire need of a rewrite. They're also a source of many bugs and confusion. They also need to be extensible.
+- we've mostly working Kernel headers (linux-headers pkg) for 5.10+ including some vendor kernels; no kernel-headers for
+  4.x kernels. sorry. We've no plans to support this.
+
+## Multiple u-boot's for same board
+
+Hmm. This is an interesting problem. We can build u-boot twice, using `UBOOT_TARGET_MAP`.
+Some example I did in https://github.com/armbian/build/blob/main/config/boards/odroidhc4.conf#L15-L20 may help.
+
+

mkdocs.yml

@@ -42,11 +42,11 @@ extra:
 
 plugins:
   - search
-  - with-pdf:
-      author: Armbian documentation team
-      copyright: © 2023 by Armbian
-      cover_title: Armbian documentation
-      cover_subtitle: Linux for ARM development boards
+  #- with-pdf:
+  #    author: Armbian documentation team
+  #    copyright: © 2023 by Armbian
+  #    cover_title: Armbian documentation
+  #    cover_subtitle: Linux for ARM development boards
 
 markdown_extensions:
   - smarty
@@ -88,7 +88,7 @@ nav:
       - "Rockchip": "Hardware_Rockchip.md"
 
   - "Developer Guide":
-      - "Development Code Review Procedures and Guidelines": "Development-Code_Review_Procedures_and_Guidelines.md"
+      - "Welcome": "Developer-Guide_Welcome.md"
       - "Building Armbian": "Developer-Guide_Build-Preparation.md"
       - "Building with Docker": "Developer-Guide_Building-with-Docker.md"
       - "Build Options": "Developer-Guide_Build-Options.md"
@@ -102,6 +102,7 @@ nav:
       - "Merge Policy": "Process_Merge-Policy.md"
       - "Continuous integration": "Process_CI.md"
       - "Documentation": "Process_Documentation.md"
+      - "Development Code Review Procedures and Guidelines": "Development-Code_Review_Procedures_and_Guidelines.md"
 
   - "Release management":
       - "Model": "Process_Release-Model.md"