diff --git a/docs/Developer-Guide_Welcome.md b/docs/Developer-Guide_Welcome.md new file mode 100644 index 00000000..94eac341 --- /dev/null +++ b/docs/Developer-Guide_Welcome.md @@ -0,0 +1,127 @@ +# Welcome to `armbian/build`, the Armbian build system + +Welcome to the Armbian build system (`armbian/build`). +The build system has significantly changed since release 23.02, when the `armbian-next` effort was merged into +our `main` branch. +Please expect some rough edges and report them to us. + +## Errors, errors everywhere + +Previously, when faced with an error, the build system would simply ignore them, hardly log them anywhere, and continue. +This led to builds completing successfully, but with (sometimes very sneakily) broken packages and images. +Get used to seeing errors stopping builds now. Inspect the whole log, specially the errors, the last few lines, and the +stack trace. +We try, as much as possible given Bash limitations, to show the source of the error, with a "stack trace" of sorts. +The topmost elements of the stack trace are usually the most relevant. Each line is prefixed with the file and line +number where the error occurred. +Open the file and go to the line number to see the source of the error. Most times, this simple inspection is enough to +understand the error. + +### (ANSI) Logging + +Logging is a bit more structured now. Logs are output to `output/logs`, in a few different formats. +We output ANSI color both to the screen and to the logs. +_Please_ add `SHARE_LOG=yes` to share your logs with us when reporting issues, that allows us to check the logs on a web +browser and keep to correct formatting. + +## Command line syntax has changed + +General CLI syntax: `./compile.sh PARAM=value OTHER_PARAM=other_value [ ...] []` + +- where `command` defaults to `build` if you don't specify it; could also be `kernel-config` or `u-boot` etc +- you _can't_ have a config file with the same name as a possible `` (system will check & bomb if so) +- also: there's no more `default` config -- you have to be explicit +- also: there's no more `docker` config -- Docker is fully auto-managed now. The system will complain if you have one. +- you can specify `PARAM=value`, `` or `` in _any order_ + +## No more `config-default.conf`, you need to specify the config file in the command line + +- No "default" config is auto-loaded anymore. Default config lead to unreproducible failing builds and was a source of + confusion. +- The configs still go to the same place, `userpatches/config-xyz.conf` -- but now you gotta tell the build system to + use that config, like `./compile.sh BOARD=xxx xyz`; otherwise works the same. + +## Artifacts, cache, what the ...? + +The `armbian/build` system is currently undergoing refactoring to improve its structure. Previously, the build system +was a single, complex Bash script that mixed the building of `.deb` packages with the creation of images. + +This was reworked into a `1-to-N` image-to-artifact dependency tree; a certain image build will depend on N possible +"artifacts". Artifacts are either `.deb` packages, a `.tar` of multiple `.deb` packages, or a `rootfs.tar.zstd`. Each +artifact can be individually built, and has a specific name and a _version_. + +Each artifact is also now **cached by default** using OCI storage at ghcr.io (GitHub Container Registry). To achieve +_consistent caching_, each artifact produces a version that includes _hashes_ of its composing files, variables, +patches, +hooks, external git SHA1 references, etc. That way we can consistently check the remote OCI cache for previously-built +artifacts, and possibly save image builders from having to build heavy packages just to produce an image. + +### TL-DR about artifacts and caching: + +- `KERNEL_ONLY=yes` and `KERNEL_ONLY=no` are no more. Use the `kernel` CLI command instead. +- `ARTIFACT_IGNORE_CACHE=yes` can help if you're getting false positives. Please also report the problem, with a + complete logfile. + +## Automatic Docker/sudo launcher + +- `compile.sh` will prefer to use Docker if it detects Docker is installed and working. + - This handles Docker Desktop and Rancher Desktop (in Docker emulation mode) under macOS/Darwin, including Apple + M1/M2. + - You **don't need and actually can't have the old docker config file**. +- If Docker is not installed, it will try to use `sudo` to run the build as root. +- If you run directly as root, it will warn you and ask you to run it without sudo. + +## Kernel Git Trees: shallow vs full + +During the build, depending on which local or remote caches are hit, it might be needed to build the Linux Kernel. + +The kernel's git repo is huge. Most build systems resort to fetching "shallow" trees directly from upstream git servers, +to save bandwidth. Unfortunately that creates immense extra CPU load on the git servers. To avoid this problem, +Armbian produces daily automated git tree exports cached in ghcr.io OCI repositories, and only uses `git fetch` to +update the relatively small new changes from the upstream git server. + +There are two types of cached Kernel git trees: + +- `full` is a complete git tree, including all of Torvald's `master` and all of the currently-supported `stable` + branches. + - `full` is very large download and requires a lot of disk space. + - `full` is more useful over time and when building multiple different kernels on the same machine, like for CI + servers or developer workstations. +- `shallow` is a shallow tree for a specific `stable` branch + - `shallow` is a much smaller download and requires less disk space + - `shallow` is appropriate for restricted devices like SBCs which will build a single kernel + +**TL,DR: ** `KERNEL_GIT=full` or `KERNEL_GIT=shallow` or let the system decide for you. + +## Consider Forking before cloning the repo + +Before cloning the repo, consider forking it first. This will allow you to make changes and submit pull requests. +You will need a GitHub account to do this; +see [GitHub's documentation](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) for more +information. +If you fork, make sure to keep your fork up-to-date with the main repo, by rebasing your fork. + +## Some really confusing stuff still remains + +This is (by far) not a complete list: + +- wifi/other kernel drivers are still using pre-armbian-next code, and are very hard to work with. it's not only the + contents are a mess, the way the whole thing works leads to more and more compounding work. To make it worse, family + patches sometimes need to patch driver code, leading to a cycle of sadness for developers. We're still coming up with + a plan to completely replace this lest most of us go insane. +- although "aggregation" has been rewritten in Python, it still mostly works using the legacy principle, by scanning + directories and files in a very complex and error-prone way. This is a source of many bugs and confusion. We plan to + replace this with pure extensions eventually. +- "minimal" images are far from really minimal. It is common to see "extensions" that actually remove stuff put there + by the core. Those should be refactored. +- board-side scripts (armbian-config, firstrun, hwoptimization, etc) haven't really changed with armbian-next, and are + in dire need of a rewrite. They're also a source of many bugs and confusion. They also need to be extensible. +- we've mostly working Kernel headers (linux-headers pkg) for 5.10+ including some vendor kernels; no kernel-headers for + 4.x kernels. sorry. We've no plans to support this. + +## Multiple u-boot's for same board + +Hmm. This is an interesting problem. We can build u-boot twice, using `UBOOT_TARGET_MAP`. +Some example I did in https://github.com/armbian/build/blob/main/config/boards/odroidhc4.conf#L15-L20 may help. + + diff --git a/mkdocs.yml b/mkdocs.yml index 57263561..0c136860 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,11 +42,11 @@ extra: plugins: - search - - with-pdf: - author: Armbian documentation team - copyright: © 2023 by Armbian - cover_title: Armbian documentation - cover_subtitle: Linux for ARM development boards + #- with-pdf: + # author: Armbian documentation team + # copyright: © 2023 by Armbian + # cover_title: Armbian documentation + # cover_subtitle: Linux for ARM development boards markdown_extensions: - smarty @@ -88,7 +88,7 @@ nav: - "Rockchip": "Hardware_Rockchip.md" - "Developer Guide": - - "Development Code Review Procedures and Guidelines": "Development-Code_Review_Procedures_and_Guidelines.md" + - "Welcome": "Developer-Guide_Welcome.md" - "Building Armbian": "Developer-Guide_Build-Preparation.md" - "Building with Docker": "Developer-Guide_Building-with-Docker.md" - "Build Options": "Developer-Guide_Build-Options.md" @@ -102,6 +102,7 @@ nav: - "Merge Policy": "Process_Merge-Policy.md" - "Continuous integration": "Process_CI.md" - "Documentation": "Process_Documentation.md" + - "Development Code Review Procedures and Guidelines": "Development-Code_Review_Procedures_and_Guidelines.md" - "Release management": - "Model": "Process_Release-Model.md"