Build configuration files
Switch branches/tags
Nothing to show
Clone or download
beanpole135 Update to build-distro.sh (and README)
1. Optimize the checkout procedures to be "smart" about when to download/use a new version of a source repo.
1. If a github commit tag was used, do not re-download/extract the source tree everytime (should make the build for world/kernel much faster and re-use the results of the previous build)
2. If a github branch was used, download the new tarball, but compare to the previous one and do not extract if there is no difference from the previous build (again - save compilation time for build later).
3. Any overlays are extracted on top of the source tree *every time*, so ports overlays will always be "clean" and built again as needed.
Latest commit 4ade064 Sep 19, 2018

README.md

trident-build

Simple build system for creating a TrueOS distribution

Directories/Scripts

  • setup-build-server/setup.sh
    • Simple script to provision a TrueOS system as a build server for TrueOS distributions.
    • This will install jenkins, nginx, git, and rsync, and setup the nginx server for viewing the poudriere build logs
    • It will also provide a couple ZFS tuning options which require a reboot to apply.
    • MAKE SURE YOU TWEAK THE SCRIPT VALUES BEFORE RUNNING
  • setup-package-server/setup.sh
    • Simple script to provision a FreeBSD/TrueOS system as a package server.
    • This will install nginx and set it up to serve a data directory (/data)
    • MAKE SURE YOU TWEAK THE SCRIPT VALUES BEFORE RUNNING

Files:

  • trident-master.json
    • JSON manifest used to build Project Trident
    • Ensure the package repo URLs match the package server setup
  • Jenkinsfile-trident-master
    • Pipeline script for Jenkins to manage the build of Project Trident and push files to a remote server.
    • Ensure the publication dirs align with the package server setup
  • Jenkinsfile-trident-promote
    • Pipeline script for Jenkins to promote "staged" packages/iso on remote server to release.
    • Ensure the remote directory structure aligns with the package server setup

build-distro.sh

Primary script to run to perform builds. Syntax:

  • build-distro.sh <command> [JSON Manifest File] Note: The JSON manifest file can also be supplied via the "TRUEOS_MANIFEST" environment variable. That manifest must be provided in order to perform the build.

Commands:

  • all : Perform the following stages (in-order): checkout, world, kernel, base, ports, release, manifest, sign_artifacts
  • clean : Cleanup any temporary working directories and output dirs/files
    • OPTIONAL : The checkout phase will automatically clean source directories as needed based upon changes to the source/ports commit tags. This can be run manually to forcibly re-build the world/kernel as desired.
  • checkout : Fetch/extract the base/ports repositories (cached by tag - will only re-download if the tag changes)
  • world : Build FreeBSD world. Corresponds to "make buildworld"
  • kernel : Build FreeBSD kernel. Corresponds to "make buildkernel"
  • base : Build/sign base packages. Corresponds to "make packages". The "PKGSIGNKEY" or "PKG_REPO_SIGNING_KEY" environment variable must be set for this stage in order to sign the base packages.
  • ports : Build/sign ports packages. Corresponds to "cd release && make poudriere". The "PKGSIGNKEY" or "PKG_REPO_SIGNING_KEY" environment variable must be set for this stage in order to sign the packages.
  • release : Build ISO files and artifacts. Corresponds to "cd release && make release".
  • manifest : Generate a manifest of all non-base packages ("artifacts/pkg.list")
  • sign_artifacts : Sign all the ISO files (if a key is provided), generate makesums, and generate a manifest.json file containing links/information about all the ISO artifact files.

Extra supported/required fields in the TrueOS JSON manifest:

NOTE: The macro "%%PWD%%" can be used anywhere in the JSON manifest to insert the path to the directory which contains the JSON manifest file. This is useful if an option in the manifest requires a full path (such as a "local" iso-overlay setting), but the location of the build repository might be programmatically generated via automation procedures.

Additional JSON manifest fields

  • "base-github-org" : [Required] (string) Name of the organization on GitHub (example: "trueos")
  • "base-github-repo" : [Required] (string) Name of the repository on GitHub (example: "trueos")
  • "base-github-tag" : [Required] (string) Tag name or commit ID to fetch from the GitHub org/repo
  • "ports-github-org" : [Optional] (string) Name of the organization on GitHub (example: "trueos")
  • "ports-github-repo" : [Optional] (string) Name of the repository on GitHub (example: "trueos")
  • "ports-github-tag" : [Optional] (string) Tag name or commit ID to fetch from the GitHub org/repo
  • "iso-name" : (string) Base name of the ISO to create (example: "mydistro")
    • [Optional] Default value is the name of the JSON manifest file ("mydistro.json" -> "mydistro-[BuildDate].iso")
  • "ports-overlay" : [Optional] (Array of JSON objects) paths to directories to add/replace items in the ports tree
    • [WARNING] This is only possible if the "ports-github-*" mechanism is used to fetch the ports repository.
    • Syntax for objects within the array:
      • "type" : (string) Either "category" (adding a new category to the ports tree) or "port" (adding a single port to the tree)
      • "name" : (string) Category name ("mydistro") or port origin ("devel/myport") depending on the type of overlay.
      • "local_path" : (string) path to the local directory which will be used as the overlay.
      • Example:
"ports-overlay" : [
  {
    "type" : "category",
    "name" : "mydistro",
    "local_path" : "overlay/mydistro"
  },
  {
    "type" : "port",
    "name" : "devel/myport",
    "local_path" : "overlay/devel/myport"
  }
]

[WARNING] If you use the "ports-github-" manifest options, you need to ensure to set the TrueOS ports type to "local" and the url to "/usr/ports_tmp". Those options allow checking out specific tags/commits, and bypass the built-in ports repo checkout procedures within TrueOS. If you want to use a standard tarball or git branch, enable the standard TrueOS port options and remove the "ports-github-" options from the manifest.

Supported environment variables (inputs/overrides)

  • "PKGSIGNKEY" or "PKG_REPO_SIGNING_KEY" [optional]
    • Format: String with the contents of the private SSL key to use when signing the packages.
    • PKGSIGNKEY: Used during the "base" process to sign FreeBSD base packages in addition to the "sign_artifacts" process to sign the ISO file.
    • PKG_REPO_SIGNING_KEY: Used during the "ports" process to sign FreeBSD ports packages
    • Note: If only one of these variables is set, it will automatically copy/use it for the other as well.
  • "MAX_THREADS" [optional]
    • Format: Integer (number of threads to use, 1 or higher)
    • This is passed via the "-j[number]" flag to the build procedures to speed up the compilation processes.
    • Default value: One less than the detected number of CPU's on the system (sysctl -n hw.ncpu): Example: An 8-core system will result in MAX_THREADS getting automatically set to 7.

Output Files

  • ISO Build artifacts (ISO, MANIFEST, various *.tgz) will be placed in a new "artifact-iso/" subdirectory relative to the location of the build-distro.sh script.
  • "Ports" Package Files will be linked to a new "artifact-pkg/" subdirectory relative to the location of the build-distro.sh script (symlink to the actual poudriere dir on disk).
  • "Base" Package Files will be linked to a new "artifact-pkg-base/" subdirectory relative to the location of the build-distro.sh script (symlink to the actual base-package dir on disk).

General Notes about creating a TrueOS distribution

Signing Packages

  1. Create a private SSL Key: openssl genrsa -out my_private_key.key [2048/4096/8192]
  2. Save the public version of that key: openssl rsa -in my_private_key.key -pubout > my_public_key.key
  3. Copy the contents of the public key file into the JSON manifest ("pkg-repo" and "base-pkg-repo" sections - add the "pubkey" variable which contains an array of the lines of the public key file.

Settings up Jenkins automation framework

  1. You will need an SSH key to use when publishing files to a remote distribution server/system:
    • To create one, run ssh-keygen and follow the prompts.
    • Then add the public key to the distribution system so it can be used for login authentication.
  2. Add the SSH key and SSL private key to the Jenkins instance and get the credential ID number for each one.
  3. Copy one of the "Jenkinsfile-*" examples from the trueos/trueos repository.
  4. In the new jenkins file, adjust all the "credentials('*')" entries with the credential ID's for your keys.
  5. In the "Publish" stage of the jenkins file, adjust the user, server, and directories as needed for your distribution system.

Custom Branding at bootup

There are 3 files which need to be created to brand the boot menu:

  1. (trueos repository): stand/lua/brand-[distro].lua (copy/modify the brand-trueos.lua file)
  2. (trueos repository): stand/lua/logo-[distro].lua (copy/modify one of the other logo-.lua* files)
  3. (trueos repository): Add the new files to the list in stand/lua/Makefile
  4. (overlay file): Add the following entries to /boot/loader.conf.local (you may need to create this file):
loader_brand="[distro]"
loader_logo="[distro]"
loader_menu_title="Disto Title"
loader_color="[YES/NO]"

The trueos repository files (brand-.lua, logo-.lua) can be submitted upstream to the TrueOS repo to reduce the management overhead of alterations in the forked repository.