Initial working version

This is a working prototype that can be used to create working images from valid config files.

Fixes:
- Specify the card/project configuration format & workflow #4
- Implement Basic program structure #7
- Implement image file manipulation #10

Starts without completing:
- Implement Progress Reporting #8: text UI (cli) is done and usable. Machine-readable JSON file is not but can be implemented via the logger and StepMachine.
- Implement Downloader #9: a basic requests-based downloader is present. Needs more work to support transfer issues and being interupted

TBC

.github/workflows/build.yaml

@@ -0,0 +1,60 @@
+name: Build
+
+on:
+  push:
+    branches:
+      - main
+      - initial
+
+env:
+  SSH_KEY: /tmp/id_rsa
+
+jobs:
+  check-qa:
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2.5.0
+      - name: Setup Python
+        uses: actions/setup-python@v2.3.3
+        with:
+          python-version: "3.10"
+          architecture: x64
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install --no-install-recommends -y python3-dev patchelf ccache build-essential
+      - name: Install python dependencies
+        run: |
+          pip install -U pip
+          pip install -U ordered-set zstandard nuitka
+          pip install -r requirements.txt
+      - name: Update environ (filename)
+        run: |
+          import os
+          ref = os.getenv("GITHUB_REF", "").split("/")[-1]
+          env = {"FILENAME": f"image-creator_{ref}"}
+          with open(os.getenv("GITHUB_ENV"), "a") as fh:
+            for name, value in env.items():
+              fh.write(f"{name}={value}\n")
+        shell: python
+      - name: Build image-creator
+        run: |
+          cd src
+          python -m nuitka \
+            --onefile \
+            --python-flag="no_site,no_warnings,no_asserts,no_docstrings" \
+            --warn-implicit-exceptions \
+            --warn-unusual-code \
+            --assume-yes-for-downloads \
+            --output-filename=$PWD/$FILENAME \
+            --remove-output \
+            --no-progressbar \
+            image_creator/
+      - name: Dump SSH Key to file
+        shell: bash
+        run: |
+          echo "${{secrets.ssh_key}}" > $SSH_KEY
+          chmod 600 $SSH_KEY
+      - name: Upload build to CI
+        run: scp -rp -P 30022 -i $SSH_KEY -o StrictHostKeyChecking=no ci@tmp.kiwix.org:/data/tmp/ci/

.github/workflows/qa.yml

@@ -15,7 +15,7 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v2.3.3
         with:
-          python-version: 3.10
+          python-version: "3.10"
           architecture: x64
       - name: Check black formatting
         run: |

README.md

@@ -1,80 +1,94 @@
 # image-creator
 
-Hotspot image creator to build OLIP or Kiwix Offspot off [`base-image`](https://github.com/offspot/base-image).
-
-## Scope
-
-- Validate inputs
-- Download base image
-- Resize image to match contents
-- Download contents into mounted `/data`
-- Post-process downloaded contents
-- Configure from inputs
-- Re-generate SSH server keys
-- *Pull* application images
-- Prepares JSON config
-
-## Inputs
-
-- Target system (OLIP or Offspot)
-- Image name
-- Hostname
-- domain name
-- SSID
-- WiFi AP password (if any)
-- WiFi Country code
-- WiFi channel
-- Timezone
-- SSH Public keys to add
-- VPN configuration (tinc)
-- Contents
-
-## App Containers
-
-- **OLIP**
- - API
- - Frontend
- - Stats
- - Controller
-- **Offspot**
- - Kiwix-serve
- - WikiFundi (en/fr/es)
- - Aflatoun (en/fr)
- - Surfer
-- IPFS daemon
-- Captive portal
-
-## data partition
-
-
-| /data subfolders | Usage |
-|---|---|
-| `offspot/zim` | Offspot Kiwix serve ZIM files|
-| `offspot/wikifundi` | Offspot WikiFundi data |
-| `offspot/files` | Offspot Surfer data |
-| `offspot/xxx` | Offspot data for other apps |
-| `olip` | OLIP data |
-
-
-## JSON Configurator
-
-JSON config file at `/boot/config.json` is read and parsed on startup by the boot-time config script.
-It looks for the following properties. Dotted ones means nested.
-
-Behavior is to adjust configuration only if the property is present. Script will remove property from JSON once applied.
-
-Configurator is also responsible for resizing `/data` partition to device size on first boot but this is not configurable via JSON.
-
-| Property| Type | Usage |
-|---|---|---|
-| `hostname` | `string` | Pi host name |
-| `domain` | `string` | FQDN to answer to on DNS |
-| `wifi.ssid` | `string` | WiFi SSID |
-| `wifi.password` | `string` | WiFi password (clear). If `null`, auth not required |
-| `wifi.country-code` | `string` | ISO-639-2 Country code for WiFI |
-| `wifi.channel` | `int` | 1-11 channel for WiFi |
-| `timezone` | `string` | Timezone to configure date with |
-| `ssh-keys` | `string[]` | List of public keys to add to user |
-| `tinc-vpn` | `string` | tinc-VPN configuration |
-| `env.all`  | `string[]` | List of `KEY=VALUE` environment variables to pass to **all applications** |
-| `env.xxx`  | `string[]` | List of `KEY=VALUE` environment variables to pass **containers matching _xxx_** |
+RaspberryPi image creator to build OLIP or Kiwix Hotspot off [`base-image`](https://github.com/offspot/base-image).
+
+[![CodeFactor](https://www.codefactor.io/repository/github/offspot/image-creator/badge)](https://www.codefactor.io/repository/github/offspot/image-creator)
+[![Build Status](https://github.com/offspot/image-creator/actions/workflows/build.yml/badge.svg?branch=main)](https://github.com/offspot/image-creator/actions/workflows/build.yml?query=branch%3Amain)
+[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
+
+
+## Usage
+
+`image-creator` is to be **ran as `root`**.
+
+```
+❯ image-creator --help
+usage: image-creator [-h] [--build-dir BUILD_DIR] [-C] [-K] [-X] [-T CONCURRENCY] [-D] [-V] CONFIG_SRC OUTPUT
+
+create an Offspot Image from a config
+
+positional arguments:
+  CONFIG_SRC            Offspot Config YAML file path or URL
+  OUTPUT                Where to write image to
+
+options:
+  -h, --help            show this help message and exit
+  --build-dir BUILD_DIR
+                        Directory to store temporary files in, like files that needs to be extracted. Defaults to some place within /tmp
+  -C, --check           Only check inputs, URLs and sizes. Don't download/create image.
+  -K, --keep            [DEBUG] Don't remove output image if creation failed
+  -X, --overwrite       Don't fail on existing output image: remove instead
+  -T CONCURRENCY, --concurrency CONCURRENCY
+                        Nb. of threads to start for parallel downloads (at most one per file). `0` (default) for auto-selection based on CPUs.
+                        `1` to disable concurrency.
+  -D, --debug
+  -V, --version         show program's version number and exit
+```
+
+
+## Configuration
+
+Image configuration is done through a YAML file which must match the following format. Only `base` is required.
+
+
+
+| Member           | Kind           | Function                                                                                                                                                           |
+|------------------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `base`           | `string`       | Version ([official releases](https://drive.offspot.it/base/)) or URL to a base-image file. Accepts `file://` URLs. Accepts lzma encoded images using `.xz` suffix  |
+| `output.size`    | `string`/`int` | Requested size of output image. Accepts `auto` for an power-of-2 sized that can fit the content (⚠️ TBI)                                                           |
+| `oci_images`     | `string[]`     | List of OCI Image names. More specific the better. Prefer ghcr.io if possible                                                                                      |
+| `files`          | `file[]`       | List of files to include on the data partition. See below. One of `url` or `content` must be present                                                               |
+| `files[].url`    | `string`       | URL to download file from                                                                                                                                          |
+| `files[].to`     | `string`       | [required] Path to store file at. Must be a descendent of `/data`                                                                                                  |
+| `files[].content`| `string`       | Text content of the file to write. Replaces `url` if present                                                                                                       |
+| `files[].via`    | `string`       | For `url`-based files, transformation to apply on downloaded file: `direct` (default): simple download, `bztar`, `gztar`, `tar`, `xztar`, `zip` to expand archives |
+| `files[].size`   | `string`/`int` | **Only for `untar`/`unzip`** should file be compressed. Specify expanded size. Assumes File-size (uncompressed) if not specified. ⚠️ Fails if lower than file size |
+| `write_config`   | `bool`         | Whether to write this file to `/data/conf/image.yaml`                                                                                                              |
+| `offspot`        | `dict`         | [runtime-config](https://github.com/offspot/runtime-config) configuration. Will be parsed and dumped to `/boot/offspot.yaml`                                       |
+
+### Sample
+
+```yaml
+---
+base: 1.0.0
+output:
+  size: 8G
+oci_images:
+- ghcr.io/offspot/kiwix-serve:dev
+files:
+- url: http://download.kiwix.org/zim/wikipedia_fr_test.zim
+  to: /data/contents/zims/wikipedia_fr_test.zim
+  via: direct
+- to: /data/conf/message.txt
+  content: |
+    hello world
+wite_config: true
+offspot:
+  timezone: Africa/Bamako
+  ap:
+    ssid: Kiwix Offspot
+    as-gateway: true
+    domain: demo
+    tld: offspot
+  containers:
+    services:
+      kiwix:
+        container_name: kiwix
+        image: ghcr.io/offspot/kiwix-serve:dev
+        command: /bin/sh -c "kiwix-serve /data/*.zim"
+        volumes:
+          - "/data/content/zims:/data:ro"
+        ports:
+          - "80:80"
+
+```

requirements.txt

@@ -0,0 +1,6 @@
+requests==2.28.1
+PyYAML==6.0
+cli-ui==0.17.2
+humanfriendly==10.0
+progressbar2==4.2.0
+docker_export==0.4

src/image_creator/__init__.py

@@ -0,0 +1 @@
+__version__ = "1.0.0.dev0"

src/image_creator/__main__.py

@@ -0,0 +1,3 @@
+from image_creator.entrypoint import main
+
+main()

src/image_creator/constants.py

@@ -0,0 +1,91 @@
+import logging
+import pathlib
+import re
+import sys
+import tempfile
+import urllib.parse
+from dataclasses import dataclass
+from typing import Union
+
+from image_creator import __version__ as vers
+from image_creator.logger import Logger
+
+# where will data partition be monted on final device.
+# used as reference for destinations in config file and in the UI
+DATA_PART_PATH = pathlib.Path("/data")
+# version of the python interpreter
+pyvers = ".".join([str(p) for p in sys.version_info[:3]])
+banner: str = rf"""
+  _                                                      _
+ (_)_ __ ___   __ _  __ _  ___        ___ _ __ ___  __ _| |_ ___  _ __
+ | | '_ ` _ \ / _` |/ _` |/ _ \_____ / __| '__/ _ \/ _` | __/ _ \| '__|
+ | | | | | | | (_| | (_| |  __/_____| (__| | |  __/ (_| | || (_) | |
+ |_|_| |_| |_|\__,_|\__, |\___|      \___|_|  \___|\__,_|\__\___/|_|
+                    |___/                                       v{vers}|py{pyvers}
+
+"""
+
+
+@dataclass(kw_only=True)
+class Options:
+    """Command-line options"""
+
+    CONFIG_SRC: str
+    OUTPUT: str
+    BUILD_DIR: str
+
+    check_only: bool
+    debug: bool
+
+    config_path: pathlib.Path = None
+    output_path: pathlib.Path = None
+    build_dir: pathlib.Path = None
+
+    keep_failed: bool
+    overwrite: bool
+    concurrency: int
+
+    config_url: urllib.parse.ParseResult = None
+    logger: Logger = Logger()
+
+    def __post_init__(self):
+        if re.match(r"^https?://", self.CONFIG_SRC):
+            self.config_url = urllib.parse.urlparse(self.CONFIG_SRC)
+        else:
+            self.config_path = pathlib.Path(self.CONFIG_SRC).expanduser().resolve()
+
+        if self.debug:
+            self.logger.setLevel(logging.DEBUG)
+
+        if not self.check_only:
+            self.output_path = pathlib.Path(self.OUTPUT).expanduser().resolve()
+
+        if not self.BUILD_DIR:
+            # holds reference to tempdir until Options is released
+            # and will thus automatically remove actual folder
+            self.__build_dir = tempfile.TemporaryDirectory(
+                prefix="image-creator_build-dir", ignore_cleanup_errors=True
+            )
+        self.build_dir = (
+            pathlib.Path(self.BUILD_DIR or self.__build_dir.name).expanduser().resolve()
+        )
+
+    @property
+    def version(self):
+        return vers
+
+    @property
+    def config_src(self) -> Union[pathlib.Path, urllib.parse.ParseResult]:
+        return self.config_url or self.config_path
+
+
+class _Global:
+    options = None
+
+    @property
+    def logger(self):
+        return Global.options.logger if Global.options else Options.logger
+
+
+Global = _Global()
+logger = Global.logger

src/image_creator/creator.py

@@ -0,0 +1,40 @@
+import atexit
+
+from image_creator.constants import Global, Options, banner, logger
+from image_creator.logger import Status
+from image_creator.steps.machine import StepMachine
+from image_creator.utils.misc import rmtree
+
+
+class ImageCreator:
+    def __init__(self, **kwargs):
+        Global.options = Options(**kwargs)
+        # make sure we clean things up before exiting
+        atexit.register(self.halt)
+
+    def run(self):
+        if Global.options.check_only:
+            StepMachine.halt_after("ComputeSizes")
+
+        logger.message(banner)
+
+        self.machine = StepMachine(options=Global.options)
+        for step in self.machine:
+            logger.start_step(step.name)
+            res = step.execute(self.machine.payload)
+            logger.end_step()
+            if res != 0:
+                logger.error(f"Step “{repr(step)}” returned {res}")
+                return res
+
+    def halt(self):
+        logger.message("Cleaning-up…", end=" ", timed=True)
+        self.machine.halt()
+        if Global.options.build_dir:
+            try:
+                rmtree(Global.options.build_dir)
+            except Exception:
+                logger.add_dot(Status.NOK)
+            else:
+                logger.add_dot(Status.OK)
+        logger.message()