Skip to content

Latest commit

 

History

History
552 lines (531 loc) · 28.9 KB

index.mdwn

File metadata and controls

552 lines (531 loc) · 28.9 KB
Raw

[[!meta title="Design"]]

Bootstrapping

Workflow

Current

Diagram

[[!inline pagenames="doc/development/design/gfx/current-workflow" raw=yes]]

Outline

  • Have two tasks:
    • Install
    • Test
  • What do these tasks do:
    • Determine the platform
      • Platforms:
        • [[Debian|doc/development/system/debian]] + [[apt|doc/development/package-manager/apt]],
        • [[macOS|doc/development/os/macos]] + [[Homebrew|doc/development/system/homebrew]],
        • [[Windows|doc/development/os/windows]] + [[MSYS2|doc/development/system/msys2]].
    • Set up environment for everything (task: Install, Test).
      • Not great... this is specific to project-renard. It downloads the test data and sets an environment variable.
    • Each platform has a _install method (task: Install).
      • Debian + apt: Install pkg:deb/debian/xvfb, pkg:deb/debian/xauth.
      • macOS + Homebrew:
        • Update Homebrew if needed (in non-CI environments). This is necessary because Homebrew can be slow to update and this slows down the CI. This might not be needed with newer versions of Homebrew.
        • Remove old Python: python@2 to clean up a pre-installed package. This is (was?) specific to GitHub CI and was causing a conflict with Python 3.
        • [SKIPPED] Installing xquartz. This might have been for an older version of gtk+3. Note that this requires adding a tap to install a cask. Good thing it isn't needed now but being able to use a Homebrew tap is still important.
        • Install pkg-config. Often needed for finding the libraries.
        • Install openssl.
      • Windows + MSYS2:
        • Prepare MSYS2:
          • Disable CheckSpace option for pacman. Checking the disk space takes time in the CI. This speed up is also used by the GitHub action https://github.com/msys2/setup-msys2.
          • Update mirror list. This was needed at some point to fix the update because the mirror was down. Need to look more into this as this was a hot fix at the time.
          • Run pacman update, then
            • Update also requires killing processes that use msys-2.0.dll. This is the same as what installers of tools like Chocolatey and the aforementioned GitHub Action do.
            • Install pacman-mirrors.
            • Install git. Needed for below when fetching repos.
            • There was a (now disabled) work around to update GCC9.
        • Install build tools using pacman.
        • Install openssl package using pacman.
        • Copy gcc to cc and mingw32-make to gmake. Not sure if this is needed anymore. This makes Perl XS builds happy.
        • Create C:\tmp for Data::UUID. Not fixed yet!
        • Install Perl. Fixes pl2bat because pl2bat.bat isn't there. Installs cpanm, cpm, and some basic Perl modules. Includes my own patches to cpm to help it work in parallel (not ready for upstream). Not great as this is Perl specific and really should be part of non-platform code. Who did this?! Oh, right.
    • Each platform has a _pre_run method (task: Install, Test)
      • Debian + apt:
        • Start the Xvfb sever. Note that the reason why Xvfb is also needed for the Install task is that running dzil listdeps or dzil build needs a display because with project-renard/curie, these commands load the code. It runs Gtk3's init on loading the module! This is a bad design, but worth working around just to prove that it can be worked around. Run DISPLAY= perl -MDevel::Hide=Gtk3 -S dzil listdeps to see the issues. This is also why errors while building occur when these modules have not yet been installed in the CI environment. A very unsensible arrangement.
      • macOS + Homebrew: no-op.
      • Windows + MSYS2: no-op.
    • Each platform has an environment (task: Install, Test)
      • Debian + apt: sets DISPLAY to the Xvfb started above.
      • macOS + Homebrew:
        • This environment depends on the Homebrew prefix.
        • Adds to PKG_CONFIG_PATH: openssl, libffi
        • Adds to PATH: openssl, gettext.
        • Sets ARCHFLAGS: This has something to do with using a macOS system Perl. Might not be needed with Homebrew. Bad idea to use the macOS system Perl anyway.
      • Windows + MSYS2:
        • Set MSYSTEM: default is MINGW64.
        • Set PATH: to the default paths for that MSYSTEM.
        • Set MSYS2_FC_CACHE_SKIP: skip font cache for fontconfig package.
        • Add hack that modifies Perl linking for EUMM: yikes! This isn't even in the same project! Boo this code!!!
        • Set ALIEN_BUILD_PKG_CONFIG to prefer PkgConfig::CommandLine: This should probably be upstream.
    • For each repo to install via depth-first walk (task: Install):
      • Get the repos using git. Note that git is already installed in most environments (except MSYS2 in the CI so that is why the _install for MSYS2 step installs git.
      • Install all the "native" dependencies of the repos first, then install the repos.
      • Each platform has an install_packages method to install the "native" packages (task: Install)
        • Debian + apt:
          • Check if all needed packages are already installed.
          • Otherwise apt-get update.
          • Then apt-get install.
          • Includes a hack to install meson via pip3. This is because the version of Meson in the specific Debian container used is too old. I would like to support this somehow for flexibility purposes. But not here. Not like this... not like this. This will actually break with newer systems due to how pip3 is now set up using an externally-managed-environment under PEP 668.
        • macOS + Homebrew:
          • Disable auto update. See above in _install for why this was slow on older Homebrew's.
          • Check if fontconfig is a dependency. If it is, use a hack to skip the font cache generation post-install step. Terrible. But the hack is still needed.
          • brew install any packages that are not already installed. Ignore errors by using || true. Alas!
        • Windows + MSYS2:
          • Uses Chocolatey and Pacman! Wait, I thought this was just Pacman for MSYS2! The horror! I believe this is for one specific thing which was to get testing of Anki working on Windows (because there is no way Anki is going to be installed via an MSYS2 package as the focus is on developer tools).
      • Install the repo itself (task: Install)
        • Right now this is only for Perl distributions.
          • via Orbital::Payload::Env::Perl->apply_roles_to_repo.
          • More coupling! Cut the knot!
        • There is some caching here.
        • $repo->install.
    • Test the main repo (task: Test).
      • $repo->run_test.

Future

  • The tooling runs on a System
    • Host System is where it all starts!
    • Target System is what you want the output to run on.
      • Targets can be nested.
      • e.g., you need to run inside of a container inside of VM on another machine. For example, you have a build box with Windows VM and you need to use specific Windows Docker containers.
  • For host, target
    • Need to identify the kernel, OS, architecture.
      • Note: some OSes are multi-arch or have a compatibility mode (x86_64 can run x86 code, some Windows could run 16-bit using NTVDM)
        • e.g., Debian supports multi-arch apt package settings.
      • Process:
        • Note: uname is from POSIX https://en.wikipedia.org/wiki/Uname
        • Kernel discovery process:
          • On Linux
            • uname -s: name
            • uname -r: release
            • uname -v : version
        • OS discovery process:
          • On Linux
            • uname -o
          • On DOS, Windows
            • ver
        • Kernel Architecture
          • On Linux
            • uname -m
          • Note: kernel is compiled for a particular architecture
          • This might not be what the CPU could potentially support.
          • e.g., a 32-bit kernel running on a 64-bit CPU machine
            • this may limit what you can run
        • CPU architecture
          • CPU info
    • Need to identify runtime environment (default: host and target are identical and local).
      • Identify which tools are available
        • Debian: apt, most likely GNU coreutils
        • MSYS2: pacman
        • Alpine: apk, Busybox (as opposed to GNU coreutils)
      • Process:
        • Depends on OS.
        • In some cases the runtime environment tools you want are not installed such as macOS and Windows that have multiple package managers.
    • Scenario 1:
  • For configuration
    • Package dependencies
      • Have a way to say what you depend on and how to install it.
      • Examples:
      • This is necessary for different kinds of scenarios:
        • Scenario 1:
          • Goal: make sure that code works with the Perl interpreter and Perl modules packaged in Debian.
          • Given a project that depends on Perl modules (e.g., pkg:cpan/module/Foo)
          • Step 1: Map the Perl modules to Debian packages (e.g., pkg:debian/deb/libfoo-perl)
          • Step 2: Install the ones that map via Debian packages
          • Step 3: Install the rest using CPAN client.
          • Note: This implies the Perl being used is the /usr/bin/perl via Debian pkg:debian/deb/perl.
          • Example: used in the BioPerl CI.
          • Process:
            • Need to have a way to enumerate Perl module dependencies.
              • Enumerating dependencies can be done via different strategies (that need priority):
                • cpanfile
                • META.json
                • Makefile.PL
              • These are part of what CPAN clients can provide.
        • Scenario 2:
          • Goal: test different build options for Alien::Build.
          • Given a Perl Alien project that uses Alien::Build that is used to provide a library bar.so and headers bar.h.
          • Alternative 1.1: On Debian, Alien install type "system" needs pkg:debian/deb/libbar-dev to work.
          • Alternative 1.2: On macOS, Alien install type "system" needs pkg:brew/formula/bar to work.
          • Alternative 1.3: Alien install type "share" only needs compiler. This is tested on multiple OSes.
          • Example: many different Alien modules
          • Process:
            • Need to have a way to define dependencies for different options / variants.
        • Scenario 3:
          • Goal: Select which CPAN mirror to download from.
          • Given a Perl project that has local patches to dists that it needs.
          • These patched dists are stored in a local private mirror.
          • Set the mirror used for particular packages.
          • Pin to a particular version.
          • Aside: List of all public CPAN mirrors http://mirrors.cpan.org/ (but this is no longer needed as CPAN uses a CDN now)
          • Aside: Important for security: also needed to set HTTPS (see cpanm docs for --mirror).
        • Scenario 4:
          • Goal: Faster builds using pre-configured images.
          • Given a project with dependencies A, B, C which are installable via the system package manager.
          • Step 1: Create a Docker image that install A, B, C if the image does not exist.
            • May need to specify level of cacheing
          • Step 2: Now run the rest of the installation inside of the Docker image.
          • Note: This will run faster on subsequent installations.
          • Note: May need to indicate what to do about checking for newer versions of A, B, C (update + upgrade)
    • Service dependencies
      • Scenario 1: Tests need X11
        • X11 can be run under various servers
          • If DISPLAY is set, have option
            • Use current DISPLAY
            • Use Xvfb
            • Use Xephyr (may need a WM to run inside)
          • If DISPLAY is not set (headless environment)
            • Set DISPLAY
            • Use Xvfb (requires pkg:debian/deb/xvfb)
      • Scenario 2: Tests need PostgreSQL
        • Alternative 1: Use existing PostgreSQL on system.
        • Alternative 2: Use Docker https://hub.docker.com/_/postgres/.
        • Alternative 3: If already using Docker for running the tests separately, use Docker Compose.
        • Requires: connection string (host, port, auth)
  • Testing of Orb database
    • Need to be able to detect updates (e.g., new version of Debian, FreeBSD, etc. released)
    • Self-tests (e.g., a configuration that uses that particular Orb).
    • Need to have an archive of test reports for particular configurations.
      • Because doing integration tests on these could be heavy (time consuming, lots of compute resources).
      • Used to track stability across versions.

Outline

  • Fetch repos:
    • current repo is used as is
    • Fetching dependencies have have several strategies
      • if have git
        • Have metadata to repo elsewhere on FS
          • if repo is git and have git, can use git worktree
        • else git clone
      • else
        • fetch over HTTP using host specific tarball?

Workflow graph

Organisation of project

  • Payload:
    • Env (for Environment):
      • Language
      • DB (e.g., PostgreSQL, MySQL)
      • XWindows
    • Sys (for System):
      • Operating system
      • Package management system
    • Serv (for Service):
      • Usually a web service
      • e.g., GitHub, Sourceforge, Bitbucket, AppVeyor, Travis CI, Coveralls, Codecov
    • VCS: version control system
      • git, bzr, svn, cvs, etc.
  • database
    • pkg/: packages. These can be from source (for FOSS code) or binary-only (for proprietary code).
      • e.g., pkg/gui/gtk3
        • inside of this create definitions such as [[!format txt """ [[!inline pagenames="doc/development/design/pkg-def.txt.ex" raw=yes ]] """]]
      • e.g., pkg/perl/Gtk3 for https://p3rl.org/Gtk3
        • depends on pkg/gui/gtk3 with feature :glib-introspection
      • Need a mapper from external packages to database.
        • e.g., CPANPackage($Name) -> Pkg('main:pkg/perl/$Name')
        • must be overrideable: in case a fork needs to be used or in the example of Perl, an https://p3rl.org/Alt module.
    • sys/: os package manager + distribution
      • e.g., sys/os/{windows,linux,macos}
      • e.g., sys/dist/{debian,ubuntu}, sys/dist/fedora
      • e.g., sys/dist/{msys2,chocolatey}
      • e.g., sys/dist/{homebrew,fink,macports}
      • e.g., sys/arch/{intel,arm}
    • env/: language {perl,python,go}
    • role/: BLAS-interface, LAPACK-interface, XWindows-server, SQL-server, web-browser
  • databases should be layered:
    • p5-Orbital-Transfer-Database distributes core database with OS, Language, Architecture files
      • actually stored in orbital-transfer-database/core
    • orbital-transfer-database/main contains database with packages such as pkg/gui/gtk3.

CI of project modules

  • Pull from launch-site.
  • For every repo in { GITHUB_SHA, cpanfile-git } and launch-site/vendor/, checkout the version of that repo.

Roadmap