Permalink
Switch branches/tags
0.0.x 0.2.x 0.3.x add-travis-windows andreypopp/debug-macos-failure andreypopp/fix-647 andreypopp/gen-bat-shell andreypopp/include-workflow andreypopp/lockfile-syntax andreypopp/test-win-padding andreypopp/try-ocurl appveyor/debug bootstrap bryphe/appveyor/cache-npm bryphe/appveyor/improve-caching bryphe/bugfix/eaccess-on-install bryphe/build/remove-bootstrapping bryphe/ci/azure-windows-caching bryphe/ci/platform-package-report bryphe/ci/test-slowtest bryphe/dependency/appveyor-esy-0.3.4 bryphe/dependency/ocaml-4.6.8 bryphe/dependency/upgrade-esy-bash-0.2.1 bryphe/docs/xplat-draft bryphe/esy-rewrite-prefix-1 bryphe/inline-fastreplacestring bryphe/investigation/debug-logging-windows-hacks bryphe/investigation/debug-logging bryphe/lockfile-fixes+slowtests bryphe/proto/esyi-unblocked-windows bryphe/proto/use-forked-submodule bryphe/prototype/unc-paths bryphe/test/slow-test-compilation bryphe/travis/windows-build bryphe/windows/bugfix-arch bryphe/windows/build-esyi bryphe/windows/build-validate-ocamlbuild-project bryphe/windows/e2e-tests bryphe/windows/esy-build-package-permission-denied bryphe/windows/esyi-bash-windows-commands bryphe/windows/forked-override-repo bryphe/windows/hack-resolve-command bryphe/windows/hook-up-slowtests bryphe/windows/lockfile-logging bryphe/windows/release-artifact-part-1 bryphe/windows/skip-flaky-test bryphe/windows/slow-tests bryphe/windows/solver bryphe/664/windows-out-of-space bryphe/667/repro docs/cleanup feat/pnp fix/linked-changes fix/memoize fix/no-package-name fix/opam-test-doc fix/slow-test intern-fastreplacestring master some-random-pr test/fast-slow windows-recovery
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
265 lines (183 sloc) 8.04 KB
id title
how-it-works
How esy works

This document describe esy internals.

Overview

Almost every esy command operates in context of a project sandbox which is defined by a sandbox manifest (usually package.json but esy.json is also supported).

Pipeline

The typical pipeline from having a clean checkout of an esy project till the point where all artifacts are built consists of the following steps:

  • Solve Dependencies

    Produces esy.lock solutuon lock out of package.json. This step is optional as esy.lock can be already present in a fresh checkout.

  • Fetch Dependencies

    Ensures all packages mentioned in esy.lock is in the Global Installation Cache.

  • Crawl Package Graph

    Crawls the sandbox's lockfile and linked packages and read them into BuildManifest.t.

  • Produce Task Graph

    Folds over the BuildManifest.t and produces the Plan.Task.t structures.

  • Build Task Graph

    For each Plan.Task.t exeute build commands in the corresponding environment using esy-build-package command.

Solve Dependencies

This step produces a solution out of dependency declarations found in project's root manifest and all transitively dependent packages' manifests.

First, a package universe (a transitive closure of all dependencies' versions) is constructed by consulting package registries (npm and opam currently) and other sources (remote URLs, local paths and various git repositories hostings).

Constructed package universe is then encoded as CUDF and then is being fed to a CUDF solver (provided by the esy-solve-cudf npm package which uses mccs solver underneath).

The result of the solver is then decoded and serialized on disk as esy.lock directory. It is advised to commit such file to a project's VCS as it captures the current state of the project's dependencies thus allowing to reproduce the exact same environment on other hosts at other points in time.

Modules of interest:

  • esyi/Universe
  • esyi/Resolver
  • esyi/Solver
  • esyi/Solution

Fetch Dependencies

This step consumes a solution produced by the previous Solve Dependencies step and ensures that all packages mentioned in the solution are fetched and cached in the Global Installation Cache.

How it works:

  • Traverse the solution
  • For each record of the solution:
    • Fetch source (either a tarball or a git repo or ...)
    • Apply all needed patches
    • Pack as a *.tgz and store in a cache

Modules of interest:

  • esyi/Fetch
  • esyi/Solution

Crawl Package Graph

This step crawls the sandbox's lockfile and linked packages and read them into BuildManifest.t.

This graph has package metadata at nodes and edges are instances of dependency relations between packages. The dependency relation is defined by the following fields in a package's manifest:

  • "dependencies"
  • "peerDependencies" - same as "dependencies" from the point of view of esy, was used by the legacy implementation of esy install command to defer installing dependencies to the root package.
  • "optDependencies" - this models optional dependencies (if they are installed they are used, otherwise - ignored), an analogue to opam's depopts which are being discouraged now.

Modules of interest:

  • esy/Plan
  • esy/BuildManifest

Produce Task Graph

This step consumes a BuildManifest.t structures and produces a Plan.Task.t structures.

The resulted graph is topologically isomorphic to the original Solution.Package.t graph but contains much more information about the build process for each of the packages in a sandbox:

  • A list of ready to execute commands
  • An environment which is needed to execute build commands

Modules of interest:

  • esy/Plan

Build Task Graph

After Task.t is constructed it's time to build it.

Each Task.t is serialized into JSON format called Build Plan which is then used to invoke esy-build-package executable.

Modules of interest:

  • esy/Build
  • esy/PackageBuilder
  • esy-build-package/Builder

Caches

There are multiple levels of caches used by esy.

There's no garbage collection mechanism provided at the moment. This can be a problem for hosts with little free disk space available.

Global Installation Cache

This cache stores sources of concrete package versions.

Location & Structure

The default location for the cache is ~/.esy/esyi/tarballs and can be indirectly controlled by the --cache-path option of esyi executable.

% tree ~/.esy/source/i
├── esy-installer__0.0.0.tgz
└── substs__0.0.1.tgz
...

Cache Key

The cache key used for the cache consists of:

  • Package name
  • Package version
  • Package source (needed if package was fetched not from a registry but a git repository or other source)
  • A hash of all contents of patches and additional files (if those are defined for the package, currently used by the opam overrides infra).

Global Build Store

This cache stores built artifacts of esy packages and related metadata.

Location & Structure

The default location for the cache is ~/.esy/3<prefix> and can be indirectly controlled by the --store-path option of esy executable.

The <prefix> part of the path is consist of a number of underscore characters _ which pads the store path so that the length of the path to the ocamlrun executable in the store is exactly 128 characters.

The number 128 comes from the fact that on some systems a path mentioned in a shebang line (first line of executable which starts with #!) is limited to 128 characters. Thus the current limit ensure that OCaml bytecode executables can be run from the store.

The padding is needed to allow relocating built artifacts between stores.

The cache looks like:

% tree ~/.esy/3_*
├── b
│   ├── ocaml-4.6.1-4f6b0960
│   ├── ocaml-4.6.1-4f6b0960.info
│   ├── ocaml-4.6.1-4f6b0960.log
│   ...
├── i
│   ├── ocaml-4.6.1-4f6b0960
│   ...
└── s

Where

  • b/<key> is a directory which is used as a build root for a corresponding package.
  • b/<key>.log is log file for the build process of a package which corresponds to the <key>.
  • b/<key>.info contains information about the corresponding build process such timer ellapsed and so on.
  • s/<key> is a stage directory for built artifact installation (packages install their own artifacts there and then esy moves s/<key> to i/<key> so that the changes to the store are executed atomically.
  • i/<key> is an installation directory, this is the directory which hosts built artifacts of the package which corresponds to the <key>.

Cache Key

The cache key used for the cache consists of:

  • Package name
  • Package version
  • Hash of all build/install commands and other esy specific metadata from a package manifest
  • Hash of all dependencies' cache keys

Local Build Store

Local Build Store follows exactly the same layout and cache key as the Global Build Store but it is local to a sandbox and located at <sandboxPath>/_esy/deafult/store path.

It is used to store artifacts of packages which don't have a stable build identity (unreleased software which changes often and doesn't warrant sroting its artifacts in a Global Build Store).

Local Sandbox Cache

Local Sandbox Cache stores a computed package and build task graph. It is located at <sandboxPath>/_esy/default/cache/sandbox-<hash> where <hash> part is a hash of:

  • Store Path
  • Sandbox Path
  • Local Store Path
  • Version of esy

The cache is stored in a format readable by OCaml Marshal module.

Reading the cache file with a version of esy which has different SandboxInfo.t layout than the one with which the cache was produced with usually results in a Segmentation Fault.