Multiple Projects

Multiple Projects

floco was designed to make managing multiple local projects easy, in fact it was originally designed to allow “recursive local builds” in a way that yarn and npm really struggle to provide.

The summary of this article for a project-a that depends on project-b is basically:

{
  _file   = "project-a/floco-cfg.nix";
  imports = [
    ./pdefs.nix
    ./foverrides.nix
    ../project-b/floco-cfg.nix
  ];
}

Files

The convention of organizating generated configs in pdefs.{json,nix}, manual configuration in foverrides.nix, and project/file organization in floco-cfg.nix is recommended.

_file Field

While the _file field is optional, it is strongly recommended and the modules for certain records are sensitive to filenames. By setting _file explicitly you’ll ensure that backtraces and deserialization of configs use the given name regardless of the “real” filename which can help avoid pitfalls.

The files floco-cfg.{nix,json}, pdefs.{nix,json}, and foverrides.{nix,json} specifically are treated specifically. At time of writing only the pdef record is sensitive to filenames, but more deserialized sensitive records may be added in the future.

For JSON files or other non-nix formats you must explicitly set _file in an inline module. The routine lib.modules.importJSON is a shorthand for this, but the following two includes are equivalent:

{
  _file = "foo/floco-cfg.nix";
  imports = [
    ( { _file = ./foo/pdefs.json; } //
      ( lib.importJSON ./pdefs.json ) )
      
    ( lib.modules.importJSON ./foverrides.json )
  ];
}

When including JSON and other formats you should remember the way “shorthand” configs get processed and you make sure that you don’t accidentally define config.config.foo = 1;.

imports Field

Imports are a module field containing paths to or inline definitions of modules that will be merged with the current module.

This field is special in the module system because it must be processed before config and generally cannot refer to any arguments other than lib. deferredModule is an exception to this rule, and is used to define configurable records.

imports may be defined for submodules as well, not just the top level module. This can be useful for recycling common code to be applied to pdef or package records.

Merged Build Plan

When multiple files or projects are combined with imports the module system merges definitions of attrsets and types recursively using rules defined in options’ type definitions.

The NixOS manual is the best resource for learning about merges, but I cover some fundamentals and notable types in floco that use custom merge routines in the Module System guide. This section will assume that you have already made yourself familiar with how configurations are merged “generally” and “priorities”.

Handling Conflicts

When combining multiple projects’ generated configs the most common area to run into conflicting metadata is in depInfo.<IDENT>.pin fields. Other fields to watch out for are fetchInfo.path, treeInfo, ltype, and depInfo.

Merging depInfo.<IDENT>.pin Definitions

This type has a custom merge function which breaks conflicts by preferring the “highest” semantic version number.

These pins may be used by submodules that attempt to generate treeInfo records when they are undefined, but remember that they will NOT have any effect on configurations which explicitly define treeInfo records since the generated fallbacks carry a low priority.

These conflicts are the most common because pdefs.nix files generated from package-lock.json files are going to “pin” whatever npm install resolves on a particular day. Notably, the updater fromPlock will use an existing package-lock.json if it exists, only adding missing lockfile entries, so one strategy for avoiding conflicting pins is to check your package-lock.json into version control, allowing it to be reprocessed by fromPlock when you make changes to your dependency graph.

Merging fetchInfo.path Definitions

Because fetchInfo.path needs to resolve relative paths from the file they are declared in, it uses a custom type which merges defintions with the routine mergeRelativePathOption, resolving relative paths to be absolute first, then asserts that all definitions are equal ( using mergeEqualOptions ).

You can use this merge implementation, and refer its usage in <floco>/modules/fetchers/path/implementation.nix if you need to use a similar relative path type in an extension to floco.

Merging ltype Definitions

ltype is short for “Lifecycle Type”, and is used to identify the state of project in the preparation process based on the type of source it is consumed from.

For example, when we consume a dependency as a tarball ( ltype = "file" ), we know that the project has already run any build phases, and there is no need to rerun them. This also means we do not care about the devDependencies for this pacakge, and can ignore them; in fact, you may notice that npm and yarn completely omit information related to devDependencies from their lockfiles.

In cases where multiple ltype definitions are found, floco will choose the “most prepared” definition so that other derived config values can skip generating treeInfo records with devDependencies, and package targets related to pre-distribution phases ( built, lint, and test ).

The priority is:

file

a distributed tarball.

dir

a local directory.

link

a symlink to a local directory, for floco this is effectively equivalent to dir.

git

a git checkout.

Merging treeInfo Definitions

treeInfo uses a regular lib.types.lazyAttrsOf type for its members, and you should remember this when making these declarations.

It is strongly recommended that you only create a single treeInfo definition for each project across all configs. Using lib.mkForce in foverrides.nix can be a good way deal to with any problematic treeInfo defintions in generated files or projects that you import.

Note that the fallback trees used if no treeInfo record is defined are produced from depInfo.<IDENT>.pin definitions. See the section above that focuses on how these are merged.

A dedicated section below covers strategies for generating treeInfo definitions with multiple projects.

treeInfo Generation Strategies

There is a guide dedicated to treeInfo that you should read as a primer for this section of the guide.

For managing treeInfo definitions across multiple projects we recommend that you only define one treeInfo record for each project, and use imports to combine all of these into a single build plan.

While you can always define treeInfo records manually, sufficiently large projects that change frequently need automation. If your project already works with npm without workspaces, the fromPlock updater is all you need. Simply use the --tree --pins flags, and then run fromRegistry for any of your transitive dependencies that need to have dependency cycles broken.

Many of these strategies refer to a “shadow tree”, which is temporary copy of a group of projects’ package.json files placed in TMPDIR with the same directory structure as the original tree. This allows fromPlock to be run with modifications to files that won’t accidentally get committed to version control.

Requirements for Local Projects ( No Workspaces )

• fromPlock wraps npm install --package-lock-only, so any dependency URIs used if your package.json files must be compatible with npm.
  • If they aren’t, you can use the “shadow tree” strategy described later.
• References to local projects must use "../foo" relative paths.
  • You can use fromPlock -pt -- --install-links to use copies instead of symlinks in the generated treeInfo record.
• Transitive dependencies that use relative paths must be declared in overrides fields such as "overrides": { "@floco/foo": "../foo" } so that npm can resolve them.

Strategy: Shallow Symlinks

This is the strategy I prefer, but it has limitations for peerDependencies that require post-processing to deal with.

It does a great job leveraging Nix’s caches for external dependencies, and handles rebuilds for other local projects quickly. With a bit of pre-processing you can sometimes even avoid regenerating metadata for local projects.

Overview

• Use fromPlock -- -pt -- --install-strategy=shallow.
• Extract subtrees to define treeInfo of direct dependencies.
• Redefine root treeInfo as symlinks to direct deps.
  • Adding link = true; and removing any subpaths for each dep.
  • For direct dependencies with peerDependencies you can’t symlink, so leave them alone.
• For further optimization you can repeat this process in subtrees that you were forced to copy.
  • Copy the root package, but symlink whichever of its direct deps that you can.
  • Only worthwhile for excessively large subtrees such as @babel/*, webpack, etc.
• If you need a copied form of a local project you need to add the flag --install-links, or post-process treeInfo to add dependencies and remove the link = true; declaration from the effected projects.
  • If you have a mix of “I want some copies and some links for local projects” you could use shadow trees or a Nix expression to hack together the desired treeInfo record. This sounds like a pain in the ass though.

Optimizations

As an optimization you can “hide” local deps during the generation of treeInfo to just create trees for external projects, then with a bit of fixup you can just refer to locals using symlinked treeInfo entries ( only works withouth peerDependencies ). This allows you avoid regenrating pdefs.nix in the consuming project if the dependency graph of your other local projects change.

This process isn’t painful to automate, but may be prone to errors if manually filled entries aren’t created properly. Nonetheless, it may be worthwhile for large dependency graphs.

Its main advantage is that you could avoid regenerating pdefs.nix files, which is a significant UX improvement for large workspaces.

• Delete local dependencies from package.json in shadow tree before running fromPlock to avoid redundant definitions of transitive deps.
• Manually add symlinks to treeInfo record for these, and manually fill depInfo record to account for its removal.
  • Splicing with an ad-hoc translation of your package.json could avoid having to implement a routine to fill these fields.
  • You should be able to create the treeInfo entry for your local dependency just by knowing its key ( ident + version ).
• Don’t forget to add the other projects’ floco-cfg.nix file to your includes field in your own floco-cfg.nix.