Compiling packages in a monorepo

[joneshf]

Concern

Compiling everything in a monorepo is too slow for large monorepos. Compiling individual packages requires recompilation work.

Context

Let's say I've got a monorepo setup similar to what's in the README:

$ tree -I 'node_modules|output|package.json|yarn.lock'
.
├── app1
│   ├── spago.dhall
│   └── src
│       └── Main.purs
├── lib1
│   ├── spago.dhall
│   └── src
│       └── Lib1.purs
├── lib2
│   ├── spago.dhall
│   └── src
│       └── Lib2.purs
└── packages.dhall

6 directories, 7 files

As the project grows, compiling everything from the top becomes slower and slower. If I wanted to iterate quickly within the lib1 package, I can move to that directory and run the build:

$ cd lib1
$ spago build --watch

This puts the output directory in lib1 and compiles everything there.

If I then wanted to iterate in lib2, I could move to that directory and run the build:

$ cd ../lib2
$ spago build --watch

Unfortunately, because the output directory is in lib1, I've got to recompile everything into lib2/output.

Workaround

This can be mitigated by running

$ spago build --watch -- --output ../output

in both lib1 and lib2. They compile to a top-level output directory, and compilation results can be shared between them. If I then move to app1 or app1/test and do similar, they also get the benefits of the pre-compiled results.

Questions

• Is this the best way to address the concern, currently? If not, what's a better way?
• Could spago make this easier somehow?
  One thing that's been in the back of my head is allowing build to know which package to build:

  $ spago build lib1

  spago could do the bookkeeping of the output directory.
  I'm open to other ideas as well.

[f-f]

Thanks for the detailed report! This makes sense. The first thing that pops into mind is that Spago could detect where a shared packages.dhall is (as described in #349 (comment)), because you'd have only one in a monorepo (we have to implement this detection anyways to fix #349), and point --output to that folder automatically, so you'd exploit a single output folder for the whole project. Makes sense?

[joneshf]

Yeah. That seems like it could work, as well.

[f-f]

Update: since #402 we do some kind of detection of where the packages.dhall is, so fixing this could be done with already existing facilities

[danieljharvey]

Hey! I'm up for giving this a pop. What what I can tell, we can now use localImportPath to turn a Dhall.Import into a nice path to pass to purs - but I'm a little bit lost as how we might create a Dhall.Import for our packages.dhall in the first place? Is it a case of reading the packages field in spago.dhall and letting Dhall do the resolution to find it? Any ideas?

[f-f]

@danieljharvey that would be great!

Is it a case of reading the packages field in spago.dhall and letting Dhall do the resolution to find it?

Yes, we now have readImports :: Text -> IO [Dhall.Import] here, to which we can pass a path and get a list of all the local imports that happen along the way when resolving the expression.
Since Dhall returns us a graph there, the part we have to figure out is how to find out which is the "root" packages.dhall (because we might have more than one importing each other, etc)

[danieljharvey]

Ace, thanks. Will there ever be more than one path to the actual Purescript package sets? Is that a reasonable measure of whether we’re at the root? Or can we end up with multiple references to them?

[f-f]

@danieljharvey we can have multiple packages.dhall files. Let's imagine a scenario like this (which might make for a great test of this feature):

.
├── app1
│   ├── spago.dhall
│   ├── packages.dhall
│   └── src
│       └── Main.purs
...
└── packages.dhall

Where:

• the top-level packages.dhall is the one containing the actual package set (we don't care if it's a remote import or if the set is materialized in there)
• app1/packages.dhall contains just ../packages.dhall

Now if we do readImports "spago.dhall" we'd get references to both packages.dhall, and we need to figure out which one is the main one. I guess an OK heuristic could be to just pick the one higher up in the filesystem hierarchy?

[JordanMartinez]

This would be a very nice improvement for my learning repository: purescript-jordans-reference. It currently has ~ 15 projects that all have to recompile most of the same code.

[f-f]

Fixed in #422