Consider splitting up the compiler into multiple packages

[natefaubion]

This was brought up in Slack (primarily by @f-f), and I think it's something we should discuss.

• The whole compiler is a very large dependency if we want to encourage external tooling that might only depend on a relatively small part (eg. parsing, corefn).
• We would like to keep the maintenance burden as equivalent to the status quo as possible (changes frequently require updating all parts of the compiler, publishing to hackage).
• What packages would we split out? What would that structure look like internally? What internal tooling would need to change or be updated to support the workflow?

[hdgarrood]

I'd suggest only splitting things up once we've identified specific cases where it would be useful to depend on only those things, so perhaps we could just start with the parser.

This sort of workflow is supported by Stack and used by projects like yesod, see for example https://github.com/yesodweb/yesod so I'd suggest using whatever Stack expects to see (although I haven't yet looked into what that means exactly).

[jmackie]

I think the parser and corefn stuff would be super useful to have in their own packages 👍

[hdgarrood]

The thing about corefn is that right now I think we only define corefn -> json conversion and not the other way; for corefn to be useful on its own we'd need the conversion in the other direction right?

[jmackie]

The other direction is here https://github.com/purescript/purescript/blob/master/src/Language/PureScript/CoreFn/FromJSON.hs#L109 ?

[hdgarrood]

Lol, ignore me then

[f-f]

@natefaubion thanks for opening the discussion on this!

In spago I sometimes wish we had access to the compiler (mostly for the parser), but we cannot depend on it in the current state as build times would get long.

So it could be nice if e.g. the parser was a separate package (e.g. I already tried to depend on purescript-cst at some point, see natefaubion/purescript-cst#10), but on the other hand I'm not sure this is a good idea on our side: after all pulp has managed just fine without a compiler dependency for long time

So far we avoided being blocked by "depending on the compiler", except for purescript/spago#165, where we might have to choose between:

1. implementing the functionality in the compiler (problem: adding maybe orthogonal functionality under the "facilitating build tools" label)
2. implementing a subset of the PS parser in spago (problems: it might get out of sync with upstream, and other build tools will have to reimplement it if they want the feature)
3. depending on the compiler (problem: build times)

Since we are open to splitting off packages, I'll make a small proof of concept of implementing the above feature by depending on the compiler as a library, so I'll get a better idea of which package I'd like (most likely just the parser)

[jmackie]

I had a quick look at this (branch) and here's what I'm proposing (if only to stimulate discussion).

We add a lib directory in the root of the project to hold purescript-* packages, and the rest of the repo remains largely the same. The stack.yaml would then look like...

resolver: lts-13.12
pvp-bounds: upper
packages:
- '.'
- 'lib/purescript-parser'
# ...etc

In theory all the packages under lib should capture some useful piece of the compiler and be published/publishable to hackage (e.g. in the first instance maybe the parser and corefn).

The issue is that there are certain "core" modules that would probably need to be shared across these packages (e.g. Language.PureScript.Errors, Language.PureScript.Names), but how we do that would depend on how granular we want this split to be. I'm assuming we can't list these modules as other-modules for several packages when they depend on each other (and that's probably a hack anyway), so we might need a purescript-core package, which also feels a bit lame.

[joneshf]

With the merge of #3793, this is in happening! Do we want to close this issue, or leave it open to focus on what the end state is and track progress towards that?

[hdgarrood]

I'd like to close it and discuss further steps in their own issues, personally.