Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Separate Compilation #218

Closed
paf31 opened this Issue Feb 21, 2014 · 17 comments

Comments

Projects
None yet
3 participants
Owner

paf31 commented Feb 21, 2014

When using :m to load a module, it would be nice if we could generate and save Javascript for that module, so that we don't have to recompile it for every expression. We can use --externs to generate type definitions, or just keep the data in the Environment around.

@paf31 paf31 added psci labels Feb 21, 2014

@paf31 paf31 added this to the 0.5.0 milestone Feb 21, 2014

Member

joneshf commented Feb 22, 2014

What would be the benefits of using externs or keeping the data available? Is it so we can type check things before attempting to compile it, or... I just realized why. There's no types annotations in js.

Also, this seems like a valid use case for XDG_CACHE_HOME or whatever it's called. In the simple case, we can just over write the files each time the module is loaded. When we get an actual module system, we can just check the commit sha or whatever we decide on in order to determine if we need to recache the files.

Member

joneshf commented Feb 22, 2014

Actually, if that sounds reasonable, then it would be better to go with the externs file. Then we wouldn't have to recompile a module that hasn't changed between psci sessions. And actually, we'd probably want to munge the generated filenames with the sha, or rehash the js files or something?

Member

joneshf commented Mar 9, 2014

So here's the current thing I'm doing.

  1. :m
  2. Find
  3. See if the file exists in the cache.
    • If the file exists in the cache, compare the time stamps
      • If the cache timestamp is newer than the file timestamp don't do anything.
      • Otherwise compile the js and e.ps files.
    • Otherwise compile the js and e.ps files.
  4. Update the imported modules.

Step 4 is where i'm stuck with the #253 I can update the files when I compile them because it has the AST still in memory. However, with the externs not being parsable, I can't load the AST into memory. That's not that big a deal because it will work once #253 is fixed.

My question is, how do we go about using this generated js? Each js file has the code wrapped in a module check and everything is local to that scope, so we cannot access things not in the same file without resorting to horrible things like generating an additional js file which requires all of the modules in scope and dumps them into the same namespace. Or something similar.

This relates to what I asked today about the bare flag for compilation. I wasn't thinking about this situation, but it seems pretty similar. This is one of the few times when it makes sense to leave the compiled js naked.

Or have I completely missed the mark and there's some clean way to do this that I'm not thinking of?

Owner

garyb commented Mar 9, 2014

I think the generated JS should be usable as it is. It's supposed to work in a way that merge-or-creates at each step, so if window.PS is already there, we just add the modules to it, and if some window.PS.Module is already there we just assign members to it.

So for example, if we exported some code that only had Prelude.($) defined, it should still merge nicely with an existing prelude module:

(function (_ps) {
    "use strict";
    _ps.Prelude = (function (module) {
        var $dollar = function (f) {
            return function (x) {
                return f(x);
            };
        };
        module["$"] = $dollar;
        return module;
    })(_ps.Prelude || {});
})((typeof module !== "undefined" && module.exports) ? module.exports : (typeof window !== "undefined") ? window.PS = window.PS || {} : (function () {
    throw "PureScript doesn't know how to export modules in the current environment";
})());

Basically, wherever we have a x || {}, we're trying to get an existing object to extend, or fall back to a new one.

Member

joneshf commented Mar 9, 2014

Oh, so I was missing something simple. I'll check it out when I get home.

Owner

paf31 commented Mar 9, 2014

@joneshf Separate compilation should now be possible, since #253 is done.

Member

joneshf commented Mar 9, 2014

Cool, thanks. I'll try and finish this up today.

Member

joneshf commented Mar 9, 2014

Hmm, when we go to generate the externs file, it puts every defined or used type into this file. So if you have Foo that imports Prelude, it will dump all of prelude's types into Foo's extern file. When you attempt to load this in psci (or even if you try to use two extern files that imported Prelude) it tries to redefine all of Prelude's operators, causing it to fail on load.

What can we do about this?

Owner

paf31 commented Mar 10, 2014

There's always --no-prelude, but that only goes so far. Ideally we'd have a way of reading a file without doing any codegen from its declarations. I'll work on something like that soon.

paf31 added a commit that referenced this issue Mar 14, 2014

Owner

paf31 commented Mar 14, 2014

@garyb @joneshf I've checked in some code for separate compilation to the make branch. It's working, except that there is an issue with the externs files when compiling the Prelude.

Compiling Prelude for the first time correctly goes through each module and generates the externs files and Javascript, but the second time, it tries to read Prelude.externs and fails with

Reading externs/Text/Parsing/Read.externs
Reading externs/Prelude.externs
Error in foreign import declaration (>=):
Error checking kind of forall a. (Prelude.Ord a) => a -> a -> Prim.Boolean:
Unknown type constructor 'Prelude.Ord'

which is very odd since psc --no-prelude Prelude.externs works fine, so I know that externs file is correct.

Note: this also works when compiling the purescript-in-purescript repo :)

Owner

garyb commented Mar 14, 2014

I'm not exactly sure what I'm doing with --make, have you got a makefile or something somewhere so I can see what I should be doing?

I managed to compile the test-writer target from purescript-transformers by generating a prelude.externs first and then including it in the build path, is that right? After that, the test-writer build worked the first time, but the second time around it wouldn't work whether I had --no-prelude or not.

Also, it's generating JS files for the Prelude - should it be doing this when the defs come from an .externs file? The output isn't very useful as the majority of the module files are empty, and the rest are missing everything apart from the typeclass dictionary access methods and export assignments. Is this like the --codegen issue you mentioned to me yesterday?

Here's the modified target I was using:

test-writer:
    psc src\Control\Monad\Trans.purs \
        src\Control\Monad\Identity.purs \
        src\Control\Monad\Writer.purs \
        src\Control\Monad\Writer\Class.purs \
        src\Control\Monad\Writer\Trans.purs \
        src\Control\Monad\Error.purs \
        src\Control\Monad\Error\Trans.purs \
        src\Control\Monad\Maybe\Trans.purs \
        src\Control\Monad\Reader\Trans.purs \
        src\Control\Monad\State\Trans.purs \
        examples\Writer.purs \
        prelude.externs \
      --main \
      --module Main \
      --tco --magic-do \
      --make

Aside from the barrage of questions, it's looking pretty good!

Owner

paf31 commented Mar 14, 2014

So, a few things:

  • Prelude doesn't get included by default, so yes, you were right to build an externs file.
  • --main does nothing when combined with --make right now.
  • Neither do --module, --codegen or --no-prelude. We should probably make all of these combinations raise an error.
  • It builds every module separately and generates a .js file and a .externs file if necessary. If the timestamps are recent, it uses the externs file to typecheck, but doesn't generate a .js file.
Owner

garyb commented Mar 14, 2014

Ah yeah, I wasn't too worried about the options like --main, --module etc. they were just left in there from before.

So was I getting the empty/broken JS files for the prelude modules because they didn't already exist in /js/ then? How am I suppose to actually get those files if I have to include the prelude as an extern?

Owner

garyb commented Mar 14, 2014

Oh wait, maybe I was doing it wrong, I used -e to make my prelude extern file.

In that case, how do I build the /externs/*.externs files for the prelude?

Owner

paf31 commented Mar 14, 2014

You can still make an externs file with -e but the "recommended" way to build a library would be to use --make and to include the Prelude.purs file in the list of included files. If we can find a way to locate files by package name, we could make this easier.

Owner

garyb commented Mar 14, 2014

Ok great.

I was thinking about that too actually. If you chuck a single filename at GHC it seems to search the path relative to that file for import (and an internal path for prelude stuff I assume), so for Control.Monad.Writer it looks under \Control\Monad\Writer.hs. Or maybe it uses the cwd... I think it's the file's folder though.

I guess the complication of doing that in PSC is it means analysing each module for imports and qualified names and then trying to find corresponding files to add to the compilation list.

It'd be nice to have though, both for --make and "traditional" psc usage.

edit: actually, GHC does just look in the cwd.

Owner

paf31 commented Mar 16, 2014

This is done for the most part, but we still need

  • recompilation based on dependencies
  • move psc --make into a new wrapper psc-make, since the options don't overlap very much.

@paf31 paf31 closed this Mar 19, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment