Can prepack run on itself? #1307
Comments
|
@sebmarkbage made a |
|
That's super cool. Even if it ever worked at one point, that's encouraging and means it's achievable. I'll check it out and see if it still works. |
|
I believe it is broken on other versions of Node since it provides some shims for node internals that change between version and even then some of it is a bit shaky. Ideally we'd do this in a membrane where we simulate what would've happened if we ran it in Node's runtime but don't actually snapshot the whole of Node's internals too. It definitely works though. Prepacking Prepack itself isn't hard. It's Node's file system handles, module system etc. that gets tricky but also doable. |
|
Do you happen to know off-hand which version of node is the last one that worked? Does prepack even need to use any of the filesystem if all I want to do is give it a source string to partially evaluate? Basically, I don't want any of the CLI. It's probably not as shaky if I ignore that part of the system, as you said prepacking prepack itself is more straight-forward. |
|
If you don't want any of the node specific things you can use https://github.com/facebook/prepack/blob/master/src/prepack-standalone.js Other than that it is basically just for the module system. You can use rollup or something on that file first, and then Prepack the result. (Compile with Babel first just like the normal build those since we may not support all features that we use.) |
|
FWIW I tried to compile it tonight. Here's the webpack config to generate a bundle of prepack standalone: module.exports = {
entry: './src/prepack-standalone.js',
output: {
path: __dirname + '/dist',
filename: 'output.js'
},
module: {
rules: [{ test: /\.js$/, exclude: /node_modules/, loader: 'babel-loader' }]
}
};This generates a I copied the same flags as the |
|
Don't use the |
|
I don't have high hopes of this working well for your use case though. I assume that what you want is to run Prepack, not just on itself, but on itself combined with your interpreter while leaving abstract paths intact. There are many limitations of Prepack that you are likely to hit in this case. There's a reason that not all compilers are as magical as Prepack and you have to be aware of the current limitations so you can apply it to scopes where the limitations of the scope unlocks some new capabilities. E.g. module initialization has the benefit that most input is already known AOT and that the start of the program is known. React components has the benefit that you can assume that render functions are pure as well as that the programming model uses lazy evaluation so that if you need to bail out you have safe places to do that. If you can't make those assumptions then it is not safe. Prepack can run its whole initialization phase and export the API without running the API through some input data. Prepack can also run itself and produce the result given static input. It cannot be given arbitrary abstract input and partially compile itself in arbitrary ways. |
|
Ok, that mostly works. Looks like it pulls in the return (typeof document !== 'undefined' && document.documentElement && document.documentElement.style && document.documentElement.style.WebkitAppearance) ||inside a If I just removed all the code within If I add this code at the bottom of console.log(prepack('var x = 0; for(var i=0; i<5; i++) { x += i }; console.log(x)'))Then creating the new bundle and prepacking it generates a file that I can run in node (I need to go to the bottom of the file and replace % node ./dist/processed.js
{ code: 'x = 0;\ni = 0;\ni = 1;\nx = 1;\ni = 2;\nx = 3;\ni = 3;\nx = 6;\ni = 4;\nx = 10;\ni = 5;\nconsole.log(10);',
map: null,
reactStatistics: null,
statistics:
_23 {
objects: 0,
objectProperties: 0,
functions: 0,
functionClones: 0,
referentialized: 0,
valueIds: 0,
valuesInlined: 0,
delayedValues: 0,
acceleratedModules: 0,
delayedModules: 0 },
timingStats: undefined,
heapGraph: undefined }But I think I see what you mean. It will probably fail if I try to give it a program that requires part of prepack to be abstract itself. It looks like it was able to run prepack on the given program entirely in the prepacking phase, so none of prepack needed to be abstract. |
|
Well, I bundled up my compiler+VM with this code in the entry point: function runSource(src) {
vm.runSource(src);
console.log(vm.reg1);
}Then pasted that code as a string into the prepack bundle, and it's now prepacking that file... been almost 5 minutes but it's still going! I doubt it will finish without an error but we'll see what happens. |
|
But this does clarify that this isn't suited for what prepack currently is built for: optimizing the startup path, since core value (the source) isn't known AOT. It's fun to think about this though. I'll update with whatever happens and then close the issue. |
|
It ran for a long time and then errored with this: |
|
I see in the prepack CLI it will not show any errors thrown if there are no compiler diagnostics. That's what results in seeing "FatalError must generate at least one CompilerDiagnostic" without any info. It might be nice to add an option or something to print out the internal error. (related to #1305) Turns out it was just a stack overflow, so I bumped up the stack which should fix it. Still running. |
|
I continually tried bumping the stack size up until it's using a huge number ( |
|
Oh, I just realized this is not V8 hitting it's stack limit, this is an error thrown by prepack itself: https://github.com/facebook/prepack/blob/master/src/realm.js#L362. I hardcoded a much larger stack limit and we'll see what error comes next! |
|
I think the You can add something like https://github.com/facebook/prepack/blob/master/src/options.js#L14-L15 That doesn't do anything special. There are two other pieces of work that might be relevant here. @hermanventer has been working on supporting loops over abstract values. That is still pretty fresh but seems like it would be critical to your use case. Definitely interesting to see. I've also experimented with allowing abstract function calls to accept mutable objects/bindings. I don't think you'll need that but it might come up at some point when all possible functions aren't known. That requires a special flag atm. You'll know if you need it though since you'll get a warning that an abstract operation isn't supported. |
|
That's awesome! I don't comprehend abstract values fully yet, or how prepack should work with libraries instead of apps, which is essentially what I'm doing. I'd love to study prepack more and contribute though when I find time. In the next week or so I'll try to make small tweaks. Prepack seems to working on prepacking my use case - but it's been running for 2.5 hours now. Not sure if it's hit an infinite loop or not. Last time I let it run for a while and it eventually finished (after an hour maybe?). In that case I forgot to export the main function so prepack just compiled it all away. Now I'm exporting a function "run(src)" (which will actually compile, not run) to the global scope. I'll leave it running until something happens. |
|
We still need a lot of work before we can handle every loop in Prepack, but in principle this is doable this year. Another big gotcha is recursive calls, which is something I'm thinking about at the moment. The real killer, however, is dynamic dispatch, such as found in Environment.evaluateAbstract. Prepack will have to be pretty much completed end-to-end before we can deal with that. |
|
@hermanventer would you expect it to get into an infinite loop while trying to partially evaluate it? I'm surprised it hasn't thrown any errors yet after almost 3 hours of running. But maybe it is just stuck in a loop. Very cool to hear that this could even be possible though. I haven't wrapped my head around what kind of code this should output, but being able to generate a compiler from my interpreter would be pretty mind-blowing and open up use cases I thought were impossible for me (as a single dev). |
|
I don't have a specific reason to expect an infinite loop, although there may be obscure bugs that cause non termination. The long runtime may have more mundane explanation: since Prepack currently unrolls every loop and inlines every call, its behavior is probably exponential for the kind of code found in Prepack itself. I'm afraid that it will be several years before Futamura projections become something we can reasonably expect of Prepack. |
|
To put a nail in this coffin, it finally did something after 7.5 hours of running: run out of heap memory! |
This is 100% not an issue but a question, so sorry for posting this here, but I don't see any other way to talk to the prepack community (any gitter/slack/etc channels?).
I'm having too much fun using prepack with interpreters & compilers. Today I was able to turn my interpreter, which interprets high-level instructions, almost into a compiler by running prepack on it with a given source code: https://twitter.com/jlongster/status/948247424023322624
I say "almost" because I don't really have a compiler yet. Prepack is running my VM and outputting the JS code itself. Evidently this is called Futamura projection. Since prepack isn't very fast though, this isn't extremely practical. I want just the raw compiler itself.
What I learned from that wiki page is we can take this even further. Supposedly I can generate an actual compiler by "specializing the specializer for the interpreter" - meaning run prepack on a program that runs prepack on the interpreter.
That would require prepack being able to run on itself. Is that possible? I'm going to try it tonight but given this is all blowing my mind, and I'm sure I'll run into problems, I'd like to know if anyone has tried that yet.
The text was updated successfully, but these errors were encountered: