Discussion: Compiling FCS to JavaScript

[ncave]

@7sharp9 @dsyme

Continuing the discussion from this thread, I'm looking for an advice on what would be a good replacement for BCL and FSharp.Core assembly metadata (types) in a JavaScript context.

Should we try to somehow export all the type metadata in some file format and use that instead of reading from assemblies? Can we generate F# type signatures from BCL and use that? If this misses the point entirely, any other ideas for a way forward? Essentially the missing piece is the TcGlobals.

Here is the current status:

• Untyped AST parsing works
• Typed AST is WIP, most of the files required for type checking already compile but still needs work as it's unclear (to me) atm what can we replace the BCL and FSharp.Core assembly metadata with.

You can take a look at the progress in this PR.

[dsyme]

@ncave Tricky but central issue. I've always thought we would use .NET metadata format but haven't thought beyond that.

The F# signature metadata format (see the signature resource) is a possible format since it can be turned into TAST nodes (that's what it's for). However it has all the elegance of a car crash. It doesn't cover .NET DLLs but maybe that's not such a problem as long as the stuff in TcGlobals is covered.

BTW there are some big cleanup PRs to integrate over from visualfsharp which may make some things simpler. Basically

1. less things in ILGlobals - just the very basics
2. more things in TcGlobals
3. better logic to search all referenced assemblies for system types

[dsyme]

@ncave Very exciting to see you work on this as it would unblock many scenarios if the full F# compiler service (and thus compiler) worked as JS

[7sharp9]

Yeah this is very exciting, its something Ive been meaning to look at just not had any time for ages!

My idea was to extract the AST for FSharp.Core and work on transpiling the typed ast, one of the big problems with that is still all the bcl mapping and the inline IL and static optimisation literals.

I did start to look at this before fable appeared using the mozilla spidermonkey AST but as usual real life got in the way.

[7sharp9]

Also, the soothsayer assembly visualiser thing I was working in gets the F# signatures from dlls and exports as json so that part is fairly easy to do.

Ive just deleted all my pretty screen shots so this one will have to do:

Not sure if that shows it off as it lacks constraint relaxing on the graph nodes but all nodes are shown for an assembly and the signature for the functions etc

[7sharp9]

@dsyme and I have talked about a skinny mscorlib and FSharp.Core to make this an easier job not sure on the way forward on stripping those assemblies down though.

[ncave]

Thank you for your input, looks like the closest options are to export/import already processed metadata:

1. Export/import to binary with TastPickle.
   FSharp.Core is already exported, need to export mscorlib's FSharpViewOfMetadata if possible.

2. Export/import to JSON.
   Need to see if export/import works for all types properly.

[ncave]

Had to redo the PR to merge latest FCS.

Notes on BCL metadata serialization:

• JSON serialization is a no-go as not everything can deserialize properly (and file size is too big).
• TastPickle works (when extended with TILObjectRepr serialization) but it results in big binaries as it does not optimize for space (no deduplication through tables).
• Reading metadata directly from assemblies works best, also code/resources can be stripped out from the assembly to cut down on the blob size. It's also better as other binaries can be referenced too.

Here is the current status:

• Untyped AST parsing works
• Typed AST parsing also works
• Next step: Compiling Fable with Fable - WIP

[7sharp9]

Sounds like great progress, thanks for looking at this stuff.

…

On 8 Jan 2017 6:39 pm, "ncave" ***@***.***> wrote: Had to redo the PR to merge latest FCS. Notes on BCL metadata serialization: - JSON serialization is a no-go as not everything can deserialize properly. - TastPickle works (when extended with TILObjectRepr serialization) but it results in big binaries as it does not optimize for space (no deduplication through tables). - Reading metadata directly from assemblies works, also code/resources can be stripped out from the assembly to cut down on the blob size. It's also better as other binary dependencies can be added. Here is the current status: - *Untyped AST parsing works* - *Typed AST parsing also works* - Next step: Compiling Fable with Fable - WIP — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#679 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAj7ypQQUqnxgu_LgKmdjIlAs4cuVnPfks5rQS1VgaJpZM4LHYEQ> .

[alfonsogarciacaro]

First of all, I've to say @ncave's job is really impressive! I thought that compiling FCS to JS was years away.

Not sure if this is completely related to this thread, but I think it's very important to find a JS-compatible portable format for F# type metadata if we're going to have a JS-only F# compiler. Right now, F# packages compiled with Fable are distributed with the .dll and the generated JS (plus a .fablemap file to link the F# source files with the corresponding JS ones), but the assembly is there just to hold the type metadata. The other option would be to distribute the F# source files but we decide this would slow down tooling (Discussion fable-compiler/Fable#498).

It'd be great if we have a specific format to hold the type metadata that can be read from a JS app (node, browser, Github Electron, VS Code extension...) and ideally that's a bit open, so we could add right there the information currently in the separated .fablemap file.

[ncave]

@alfonsogarciacaro As long as that custom format does not produce huge files it may make sense for some scenarios, but it will be yet another format for all consumers/producers to support.

As mentioned above and after trying some alternative formats, sticking to what we already have out of the box and keeping things simple makes a lot of sense to me, i.e. the assembly metadata is an open format and JavaScript is perfectly capable of reading it (and producing it if needed, I'm sure, but for now reading is all that's needed), and also the file size is smaller as it's already optimized for space.

That also solves the problem with other binary references.

[alfonsogarciacaro]

Thanks for your reply @ncave! I'm completely fine with the assembly format if it can be handled from JS. My actual knowledge of .NET IL is very limited so this may look as a dumb question but, currently when you compile a Fable package, a .dll is generated with the only purpose to hold the metadata (but it also contains the compiled IL); in the case of a JS-only Fable, would it possible to produce an empty .dll with only the type metada so it can be read from other Fable projects referencing it?

[ncave]

@alfonsogarciacaro Yes, it should be possible in a Node setting (not sure if it makes a lot of sense in a browser setting), if the full FCS is compiled with Fable. As of right now, only the Typed AST parsing is compiled (which is about 70-80% of FCS by volume), as that's all the Fable compiler needs. Compiling binaries by the node client is a different scenario that will require the full FCS.
(If it's just signatures that we need written out, then the effort is less as it can be done by the Typed AST serialization).

[matthid]

@alfonsogarciacaro
Sorry if this is nonesense but I think empty assemblies are actually already used in the .net ecosystem and called reference assemblies.

The F# compiler already has code to parse the IL. It's just not exposed nicely, so a combination of those two might help?

[ncave]

@matthid @alfonsogarciacaro Yes, as mentioned above you can definitely use it to export assemblies without code/resources to cut on the blob file size (but that's using the full FCS on dotnet).

[alfonsogarciacaro]

In case it helps, I'm providing a couple of links to Fable code for reference:

• This is where Fable checks if there's a .fablemap file next to the referenced .dll (so it's considered a Fable package)
• Then, when there's a reference to an FSharpEntity in the Typed AST, Fable will try to get the internal file containing it. Fable considers internal files those in the current project and those in referenced assemblies with a .fablemap.

Non-internal FSharpEntitys must be replaced (either by a plugin or the Replacements module) or Fable will throw an error.