Serializing in Fable 2

[alfonsogarciacaro]

Continuation of discussion started here SaturnFramework/Saturn#33

There're also some interesting comments about implicit vs explicit serialization in this Twitter thread.

Probably I wasn't good enough expressing my intentions in the issue linked about and (like any breaking change) this has started some controversy. I'll try to expose my current thinking below to have a better foundation for the discussion :)

• Fable 2 alpha would ship initially without reflection support. My assumption is this will mainly affect ofJson/toJson as I think there are not many other of reflection right now in Fable. Please note the alpha version won't be obviously meant for production, but for users to give a try and provide feedback.

• Why drop reflection support? Well, at the end I'm rewriting large parts of the code to (hopefully) make it cleaner, more maintainable and appealing to contributors. In the refactoring I've noticed the reflection model in Fable is not consistent and pollutes a lot both the generated JS code (reducing bundle sizes is one of the main goals for Fable 2) and Fable code base. That's why I'd like to start fresh with Fable 2 alpha to see users real needs and reimplement it from scratch (or not, if we don't need it).

• How would reflection be reimplemented? Right now serialization info is embedded in the types. This makes the types look fatter and it's a problem when people are comparing alternatives in the REPL as they will see Fable generates much more code for simple types (it has already happened). For Fable 2 I've considered two options:

  • Make reflection info available through a static method in the hope that it will be removed by tree shaking when building for production. This would probably be the easiest way but it will still display the code in the REPL.
  • Generate reflection info in the call site to replace typeof<Foo> for example. I think this will be a good for most of the cases, but it can penalize apps that use reflection extensively, as there will likely be duplicated code.
  • I also considered a third option: inject an extra file with the reflection info so it remains hidden to the user and it's only retrieved if needed. But the way Fable interacts right now with JS bundlers and tools (Webpack...) makes this complicated.

• How could auto-serialization work in Fable 2? Records will become plain JS objects, and unions, JS arrays. So in most cases, just calling native JSON.parse/stringify would work. The problem would be things that are not compatible with the browser JSON api, like Maps, Sets, dates, longs, etc... So Fable will still need to know information about the fields at runtime in order to properly inflate/deflate them.

• What I don't like about current serialization? There are a few things

  • It works for most of the cases but not all the cases, and it can give you surprises at runtime, which is not something good if we are selling a safe language.
  • It somehow mirror Newtonsoft.Json on the frontend-side, and some people expect it to support things like attributes, embedding type info in the json, etc.
  • Other languages I know (F# included) don't have serialization embedded in its core system. This increases the maintenance cost of Fable code base and makes it more difficult to refactor it (as it happened with Fable 2). It would make me very happy to move serialization to an external library.

• What are the alternatives? As commented in the issue above, the main alternative at the moment that should work with Fable 2 alpha right away is Thot.Json. This library gives you more control over the generated JSON and much better validation. The only drawback is you need to write the decoders yourself, but there's already work to generate them automatically.

[davidtme]

One of the reasons I like Fable is because of its JSON support. Having to go and add encoder/decoder function all over the place is going to be a hard sell. I remember in very, very early release, F# types where very close to JS objects and most of the time you could just JSON.parse/stringify and knowing that limitation meant I could just about get it working. Unfortunately as Fable got better I started using Lists and DateTimes in my JSON so if they went that would be a bit of a project rewrite :S

If Thot.Json code generation could be part of the build chain for both client and server (in net46x - yeah I know, one day I’ll have to upgrade), maybe as some sort of pre build event which calls fake (which I use to deploy a sql database for FSharp.Data.SqlClient) then could be workable? Or are MS build tasks/targets still a thing... How does paket to it's auto restore?

I fell out with Newtonsoft.Json years ago.

[jgrund]

Just to generate a counter-point, I am currently using reflection in a production Fable 1 node app both for deserializing message types in a DU:

https://github.com/intel-hpdd/device-scanner/blob/16233ff62ad710aa02d6c8fe8acdbcad0c3e1e3e/IML.DeviceScannerDaemon/src/Main.fs#L13-L20

and for auto-determining the encoding types of node streams so I can compose transforms together and have them typechecked / configured as expected at runtime:

https://github.com/intel-hpdd/fable-import-node-powerpack/blob/4004f9c430517c1f26bf47f9c2f766598e500b0d/fable/Stream.fs#L99-L120

https://github.com/intel-hpdd/device-scanner/blob/16233ff62ad710aa02d6c8fe8acdbcad0c3e1e3e/IML.DeviceScannerDaemon/src/Main.fs#L23-L42

Consumers of this service are expected to send a message to the daemon to get streamed data: https://github.com/intel-hpdd/device-scanner/tree/master/IML.DeviceScannerDaemon (so keeping it a record / string instead of an array is ideal) but I think I can just string match before attempting to serialize to get around that.

The larger issue for me is losing the ability to auto-configure node streams based on lack of reflection info at runtime, but I may be able to work around this as well.

[MangelMaxime]

@davidtme I am exploring way to integrate Thot.Json decoders/encoders generation in the build chain or via TP support etc.

I will open an issue on Thot.Json to track my futur progress on this subject.

For your info, I just publish Thot.Json.Net who provide the same API as Thot.Json for Fable.

The idea being you can use compiler directive to share the code:

// By adding this condition, you can share you code between your client and server 
#if FABLE_COMPILER
open Thot.Json
#else
open Thot.Json.Net
#endif

Documentation

[MangelMaxime]

@alfonsogarciacaro Would it be possible to detect type info at compilation time and adapt the serialization process depending on that ?

[alfonsogarciacaro]

The type info is available at compile type, yes. Although if we want both to access type info without reflection and move serialization outside the compiler, we would need some kind of plugin (which will be also unavailable in Fable 2 alpha 😉). But what do you mean exactly by "adapting the serialization process"?

[Zaid-Ajaj]

TBH, I find it weird to exclude functionality just to make the generated javascript appeal more to people evaluating Fable. I know it is the motto of Fable to generate nice javascript and I really like it, but IMO, the stongest selling point Fable has as a compiler to javascript, is using F#, the awesome interoperability with the JS ecosystem and being applicable to any javascript runtime. Nice generated javascript is just that: nice to have. (How about actually making a poll to ask users?)

it works for most of the cases but not all the cases, and it can give you surprises at runtime, which is not something good if we are selling a safe language.

Then we implement the failing exceptional cases. Automatic serialization/deserialization failures seem to come in places where we don't have enough metadata at runtime.

It somehow mirror Newtonsoft.Json on the frontend-side, and some people expect it to support things like attributes, embedding type info in the json, etc.

I would say it provide the same ease of Newtonsoft.Json and users are already very happy with it to use as the default. If people want more control and customizations then Thot.Json or Fable.SimpleJson provide the needed level of control.

Other languages I know (F# included) don't have serialization embedded in its core system. This increases the maintenance cost of Fable code base and makes it more difficult to refactor it (as it happened with Fable 2). It would make me very happy to move serialization to an external library.

I agree that maintenance costs are increased with ofJson<'a> and 'toJson` but it is really worth it. If we were to make an external library then reflection must be well implemented for consumers to be able to write such converters

[MangelMaxime]

@Zaid-Ajaj From my point of view, it's not only to make the generated JavaScript look nice but also to reduce the bundle size.

We see lot of people complaining about that especially in country where internet connexion is still slow :)

@alfonsogarciacaro I was thiking about making Fable customize the serializer call to support Maps, Sets, etc. by using the property name as a key but it's kind of a bad idea. This would mean a lot of code duplication and also each serializer call would not be the same. Let's forget this idea :)

[MangelMaxime]

@alfonsogarciacaro Would it be possible to embed the types info into a "type module".

The module could contain all the type info of the app and if it's not used Webpack should be able to remove it no ?

[alfonsogarciacaro]

Thanks a lot for your comments. I understand that the current ofJson/toJson helpers are very convenient and they shouldn't be replaced unless we provide something that's almost as easy to use. Thanks also for the samples @jgrund, it's very useful to see how Fable is used in production. For what I can see reflection is only used for ofJson, is there any other place where you use typeof<'T> or similar?

Thanks also for your insights @Zaid-Ajaj. As Maxime says, it's not only a matter of making the code more readable (actually in some places it may become less readable but more optimized in Fable 2), but reducing bundle sizes and making the generated code a better fit for JS optimization tools. It's also a matter of survival as now that we have more mature and complex libraries like Fulma if the bundle sizes are too big, users may consider other options (and we know competition is tough among functional languages compiling to JS). Fable tries to compile the F# language with most of its features and FSharp.Core and some of the BCL and deciding whether reflection is an F# language feature or part of the BCL/.NET runtime is another topic. Of course, it's something very useful, but I'd like to use Fable 2 alpha release to assess its costs/benefits and whether we've viable alternatives.

@MangelMaxime Yes, that's the third option I was considering above. I've some doubts with watch compilations, because this Types module may become in an inconsistent state. But I'm not sure, I guess we can try.

Again, thank you all for your comments, they're really useful and please be sure it's the least of my intentions to do anything that can hurt current Fable users. I'll try to release a Fable 2 alpha version so it's easier to compare the alternatives and their effects 👍

[et1975]

My single most important reason for picking Fable over Elm was being able to use F# on both sides and reuse the types across the tiers. I don't need the reflection per se, but like @davidtme said, the earlier implementation "just worked". I'd be comfortable with giving up on some of the aspects of the type system for the purpose of JSON serialization, but that's true for pretty much any cross-platform serialization format and I'm fine with that.

Serialization extracted into a compiler plugin would be a great compromise, if you could pull it off ;)

[jgrund]

is there any other place where you use typeof<'T> or similar?

Sorry, didn't highlight the relevant section. https://github.com/intel-hpdd/fable-import-node-powerpack/blob/4004f9c430517c1f26bf47f9c2f766598e500b0d/fable/Stream.fs#L144

By doing this, I'm able to set a streams encoding and the state of the readable side just from compile time information, which saves me from configuring each stream manually with the information that's already encoded by the types.

Based on that, I can do things like compose streams together without worrying about hardcoding options beforehand, which is pretty nice: https://github.com/intel-hpdd/fable-import-node-powerpack/blob/4004f9c430517c1f26bf47f9c2f766598e500b0d/test/Stream.Test.fs#L221-L232

[irium]

Another clever reflection usage here:
https://github.com/Zaid-Ajaj/Fable.Remoting/blob/master/Fable.Remoting.Client/Proxy.fs#L70

and here:
https://github.com/Zaid-Ajaj/Fable.Remoting/blob/master/Fable.Remoting.Client/Proxy.fs#L169

Entire Fable.Remoting project idea is based on availability of reflection both at server and client.

[zpodlovics]

Probably not a popular request - unless the communication is latency or performance sensitive - but how about supporting raw value types for transport (with or without zero-copy support) ? For example supporting .NET packed and unmanaged ("blittable") value types? By definition it have the same representation everywhere both in native and both in .NET. This raw representation could be critical, as sometimes the json serialization/deserialization just too cpu/memory/gc/latency intensive on the server (especially with lot's of clients and/or high message rate). Zero copy on the client could be also supported by using a ArrayView/TypedArray/DataView backing (the fields will read/write directly from/to the buffer).
Maybe this could be also usable for NodeJS native interop too (with ffi).

[<Struct>]
[<StructLayout(LayoutKind.Sequential, Pack=1)>]
type Vector3 =
    val X: int32
    val Y: int32
    val Z: int32
    new(x,y,z) = {
      X=x;
      Y=y;
      Z=z;
    }
    new(dataview: DataView,offset) = {
    //TODO
    }

[alfonsogarciacaro]

Thanks for your comment @zpodlovics. I must say I'm not familiar with this kind of serialization. Are you talking about JSON or binary serialization? (I've played a bit with binary serialization using Google flatbuffers protocol in one project, but it involved a lot of boilerplate code.) Can you give an example of how the serialized data would look like? It may be a bit difficult to have the exact same representation both in .NET and JS as Fable removes some data from the types to optimize them in JS code and runtime. Besides the limitations in JS, like not being able to define value (struct) types.

Also, as commented above, I'd prefer to keep serialization separated from the compiler core code. In Fable 2 we'll probably keep some kind of default JSON serialization as it's now for users' convenience, but more complicated things should preferably done in a separate package.

[zpodlovics]

@alfonsogarciacaro Thanks for the comment!

Well, it's not really a serialization format, but a direct memory representation as it would be represented in a native C struct. The "serialized" format would exactly looks like a c struct (native) [1]. This also required for native interop, as it use the same memory representation as the native application. The earlier example will use 3 * 4 = 12 byte memory region. A Struct could be "emulated" as object with a method that replace the backing memory region (.Wrap).

The earlier struct could be translated to something like this pseudocode using the JS DataView API (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView/getInt32):

eg.:

type Vector3 =
    val mutable view: DataView
    val mutable offset: int32
    static member XOffset=0
    static member YOffset=4
    static member ZOffset=8
    new(v: DataView,o:int32) = {
      view=v;
      offset=o;
    }
    member __.Wrap(v: DataView,o:int32) = {
      view=v;
      offset=o;
    }
    member __.X
        with get() = __.view.getInt32(__.offset+Vector3.XOffset)
        and set(v) = __.view.setInt32(__.offset+Vector3.XOffset,v)
    member __.Y
        with get() = __.view.getInt32(__.offset+Vector3.YOffset)
        and set(v) = __.view.setInt32(__.offset+Vector3.YOffset,v)
    member __.Z
        with get() = __.view.getInt32(__.offset+Vector3.ZOffset)
        and set(v) = __.view.setInt32(__.offset+Vector3.ZOffset,v)       

Yes, this could be created manually, but doing it manually would time consuming and error prone. Especially if I the compiler already has all the information needed: the types, the memory layout, fields offsets, etc. Some kind of compiler extension plugin/api/ast would be also perfectly fine with custom created codegen. Eventually everybody will face the binary problem - how have to read/write binaries eg.: image/audio/video/document/network packets/etc.

Built-in raw memory repreesentation as serialization format for interop (to support ffi) could have an incredible value to everybody. On every platform the developers have two options to manipulate the platform: 1) write the interop code as platform native code (platform compiler, different toolchain, different build/test/deploy process, etc) 2) write code that represent raw memory for interop + ffi. It's not an accident that the .NET have built-in pinvoke support.

C:
[1] https://en.wikipedia.org/wiki/Struct_(C_programming_language)
.NET
[2] https://www.developerfusion.com/article/84519/mastering-structs-in-c/
JS
[3] https://github.com/TooTallNate/ref-struct

[alfonsogarciacaro]

Thanks a lot for the more detailed explanation @zpodlovics! I explored using data views to emulate structs in the past but I didn't do much because there're still limitations (like only being able to use numbers, nesting structs is difficult, copying a struct out of an array is also tricky, etc). Also, it's already possible to better control memory with Fable using numeric arrays, as they will be translated to Typed Arrays, but this hasn't had much appealing among users so far.

We probably should open a new issue for this since, as you said, it's not exactly related to serialization. I cannot consider it a priority at the moment, but it'd be very nice to see contributions on this regard. The plugin story for Fable 2 is still TBD but we could make it so it allows something like this if you're interested in working in such a plugin.

One thing to consider though is WebAssembly support is coming to .NET/F# and the memory representation in that case will be closer to the .NET model. When this happens, Fable may remain as a way to integrate F# closer to JS ecosystem (as it's now) and take advantage of the current tooling/libs available for building the frontend of your apps.

[MangelMaxime]

@alfonsogarciacaro Can we close this issue ?