Cross-platform libraries

[alfonsogarciacaro]

After #3212 I've been thinking about how to handle IO with Fable. So far, I've always been reluctant to extend support for .NET BCL, but if we want to promote Fable as a true cross-platform solution, we will need to count with ways to deal with things like files, http or json in a consistent manner across languages. However, I'm still not sure if the BCL is the best way for this: because it's easy to miss things (overloads, new APIs...) and because it increases the workload in this repo.

In my opinion it would be preferable to have cross-platform libraries with minimal APIs that we can control better. There have been some attempts by the community but probably we should try to discuss a common approach and publish a couple of libraries that the community can also use as a model.

One solution is two publish two packages that have the same API, like Thoth.Json and Thoth.Json.Net do, but this has caused problems to user, for example when they want to write a cross-platform library (not an app) that uses Thoth.Json as dependency.

The alternative is to have a single package and in the source files use #if FABLE_COMPILER for JS-specific code like Fable.React does. But this can become cumbersome pretty quickly and adding more languages to the mix won't make it easier.

Recently I added the FABLE_COMPILER property when invoking MsBuild so devs can also use it in the .fsproj as well. I'm wondering now if it makes sense to add the language-specific constant as well (e.g. FABLE_COMPILER_RUST) so we can have an .fsproj that includes different files depending on the language target.

    <Compile Condition="'$(FABLE_COMPILER_RUST)' == 'true'" Include="Rust/Lib.fs" />
    <Compile Condition="'$(FABLE_COMPILER_PYTHON)' != 'true'" Include="Python/Lib.fs" />

We could have a common signature file to ensure all files expose the same API, and when developing we could have different Solution Configurations that select one language or the other.

What do you think? Should we give this a try or do you have any other suggestion (or do you still prefer to have more BCL support)?

@MangelMaxime @dbrattli @ncave @alexswan10k

[ncave]

My +1 for sticking to BCL support as API (i.e. System.IO), and managing it as a separate package (e.g. fable-io or something like that). We can have a feature matrix in the package repo to document what is supported for which language.

[alexswan10k]

This is a fun one. BCL is the obvious impulsive choice since it is the default any developer is going to reach for when trying to do something like file interop.

That being said, there are rough edges that I am not crazy about, namely:

• Some API's exist, some don't, and some only work when you install some special library. None of this is obvious to a user consuming these. It is not a great user experience.
• A lot of System api's are not very F# friendly (out params, mutability, no tuples), and are also showing their age (from pre generic days, etc), as all API's have to exist forever due to creating breaking changes. Again, not a great user experience when you add it all up. It is easy enough to thinly abstract over this, but I do feel like I need to do this for almost every project.

I am torn. I guess I am still probably inclined to vote for BCL first, with separately installable libraries to pad it out, but only marginally.

It was quite refreshing in Rust to see a very modern, minimalist BCL, where nearly everything is delegated to packages. On the flip side though, the rust ecosystem correctly handles transitive collisions, so generally, you can have deeper dependency graphs without problems. You often end up with deeper graphs and more collisions when you push everything out to packages as they try to reuse common code not found in the BCL (think serialization for example). As we know this is problematic in .NET as the runtime only allows 1 version of the DLL to be loaded at a time. One can dream I guess.

The signature file is an interesting idea by the way.

[MangelMaxime]

json in a consistent manner across languages.

One solution is two publish two packages that have the same API, like Thoth.Json and Thoth.Json.Net do, but this has caused problems to user, for example when they want to write a cross-platform library (not an app) that uses Thoth.Json as dependency.

For information, Thoth.Json next version will provide a single library which works on all the platform at the same time.

---

I suppose BCL via a separate library is probably the easiest solution. It will also make the documentation job/learning curve easier because people will be able to use what they already know.

Personal comment/experience:

However, the BCL is not really friendly and I find that it is not fun to play with. I had to use it for Nacara and in the end I decided to drop it in favour of minimising my needs to write code that use the BCL especially for watchers.

I am still uncertain if this is related to the BCL API per se or the .NET runtime.

[dbrattli]

I little background. The main reason for #3212 was that I tried to transpile some Bioinformatics examples at NDC and suddenly discovered that it contained a File.ReadAllText that Fable choked on. File handling is also something I will eventually need for Fable.Giraffe that could be written in a way so it could run on any runtime e.g .NET, Python, Node or being ultra fast using Rust. So I started adding System.IO functionality without really thinking about any of this.

Leaning towards the BCL is the easiest and more convenient choice. BCL might be the easy (convenient) choice, but as you all agree, it's not a great (simple) experience for F# programmers, and we could do much better with Fable. The idea of minimal Fable library(s) that makes it easy and simple to write cross runtime applications is super interesting the more I think of it. An F# optimized library you would use even when targeting .NET, then as a bonus your code will also work when targeting Python, Node.js, Rust or Dart. I find such an idea really really interesting. One Fable to rule them all ... even when using .NET.

[alexswan10k]

I do like this. A clean slate.

[ncave]

BCL might be the easy (convenient) choice, but as you all agree, it's not a great (simple) experience for F# programmers...
The idea of minimal Fable library(s) that makes it easy and simple to write cross runtime applications is super interesting...

While this is definitely true, I don't see a reason why we can't do both, i.e. provide support for some targeted BCL IO APIs and a more pleasing F# facade built on top of it. This way we can still enable easy path for porting existing IO code via the well-known and well-documented BCL APIs, and also enable an exciting new API for those that would be willing to try it.

IMO this would expand the potential reach of this library to more projects, and perhaps get more collaboration in the process.

[alfonsogarciacaro]

Thanks a lot for your comments! I understand that increasing support for the BCL has some benefits:

1. Avoids the need of designing new APIs
2. Increases the chances that existing projects become Fable-compatible

Still, I'm not sure they outweigh the drawbacks:

1. Support for the BCL will always be incomplete
2. Users will need to check constantly a feature matrix (that we need to maintain) or do trial and error to know what's supported by Fable, IDE won't help
3. There may be crucial differences in the way some APIs behave at runtime

One example of 3 (and I'm sure there are many more) is File.ReadLines API. I always use this when handling text files in dotnet. However, afaik in NodeJS it's not possible to read a file line by line synchronously. The only way to emulate this is to load the whole file and then split it by lines, this can take users by surprise as memory may blow up for big files when they don't expect it (on the other hand, we can read lines in NodeJS as AsyncSequential which is actually supported by Fable).

I also think we can handle the points listed in BCL benefits by writing libraries that resemble the BCL APIs. Say, for example, users only need to replace open System.IO with open Fable.IO and the IDE will automatically tell them what is supported and what is not (remember the library will also be dotnet compatible so in the case of dotnet the library will just redirect to BCL APIs). We can even use the Obsolete attribute to warn users in cases like File.ReadLines that they should use an alternative method.

@ncave When you say adding support for the BCL in a separate package, do you mean that users can add a Nuget reference to their project and this will automatically add the new replacements? At the moment this is not possible with Fable, but we can try to extend the plugin mechanism to support this.

[ncave]

@alfonsogarciacaro

However, afaik in NodeJS it's not possible to read a file line by line synchronously... on the other hand, we can read lines in NodeJS as AsyncSequential

I assume you mean async stream reading using readline.
We can just choose to implement only the async BCL APIs (e.g. System.IO.File.ReadLinesAsync etc.).

do you mean that users can add a Nuget reference to their project and this will automatically add the new replacements? At the moment this is not possible with Fable

The IO package source will be compiled with the rest, no? Maybe I'm missing the point you're describing.

I guess my argument for a combined IO library (mostly static BCL IO API + a nice F# facade), is to increase its chance to be useful to a larger audience, and prevent it from falling into obscurity. But perhaps I'm overthinking it, there is always room for more than one IO library, if there is somebody to support it.

[alfonsogarciacaro]

At any rate, I would prefer to move IO, http, etc support to a different repo. Doing it here has several disadvantages:

• Adds more complexity to this repo
• Forces us to push a new Fable release for every library change in any language
• Many users find it more daunting to contribute to the compiler repo than a library

It's true that we have readline in NodeJS, but even this cannot be used to emulate File.ReadLinesAsync (at least I couldn't manage to do it). This is because IEnumerable still has a synchronous API but readline reports each line asynchronously, so it can only be handled by something like AsyncSequential or an observable. Also, to support BCL -Async APIs in JS and Dart we still need to add support for Task.

[ncave]

@alfonsogarciacaro Makes sense.

At any rate, I would prefer to move IO, http, etc support to a different repo.

Absolutely, that's what I meant by separate package, apologies if that was unclear.

Ultimately, the best outcome is something that people would be happy to contribute to, so in that sense I don't really have a preference either way, just sharing some non-strongly held opinions to be taken with at least 2 grains of salt.

[ncave]

On that note, are there any pre-existing functional IO libraries to model after, or are we starting with a clean slate?

[alexswan10k]

Oh one more point about the BCL. Because it starts from C# foundations, it assumes overloads. Overloads are a PITA and are sometimes not supported in other languages. These then often require replacements to be coded into the compiler, which mean they are more than just a namespace redirect.

Because of this, splitting this out into a library could create complexity as you might have to make changes in 2 repos. Can plugins help here perhaps?

[alfonsogarciacaro]

@alexswan10k The way I see it, if we want to create these libraries in their own repos we have two options:

1. As a "normal" Fable library: the only thing we'd need to find is a convenient way to manage different code for each language target.
2. As a plugin: the user would add a package but wouldn't use the code directly. When Fable starts it will detect the plugin to extend "Replacements", so when it cannot find a built-in replacement it will ask the plugin instead. I assume this will work similar to the current redirections as for StringBuilder.

Note that 2. is not currently possible but it should be feasible to extend the plugin mechanism for that purpose.

In both cases the packages would be distributed through Nuget independently of the Fable tool. I guess that in the case of 1. we'll have to coordinate more between languages because everything will be in the same repo, while in the case of 2. we can distribute replacements for each language separately.

[MangelMaxime]

I think in the end, it depends on what we want to focus on.

Do we want to make contribution easy?

A normal F# library is probably easier to contribute to than plugins or replacements.

Do we want to make the API easy to discover?

A normal F# library, means that only the supported function are available. No need for a compatibility matrix

Do we want to re-use existing knowledge ?

Using the BCL, is the to go in this case. Because, we don't have to design the surface API

Do we prefer to make the user experience first class or make it easier for us to "design"?

Similar to "API easy to discover", because there will be no "No replacement found for XXX".

---

I think that whatever direction, we choose we need to distribute it as a Package NuGet to avoid polluting the F# compiler.

Doing so, will avoid forcing the user to upgrade the compiler itself to get bug fixes for an API. Upgrading the compiler, is always something that can be dangerous if a behavior changed in it etc.

While, upgrading a library is more "normal".

[dbrattli]

What about modeling such a library e.g Fable.Core.IO closer to FSharp.Core than System.IO BCL? FSharp.Core.String is more convenient for F# developers than System.String, but you can still use both if you prefer. If you use Fable.Core.IO then your code would work across Fable incl .NET.

[absolutejam]

The hardest part of this is everyone agreeing upon what the API surface looks like 😂

I assume we can use a single .fsi file for each area (eg. System.IO/Fable.Core.IO) then each language provides an implementation?