How to read the AST with provided types

[alfonsogarciacaro]

I'm trying to compile a script using the JSON type provider with Fable. However, whenever I try to access an expression using a provided type an exception is thrown.

Repro steps

Try to compile the following script with Fable.

#r "../../../bin/FSharp.Data.dll"
open FSharp.Data
type Simple = JsonProvider<""" { "name":"John", "age":94 } """>
let simple = Simple.Parse(""" { "name":"Tomas", "age":4 } """)
simple.Age

Expected behavior

The AST is fully accessible and can be compiled to JS.

Actual behavior

When examining the tree and reaching simple.Age, trying to match the expression against any of the active patterns in BasicPatterns module results in the following error:

A call to 'FSharp.Data.Runtime.JsonValueOptionAndPath::get_Path(...)' could not be resolved

Stack trace:

  at Microsoft.FSharp.Compiler.SourceCodeServices.ExprUtilsImpl.bindILMethodRefToSymbol (Microsoft.FSharp.Compiler.SourceCodeServices.cenv cenv, range m, Microsoft.FSharp.Compiler.AbstractIL.ILMethodRef ilMethRef) <0xd6f8410 + 0x0073f> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExprConvert.ConvExprPrim (Microsoft.FSharp.Compiler.SourceCodeServices.cenv cenv, Microsoft.FSharp.Compiler.SourceCodeServices.ExprTranslationEnv env, Microsoft.FSharp.Compiler.Expr expr) <0xd6ee7c8 + 0x009bf> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExprConvert.ConvExprPrimLinear (Microsoft.FSharp.Compiler.SourceCodeServices.cenv cenv, Microsoft.FSharp.Compiler.SourceCodeServices.ExprTranslationEnv env, Microsoft.FSharp.Compiler.Expr expr, Microsoft.FSharp.Core.FSharpFunc`2 contf) <0xd6f65f0 + 0x00463> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExprConvert.ConvExprPrim (Microsoft.FSharp.Compiler.SourceCodeServices.cenv cenv, Microsoft.FSharp.Compiler.SourceCodeServices.ExprTranslationEnv env, Microsoft.FSharp.Compiler.Expr expr) <0xd6ee7c8 + 0x00a1f> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExprConvert.ConvExprPrimLinear (Microsoft.FSharp.Compiler.SourceCodeServices.cenv cenv, Microsoft.FSharp.Compiler.SourceCodeServices.ExprTranslationEnv env, Microsoft.FSharp.Compiler.Expr expr, Microsoft.FSharp.Core.FSharpFunc`2 contf) <0xd6f65f0 + 0x00463> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExprConvert+ConvExprOnDemand@903.Invoke (Microsoft.FSharp.Core.Unit unitVar0) <0xd6ee758 + 0x0002f> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExpr.get_E () <0xd6ee710 + 0x00030> in <filename unknown>:0 
  at Microsoft.FSharp.Compiler.SourceCodeServices.BasicPatterns.|Call|_| (Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExpr e) <0xe089fa0 + 0x00017> in <filename unknown>:0 
  at FSI_0002+printDecls@48-10.Invoke (Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExpr _arg1) <0xe089e38 + 0x00013> in <filename unknown>:0 
  at FSI_0002.lookup (Microsoft.FSharp.Core.FSharpFunc`2 f, Microsoft.FSharp.Compiler.SourceCodeServices.FSharpExpr expr) <0xe089da8 + 0x00018> in <filename unknown>:0 
  at FSI_0002+printDecls@36.Invoke (Int32 i, Microsoft.FSharp.Compiler.SourceCodeServices.FSharpImplementationFileDeclaration decl) <0xd6e6c28 + 0x0048b> in <filename unknown>:0 
  at Microsoft.FSharp.Collections.SeqModule.IterateIndexed[T] (Microsoft.FSharp.Core.FSharpFunc`2 action, IEnumerable`1 source) <0xd6e2180 + 0x000d0> in <filename unknown>:0 
  at FSI_0002.printDecls (System.String prefix, Microsoft.FSharp.Collections.FSharpList`1 decls) <0xd6e6a40 + 0x00067> in <filename unknown>:0

Known workarounds

None.

Related information

• Operating system: OSX El Capitán
• Branch: FSharp.Compiler.Service 3.0.0
• .NET Runtime, CoreCLR or Mono Version: Mono 4.4.0
• Editing Tools (e.g. Visual Studio Version): VS Code/Xamarin

[alfonsogarciacaro]

UPDATE: I'm using a simple script to inspect the AST. I've noticed when this script is not in the same path as the file containing the code above I can actually read the AST, but it gets converted to exception raising. Both Simple.Parse(""" { "name":"Tomas", "age":4 } """) and simple.Age expressions become:

Call
  (null,val raise,[],[type Microsoft.FSharp.Core.obj],
   [Const (1,type Microsoft.FSharp.Core.int32)])

I guess this is because the ASTViewer.fsx script cannot access the path where FSharp.Data.dll is.

[7sharp9]

I cant tell from a quick review of the compiler code, I reckon debugging bindILMethodRefToSymbol should give a better idea.

[alfonsogarciacaro]

I replaced the FSharp.Compiler.Service reference with a custom build in Debug mode. However now when I try to parse the script I get the following error: Invalid host of cross-targeting type provider. Maybe this is related with the following issue?
dotnet/fsharp#23

In any case, I just tried with a very simple Type Provider and it worked so it might be not a problem with type providers per se but something else. I'll try to investigate more.

[alfonsogarciacaro]

I narrowed down the problem. It seems to happen when code from the BCL or from another library is called in the quotations. For example, if I have the following provider:

[<TypeProvider>]
type MavnnProvider (config : TypeProviderConfig) as this =
    inherit TypeProviderForNamespaces ()
    let ns = "Mavnn.Blog.TypeProvider.Provided"
    let asm = Assembly.GetExecutingAssembly()
    let createTypes () =
        let myType = ProvidedTypeDefinition(asm, ns, "MyType", Some typeof<obj>)
        let myProp = ProvidedProperty("MyProperty", typeof<string>, IsStatic = true,
                                        GetterCode = (fun args -> <@@ "Hello world" @@>))
        myType.AddMember(myProp)
        [myType]
    do this.AddNamespace(ns, createTypes())

This works. But if I replace <@@ "Hello world" @@> with, say <@@ Regex(%%args.[0]) @@> it will fail with a similar error to that reported above:

A call to 'System.Text.RegularExpressions.Regex::ctor(...)' could not be resolved

Interestingly, calling methods from FSharp.Core (e.g. <@@ Array.init 5 (fun _ -> %%args.[0]) @@>) works.

[Jand42]

I have also seen that some IL method calls produced by type providers are not yet resolved correctly to their F# method symbols. These include:

• Static methods with curried arguments
• Methods with a ParamArray argument
• Compiler generated methods of F# records/unions
• Interface methods

However these are all handled correctly when creating quotation AST for ReflectedDefinition. Of course the FSharpMemberOrFunctionOrValue symbol exposes much more information than just the MethodInfo encoded in quotations.
I am contemplating if that resolving everything is too complex, there could be an extra case for wrapping IL methods to expose as symbols. I already have stashed progress on this from some time ago, I will share it if I can mine it out.

[7sharp9]

There is a comment in the Expr.fs code that mentions curried args may not be represented properly.

[alfonsogarciacaro]

Closing the issue for now. I'm solving the issue for Fable by emitting Fable's AST directly from the Type Provider. The disadvantage is the TP must be explicitly made for Fable, but most of the current ones would have required special treatment anyway, so it should be fine.

[dsyme]

@alfonsogarciacaro Thanks for the update. I'll reopen to keep this issue active as we need to fix the problem. For erasing providers you should be able to read the TAST correctly.

[Jand42]

My pull request fixes multiple kinds of failures that I could find.

• CompiledName, type parameter constraints, curried members are handled.
• FSharpFunc.Invoke, union and record generated members are resolved to their Expression representations. Tuple construction and item get is currently not.

There is a commented out failing test where F# type checking itself fails (same with current fsc.exe)

[alfonsogarciacaro]

@dsyme I've tried to compile a type provider using the latest version of FCS but I'm still experiencing the same issue reported on top. Should we reopen the issue?

[Jand42]

@alfonsogarciacaro WebSharper 4 could translate your snippet: http://try.websharper.com/snippet/JankoA/00009T

It uses FCS 6.0.2, but still version 2.2.5 of FSharp.Data, I will test updating that too.

[Jand42]

Newer FSharp.Data fails. One change is that it is including assembly for redirection for multitargeting.
I have also included latest files from TypeProviders.StarterPack in WebSharper.UI.Next template TP for assembly redirection and those work now.
For some reason I had to try to create the ProvidedTypesContext in two ways for it to work both in FCS code service and WebShaper 4 translation, see https://github.com/intellifactory/websharper.ui.next/blob/zafir/WebSharper.UI.Next.Templating/Main.fs#L105

[alfonsogarciacaro]

@Jand42 Thanks a lot for the info! As you say, trying to read the snippet AST with my simple script fails, but if I try to compile it directly with Fable works (well, it doesn't finish compilation because there's no replacement for System.IO.StringReader constructor, but that's another problem).

I'm closing the issue again, sorry for the trouble! 😅 I'll try to make the provider work with Fable and if I find another problem along the way, I'll report it.

Thanks again for your help!