New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC FS-1023 Discussion] - Allow type providers to generate types from types #125

Open
kurtschelfthout opened this Issue Sep 22, 2016 · 56 comments

Comments

Projects
None yet
@kurtschelfthout
Member

kurtschelfthout commented Sep 22, 2016

This is the discussion thread for RFC FS-1023.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 3, 2016

Would another type providers type be valid for input?
Would erasing and generative both be supported in this scenario?

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Nov 3, 2016

Would really like this to be able to know about existing F# types and not just work strictly in terms of CLR level types.

Also Run-time type and IL generation seems like a poor alternative if unsafe code is not allowed for cross platform .NET Core libraries. I may be mistaken about this.

@dsyme

This comment has been minimized.

Contributor

dsyme commented Nov 3, 2016

@7sharp9 asks:

Would another type providers type be valid for input?

I suppose so. Tricky to test thoroughly though. Would you see the full form or the erased form?

Would erasing and generative both be supported in this scenario?

I think it would be difficult to get generative type providers working with this. Generative results are provided as an assembly. That assembly would have to include references to the assembly being compiled (without actually having that assembly available on-disk yet).

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Nov 3, 2016

I think it would be difficult to get generative type providers working with this. Generative results are provided as an assembly. That assembly would have to include references to the assembly being compiled (without actually having that assembly available on-disk yet).

Erasing type providers only would severely limit the potential, but it seems maybe a good target as a first step forward. It seems to me that they could still be used to generate lenses or serialization but not more complex things like type proxies or future reflectable artifacts.

Maybe for the sake of moving this forward it would be best to limit the scope as much as possible and allow more complex use cases later.

@ReedCopsey

This comment has been minimized.

Contributor

ReedCopsey commented Nov 3, 2016

Would erasing and generative both be supported in this scenario?

I think it would be difficult to get generative type providers working with this. Generative results are provided as an assembly. That assembly would have to include references to the assembly being compiled (without actually having that assembly available on-disk yet).

All of the use cases I'd personally have for this feature would pretty much require generative type providers - erasing really wouldn't add much benefit.

Most of my "real world" use cases that I can think of where I'd use this in the near term would require non F# code to be able to use the generated types - and expect them to implement specific interfaces, etc.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 3, 2016

Im struggling to invent many actual real world uses for this feature. For me this look like it just provides a typed input to be a parameter to be queried, but that does give much more than what you can do now with tom. yaml, xml, json as input parameter.

I could see a use in using a record as an input parameter to a type provider, where the record only had primitive types. At compile time the record could be erased and the contents were available as input parameters as normal. But thats mainly as a means for efficiently inputting data inot a type provider.

Maybe Im missing some bigger picture here, anyone want to enlighten me?

@ReedCopsey

This comment has been minimized.

Contributor

ReedCopsey commented Nov 3, 2016

I could see a use in using a record as an input parameter to a type provider, where the record only had primitive types. At compile time the record could be erased and the contents were available as input parameters as normal. But thats mainly as a means for efficiently inputting data inot a type provider.

That would require the input to be instances of a type, not just the type, right? That seems even more difficult...

Im struggling to invent many actual real world uses for this feature.

The main one I've tried are doing in the past was auto-implementation of INPC (I had a type provider that did this using strings to refer to the type in a separate assembly).

However, I could see this being very useful for generating proxy types to work around existing types (think wrappers to automatically null->to->option an API or similar), and things along those lines.

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Nov 3, 2016

Im struggling to invent many actual real world uses for this feature.

I mostly want this for compile time code generation. That is, fast lenses and serialization (no runtime reflection). This is particularly important now with how limited Core CLR reflection and code gen is. Generics support is only a nice to have, not all that important out of the gate for me.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 3, 2016

How does having a type as input make it easier to generate a lens, and what would such a generated lens look like?

What about extrapolating json or xsd schema from a record then using that as input to a type provider?

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 3, 2016

Im just worried whats been discussed doesn't amount to any real use-case, just an interesting extension to type providers.

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Nov 3, 2016

How does having a type as input make it easier to generate a lens, and what would such a generated lens look like?

The great thing about a language feature like this is we don't need to decide exactly what they would look like now, it can be iteratively improved.

What about extrapolating json or xsd schema from a record then using that as input to a type provider?

I don't really understand what you are suggesting here. You can't pass a runtime string into a type provider and keeping all of your records as xml schemas is pretty nuts.

Im just worried whats been discussed doesn't amount to any real use-case, just an interesting extension to type providers.

One area where F# is extremely lacking compared to Haskell or OCaml is compile time code generation. It seems natural to me to extend type providers to fill this gap rather than having a whole new mechanism which would require new and separate tooling.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 3, 2016

By changing the activation context of the type provider mechanism a single type could be provided to the type provider as input. I implemented some of that as a breaking change to the compiler but never progressed it far enough to know if it had merit.

Im still not convinced this feature warrants the increased complexity and effort required.

@cloudRoutine

This comment has been minimized.

cloudRoutine commented Nov 4, 2016

(if the semantics are wrong, just imagine the closest thing that would be right 😜 )

CmdletProvider<...>

The first example I can think of involves Argu and posh (powershell). This is a basic implementation of CLI args as a DU with Argu that gets you an argument parser with relative ease.

// CLI argument DUs

type AddReferenceArgs =
    | [<CLIAlt "-n">] Name of string
    | [<CLIAlt "-p">] Project of string

    interface IArgParserTemplate with
        member this.Usage =
            match this with
            | Name _-> "Reference name"
            | Project _ -> "Project to which reference will be added"

type AddProjectArgs =
    | [<CLIAlt "-n">] Name of string
    | [<CLIAlt "-p">] Project of string

    interface IArgParserTemplate with
        member this.Usage =
            match this with
            | Name _-> "Project reference path"
            | Project _ -> "Project to which reference will be added"

// command handlers
let addReference cont (results : ParseResults<AddReferenceArgs>) =
    maybe {
        let! name = results.TryGetResult <@ AddReferenceArgs.Name @>
        let! project = results.TryGetResult <@ AddReferenceArgs.Project @>
        Furnace.loadFsProject project
        |> Furnace.addReference (name, None, None, None, None, None)
        |> ignore
        return cont
    }

let addProject cont (results : ParseResults<AddProjectArgs>) =
    maybe {
        let! path = results.TryGetResult <@ AddProjectArgs.Name @>
        let! project = results.TryGetResult <@ AddProjectArgs.Project @>
        let name = Path.GetFileName path
        let newProject = Furnace.loadFsProject path
        Furnace.loadFsProject project
        |> Furnace.addProjectReference(path, Some name, None, newProject.ProjectData.Settings.ProjectGuid.Data, None)
        |> ignore
        return cont
    }

These CLI arg types and the handlers that process the argument values could be fed into a CmdletProvider<...> in order to generate a new type that will compile to a posh Cmdlet. These types have a set of conventions for their required attributes and properties that can be reasonably mapped onto from the Argu DU cases. Then the internal logic for the Cmdlet type's methods could be implemented using the Argu argument handler functions to create a posh Cmdlet with identical functionality to the Argu CLI parser.

This is what an F# type that compiles into a Cmdlet looks like :

[<Cmdlet("Search", "File")>]
type SearchFileCmdlet() =
    inherit PSCmdlet()
    /// Regex pattern used in the search.
    [<Parameter(Mandatory = true, Position = 0)>]
    [<ValidateNotNullOrEmpty>]
    member val Pattern : string = null with get, set    
    /// Array of filename wildcards.
    [<Parameter(Position = 1)>]
    [<ValidateNotNull>]
    member val Include = [|"*"|] with get,set    
    /// Whether or not to recurse from the current directory.
    [<Parameter>]
    member val Recurse : SwitchParameter = SwitchParameter(false) with get, set
    /// Encoding to use when reading the files.
    [<Parameter>]
    member val Encoding = Encoding.ASCII with get, set   
    /// Toggle for case-sensitive search.
    [<Parameter>]
    member val CaseSensitive : SwitchParameter = SwitchParameter(false) with get, set
    /// Do not use regex, just do a verbatim string search.
    [<Parameter>]
    member val SimpleMatch : SwitchParameter = SwitchParameter(false) with get, set
...

With this kind of TypeProvider anyone who implemented a console CLI parser could get the equivalent posh Cmdlet for free

ArgParserProvider<...>

To push it even further a TypeProvider could be implemented to generate the Argu DUs that implement ArgParser in the first place

Instead of

type NewCommand =
    | [<First>][<CLIAlt "project">] Project
    | [<First>][<CLIAlt("file","-f","fl")>] File
    | Nada
    interface IArgParserTemplate with
        member this.Usage =
            match this with
            | Project -> "Creates new project"
            | File -> "Creates new file"
            | Nada -> "doesn’t do much"

An ArgParserProvider<...> could consume a list of records and use the cases to generate the DU with appropriate attributes and the corresponding interface implementation

// library implementations

type ArgData = {
    Case  : string
    Alts  : string
    Attrs : string list
    Msg  : string
    Handler : string -> unit
} 

let argData case alts attrs msg hdl =
    { Case = case; Alts= alts; Attrs=attrs, Msg = msg; Handler = hdl }

// user code

type NewCommand = 
    ArgParserProvider< 
        [   argData  "Project" ["project"] ["First"]  "Creates new project" projFn 
        ;   argData  "File"    ["project"] ["First"]  "Creates new file"    fileFn
        ;   argData  "Nada"    []          []         "doesn’t do much"     nadaFn
        ]
    >

And then if this generated type is fed into the CmdletProvider<...> the user of this library gets
a lot of useful functionality from a small amount of code.

UnionUtilityProvider<...>

A TypeProvider used to enhance DUs that takes a type that satisfies FSharpType.IsUnion(...)
as its first parameter and an enum representing a standard set of boilerplate type
methods that can be added onto the union

type UnionUtility =
    | ToString = 0
    | Parse = 1
    | ParseCaseInsensitive = 2
    | TryParse = 3
    | TryParseCaseInsensitive = 4


type MyDU = UnionUtilityProvider< typeof<MyDUBase>, ToString|||Parse|||TryParse >

Another variant of this kind of provider could take an enum and transform it into a DU where it has an additional transformation method (e.g. FromInt ) that matches against the backing value type of its cases.

StaticMemberProvider<...>

A TypeProvider that takes a type and generates corresponding curried
static methods for each instance method without any overloads that passes an arg of its type as the final method arg for easy pipelining

BehaviorProvider<...>

Working with WPF entails a lot of boilerplate -

type BlockShift() as self =
    inherit Behavior<ScrollBar>()

    let subscriptions = ResizeArray<IDisposable>()
    static let depReg<'a> name = DependencyProperty.Register(name,typeof<'a>,typeof<BlockShift>)

    [<DefaultValue(false)>] 
    static val mutable private SizeSmallProperty  : DependencyProperty

    [<DefaultValue(false)>]                        
    static val mutable private SizeBigProperty    : DependencyProperty

    [<DefaultValue(false)>]                        
    static val mutable private AttachedToProperty : DependencyProperty

    [<DefaultValue(false)>]                       
    static val mutable private BlockZoneProperty  : DependencyProperty

    static do 
        BlockShift.SizeSmallProperty    <- depReg<float>            "SizeSmall"   
        BlockShift.SizeBigProperty      <- depReg<float>            "SizeBig"     
        BlockShift.AttachedToProperty   <- depReg<FrameworkElement> "AttachedTo"  
        BlockShift.BlockZoneProperty    <- depReg<float>            "BlockZone"   

    member __.DisposeOnDetach disposable = subscriptions.Add disposable

    member __.SizeSmall 
        with get() = self.GetValue BlockShift.SizeSmallProperty :?> float
        and  set (v:float) = self.SetValue ( BlockShift.SizeSmallProperty, v )

    member __.SizeBig 
        with get() = self.GetValue BlockShift.SizeBigProperty :?> float
        and  set (v:float) = self.SetValue ( BlockShift.SizeBigProperty, v )

    member __.BlockZone 
        with get() = self.GetValue BlockShift.BlockZoneProperty :?> float
        and  set (v:float) = self.SetValue ( BlockShift.BlockZoneProperty, v )

    member __.AttachedTo
        with get() = self.GetValue BlockShift.AttachedToProperty :?> FrameworkElement
        and set (v:FrameworkElement) = self.SetValue ( BlockShift.AttachedToProperty, v ) 

There's barely any distinct content in this type definition, we could probably cut this down to
something as small as -

type BlockShift = 
    BehaviorProvider<
        [   depReg<float>            "SizeSmall"
        ;   depReg<float>            "SizeBig"  
        ;   depReg<FrameworkElement> "AttachedTo
        ;   depReg<float>            "BlockZone"
        ]
    >

ViewModelProvider<...>

Defining a ViewModels type also involves far too much boilerplate

type QueryBoxConfig = {
    TextColor           : Color
    BlockColor          : Color
    ScrollVisibility    : ScrollBarVisibility
    HighlightColor      : SolidColorBrush
    ResultWidth         : float
    ScrollOpacity       : float
    BlockWidth          : float
}

type QueryBoxViewModel<'DataType>(config:QueryBoxConfig, filterfn:string->'DataType->'DataType) =
    inherit ViewModelBase()
    let ( =+= ) arg1 arg2 = self.Factory.Backing(arg1,arg2)

    // I've defined plenty of viewmodel types with 3X this number of properties
    let textColor        =  <@ self.TextColor         @>  =+=  config.TextColor
    let blockColor       =  <@ self.BlockColor        @>  =+=  config.BlockColor
    let scrollVisible    =  <@ self.ScrollVisibility  @>  =+=  config.ScrollVisibility
    let highlightColor   =  <@ self.HighlightColor    @>  =+=  config.HighlightColor
    let resultWidth      =  <@ self.ResultWidth       @>  =+=  config.ResultWidth
    let scrollOpacity    =  <@ self.ScrollOpacity     @>  =+=  config.Opacity
    let blockWidth       =  <@ self.BlockWidth        @>  =+=  config.BlockWidth

    member __.TextColor         
        with get() = textColor.Value and set v = textColor.Value <- v
    member __.BlockColor        
        with get() = blockColor.Value and set v = blockColor.Value <- v
    member __.ScrollVisibility 
        with get() = scrollVisible.Value and set v = scrollVisible.Value <- v
    member __.HighlightColor
        with get() = highlightColor.Value and  set v = highlightColor.Value <- v
    member __.ResultWidth
        with get() = resultWidth.Value and  set v = resultWidth.Value <- v
    member __.ScrollOpacity 
        with get() = scrollOpacity.Value and set v  = scrollOpacity.Value <- v
    member __.BlockWidth
        with get() = blockWidth.Value and set v  = blockWidth.Value <- v

// with a concrete form to back a specific GUI control 

type CommandPaletteControlVM() =
    inherit QueryListBoxViewModel<CommandData>(paletteConfig, filterPalettefn)

Unfortunately a Concrete viewmodel type is needed because parametrically polymorphic types
do not work well with all of WPF and the WPF editor's design time features.

This could probably be cut down to

type CommandPaletteControlVM = 
    ViewModelProvider< 
        {   TextColor        = Brushes.LightGray.Color
            BlockColor       = Brushes.DarkCyan.Color
            ScrollVisibility = ScrollBarVisibility.Hidden   
            HighlightColor   = Brushes.Black
            ResultWidth      = 4.5
            ScrollOpacity    = 0.40
            BlockWidth       = 5.0
        } : QueryBoxConfig
        ,   typeof<CommandData  
        ,   filterPalettefn
    >

PhantomProvider<...>

I'm not sure this can work, or what the proper syntax would be, but the idea
is to generate a wrapper type that stores a primitive value type that can't work with
units of measure (e.g. char [],string,Vector3) with a tag/dummy generic parameter.
The innovation here would be to pull all of the instance methods from the stored data type
to the wrapper type. Methods that would have returned the internal type instead return
the wrapped type with the phantom tag.

type Message<'Tag> = PhantomProvider<string> 

type DocumentBlock<'Tag> = PhantomProvider<char []>

type Letter = class end 
type Email = class end 
type SmokeSignal = class end 

type Header = class end 
type Footer = class end 

let fn1 (block:DocumentBlock<Header>) : DocumentBlock<Header> = 
    block.GetLowerBound() |> ...

 let fn2 (block:DocumentBlock<Footer>) : DocumentBlock<Footer> =
    block.[0..5] |> ...

let fn3 (dispatch:Message<SmokeSignal>) : Message<Email> =
    dispatch.Split [|';' ; '|' |]...    

Further Potential (the crazy stuff)

Perhaps we could define TypeProviders that require the input type to implement an interface or
to satisfy a set of statically resolved type parameters that are used as the building blocks for the
implementation of the methods for a static type (akin to the Collections API for Seq,Array,List).
Although this seems to be treading close to (over?) the purview of TypeClasses, so it may not be the
best usage for a TypeProvider.

It might also be possible to implement a TypeProvider similar to Nitra

Nitra generates a set of AST classes based on the syntax rules in your grammar. Inside these classes, it generates properties
named after the components of each syntax rule. For example, Nitra will generate a CompilationUnit class, with a TopLevelDeclarations
property. If you like, you can also set explicit names for AST properties - we’ll see an example of that soon.

The ASTProvider<...> takes a set of SyntaxRule types to generate a lexer, a parser & the AST types. These types can get fed into a WorkspaceProvider<...> that creates a Roslyn workspace around the generated compiler, the workspace is fed into the LanguageServerProvider<...> which creates a language server type that implements the Language Serer Protocol or maybe a set of types that implement the Visual Studio Extension point classes so your mini language gets VS editor tooling for free.

@vasily-kirichenko

This comment has been minimized.

vasily-kirichenko commented Nov 4, 2016

I think all the use cases above can be divided into two categories:

  • using a type as a pure template for generating other types. The type is not used anywhere else.
  • passing a real type in order to generate additional stuff, like other types or (extension?) static members.

The first category isn't worth to implement as you can do exactly the same stuff passing, say, JSON or XML or some sort of micro DSL to a TP, which may be more readable, than a F# record. As it's compile time checked (it is), I see little difference between this and passing a one-shot-use record.

What do you guys think about implementing fsharp/fslang-suggestions#450 instead? However, it would result with ugly code like

type TP = MyTP<<@ typeof<T> @>>

I.e. you cannot pass other type easily, like MyTP<@ T @>.

Honesty, all these attempts to use TPs in cases where macros should be used result with restricted solutions and clunky client code. Considering how much time it would certainly take to implement, I'm not sure anymore it's worth it.

Real macros should have access to everything available on a certain compilation stage, like current method, type, module, namespace, project. And it should be able to amend some parts of the typed AST, like adding/removing members, types, attributes and the like.

I think the best thing we can do to add real metaprogramming into F# is learning how macros are implemented in, say, Nemerle, refine the idea, then make a plan to create something similar in F#. This is certainly deserve a lot of time and resources to implement.

(somewhat) better TPs? I'm not sure.

@cloudRoutine

This comment has been minimized.

cloudRoutine commented Nov 4, 2016

@vasily-kirichenko

type UC = {a:int; b:string}
let QT<'a> = <@ typeof<'a> @>
type PT = MyTP<QT<UC>>
@vasily-kirichenko

This comment has been minimized.

vasily-kirichenko commented Nov 4, 2016

@cloudRoutine compare it this Nemerle style:

[<MyTP>]
type UC = { a: int; b: string }

which would amend UC type itself, no need for PT at all.

@kurtschelfthout

This comment has been minimized.

Member

kurtschelfthout commented Nov 4, 2016

I think all the use cases above can be divided into two categories:

using a type as a pure template for generating other types. The type is not used anywhere else.
passing a real type in order to generate additional stuff, like other types or (extension?) static members.

That's a misleading characterization I think. The type is used as input to generate code at compile time that can use the type as if you'd write the code yourself (e.g. create instances of it, in the case of deserialization). So the type is definitely used somewhere else. The RFC has some examples of this, e.g. for defining messages.

Yes you can technically define a type as an xml snippet right now but that's a race to the bottom: after all, you can do pretty much everything TPs and macros do, by code or IL generation too. So why bother with that?

Not that I am opposed to some form of macro support but if we can get similar possibilities - code generation based on real code - using the existing TP machinery I think that is preferable. In any case you'd hit pretty much the same problems as are being discussed here with macros because they'll interact with TPs too.

I see this RFC as a first step to build more support on later. Even if it only sees real types (i.e. that the programmer typed in) and can only generate erased types from those I would find it useful enough to be included (as long as restrictions don't box us in for later).

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Nov 4, 2016

It seems to me that there's little value in adding some kind of constraints system on types for type providers as you can do arbitrary checks and error out in the type provider itself and this happens at compile time.

Also, the attribute approach is nice looking but much more limited in terms of what you can do as you can only ever put them on one type/member/property. Potentially with the type provider approach you could pass in multiple types.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 4, 2016

Im still undecided, I would like to see more examples of what this allows. Any thoughts @dsyme ? As you approved this you must have some thoughts on its direction and implications.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 4, 2016

I wonder if intrinsic type extensions along with this allows even more options?

@chkn

This comment has been minimized.

chkn commented Nov 21, 2016

I wonder if intrinsic type extensions along with this allows even more options?

Intrinsic type extensions + this RFC is what I was imagining for GUI designer codebehind (where partial classes are generally used in C#), e.g.:

type MyViewController = designable<UIViewController> with
  override this.ViewDidLoad() = this.MyLabel.Text <- "Hello World!"

This would yield a generative type equivalent to:

[<Register("MyViewController")>]
type MyViewController(handle : IntPtr) =
  inherit UIViewController(handle)

  // Some outlet declared in the designer
  [<Outlet("MyLabel")>]
  member val MyLabel = Unchecked.defaultof<UILabel> with get, set

  override this.ViewDidLoad() = this.MyLabel.Text <- "Hello World!"

In this case, the type argument specifies the base class to use for the generated type.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 22, 2016

Im thinking the kickstarter Ive been mentioning would cover this if was funded sufficiently.

@robkuz

This comment has been minimized.

robkuz commented Nov 22, 2016

My absolute favorite use case would be creating a Lens TP.

type Foo = { a: int; b: string option }
type FooLenses = LensProvider<typeof<Foo>>

so that

let fl = new FooLenses()
let foo = { a = 1; b = Some "foo" }

let aval = fl.a.get foo
// aval = 1

let foo' = fl.b.set "bar" foo
// foo' = { a = 1; b = Some "bar" }

Also this should be possible for generic types. so that

type Bar<'a, 'b> = { a: 'a; b: 'b }
type BarLenses = LensProvider<typeof<Bar<_,_>>>
@robkuz

This comment has been minimized.

robkuz commented Nov 22, 2016

Another scenario I have in mind is some emulation of Higher Kinded Types
For example when doing type driven development. Lets assume a type that is used for
"working" (any kind of changes are allowed and that does some kind of tracking of changes)
and a sibling type that does not allow for any changes

type ChangedItem<'a> = { value: 'a; changedAt: DateTime }
type WorkItem<'a> =  {actual: ChangedItem<'a>; prevValues: ChangedItem<'a> list option}

type ClosedItem<'a> =  {value: 'a; prevValues: ChangedItem<'a> list}

type WorkingFoo = {
  a: WorkItem<String>;
  b: WorkItem<int>
}

type ClosedFoo = ClosingTypeProvider<typeof<WorkingFoo>>

Where ClosedFoo would look like this

type ClosedFoo = private {
  a: ClosedItem<String>;
  b: ClosedItem<int>
}
with
   member a = this.a
   member b = this.b
@7sharp9

This comment has been minimized.

Member

7sharp9 commented Nov 22, 2016

@robkuz

This comment has been minimized.

robkuz commented Nov 22, 2016

Well, as long as one knows how a record is implemented with c#/.net there is probably nothing one couldnt do ;-)

@dsyme

This comment has been minimized.

Contributor

dsyme commented Apr 12, 2017

@dsyme Is the current cut erasing only?

I don't know of any specific reason why generative wouldn't work

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Apr 12, 2017

@dsyme

This comment has been minimized.

Contributor

dsyme commented Apr 12, 2017

I would try but vfsharp is currently broken on osx due to the resources

Let's get that fixed. Could you try shifting VFT to the latest FSharp.Compiler.Tools?

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Apr 13, 2017

@dsyme Could we get the osx fix cherry picked so this is easily testable?

The main pain points are I also have to port this to FCS to get tooling support.

@smoothdeveloper

This comment has been minimized.

Contributor

smoothdeveloper commented Apr 13, 2017

It would be greatly useful to have a compilation flag in visualfsharp repository which compiles "a source compatible with FCS" assembly roughly compatible with FCS, it means workflow in visualfsharp would allow to test/integrate in tooling outside of Visual Studio.

@vasily-kirichenko

This comment has been minimized.

vasily-kirichenko commented Apr 13, 2017

@smoothdeveloper afaik the only thing that's needed is conditionally make the symbols api public. That's it.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Apr 13, 2017

Why does it ned to be internal?
I really think FCS should be retired asap and we work out how to make vfs work.

@vasily-kirichenko

This comment has been minimized.

vasily-kirichenko commented Apr 13, 2017

vfs? About internals, it's a MS requirement. No idea what they hope to achieve with this.

@smoothdeveloper

This comment has been minimized.

Contributor

smoothdeveloper commented Apr 13, 2017

No idea what they hope to achieve with this.

Annoying Developers, Developers, Developers!

@zpodlovics

This comment has been minimized.

zpodlovics commented Apr 14, 2017

Please forgive me for the late question: is it possible to generate value types (plain structs with different programmable layout options (eg.: layout for interop, cache line alignment, etc), struct tuples, struct du, struct records, etc) with this type provider ?

High performance serialization and messaging is not possible without proper value types. The motivation example could spent significant amount of time in GC:

type Stock = { Ticker: string; Price: float; Date: DateTimeOffset }
type StockReader = CsvProvider<Stock>
let stocks : Stock seq = StockReader.Read("path/to/file.csv")

A value type based streaming could be an order of magnitude faster - at least this is what I experienced when I implemented a value type based record streaming for reading 100M+ rows from database...

@dsyme

This comment has been minimized.

Contributor

dsyme commented Apr 14, 2017

Please forgive me for the late question: is it possible to generate value types (plain structs with different programmable layout options (eg.: layout for interop, cache line alignment, etc), struct tuples, struct du, struct records, etc) with this type provider ?

Yes, it would be possible to build such a type provider

[<Struct>]
type Stock = { Ticker: string; Price: float; Date: DateTimeOffset }
type StockReader = CsvProvider<Stock>
let stocks : seq<Stock> = StockReader.Read("path/to/file.csv")

though the use of seq might also have to change to be maximally performant

@Thorium

This comment has been minimized.

Thorium commented Jun 6, 2017

No workarounds even?
I always thought that this is already possible, at least with a property or a method. Am I missing something here?

There is a baseType in ProvidedTypes.fs:

type ProvidedTypeDefinition(container:TypeContainer, className : string, baseType  : Type option, convToTgt) =

...and method can return a type, so I name my method as "GetBaseItem":

ProvidedMethod("GetBaseItem",[],typeof<MyRecord>,...)

or dynamically

let returnType = 
    let rt = ProvidedTypeDefinition("MyGeneratedType",Some typeof<MyRecord>, HideObjectMethods = true)
    //...
    rt :> Type
ProvidedMethod("GetBaseItem",[],returnType,...)
@colinbull

This comment has been minimized.

colinbull commented Jun 6, 2017

@Thorium the problem is 'MyRecord' in this case would have to be provided in an external assembly which is loaded via reflection, or Compiled with the TP.

@Rickasaurus

This comment has been minimized.

Rickasaurus commented Aug 17, 2017

I hope this doesn't die out. It's by far the most exciting thing happening in F# land these days.

@colinbull

This comment has been minimized.

colinbull commented Aug 17, 2017

Latest is here, https://github.com/colinbull/visualfsharp/tree/rfc/fs-1023-type-providers not worked on it in about 2 months, since I have been on a large time consuming project at real work :(. But hoping to get back to it soon.

@dsyme

This comment has been minimized.

Contributor

dsyme commented Feb 8, 2018

I wrote up a sent of notes for Toby Shaw on @colinbull's prototype before Christmas, I'm pasting it here for safe keeping:

  • The approach used is basically to implement the System.Type programming model with the TAST as the backing data (TastReflect.fs).

  • The systematic testing approach can be seen here where we basically check through equivalences between a provided thing and the results from typeof+reflection.

  • Latest TPSDK. There is a substantial chunk of work in upgrading the samples and testing in the PoC to use the latest TPSDK. Using the latest TPSDK in some basic “visualfsharp” TP tests is a good thing we should do in any case and could perhaps be a separate PR. The TPSDK has its own testing too.

  • Type translation. The latest TPDK adds a translation process from “design time” types to “target types” (which you’ve likely seen the PRs for.) In this setting, type provider authors implement w.r.t. “design-time” types. It’s important to note that TastReflect.fs implements “target types”. In theory these should be translated back to “design-time” types before being presented to the type provider author. Strictly speaking this should be done in the TPSDK as the (target) TastReflect Type objects flow into the SDK when the compiler calls ApplyStaticArguments. You might get away with not doing this for a while but eventually it will be needed.

  • Implementing symbols and infos. TastReflect.fs has an implementation of System.Type/MethodInfo etc closely related to the implementations in the TPSDK (ProvidedTypeSymbol, ProvidedTypeDefinition, TypeSymbol, TargetTypeDefinition and friends).

In the long term it’s possible that the TastReflect implementation could replace both TypeSymbol and TargetTypeDefinition. However it’s best to keep them separate for now

In the short term, the implementations in the TPSDK are more complete and “better” than the ones – Colin copied/adjusted the early TPSDK implementations but I found a whole host of problems with them after that, and largely rewrote them. I’d recommend merging the “best bits” of TypeSymbol/TargetTypeDefinition with the “best bits” of TastReflect.fs.

  • Making it useful There is a substantial chunk of work in “making sure this is useful” by pushing through real-world applications. We expect this will throw up a range of issues and possibly some necessary incompletenesses.

  • It’s tricky work 😊 I understand some parts of TPs can be mind-bending and the TPSDK implementation is not easy. I expect we may need to talk through technical issues from time to time, and I’d be available for that.

@dsyme

This comment has been minimized.

Contributor

dsyme commented Feb 8, 2018

@colinbull added this:


As stands currently stands the following things have a implementation (some with far rougher edges than others)

• Support for custom attributes.
• Partial support for FSharp reflection (mainly for records at the minute). This should be extended so as much of the API is sensible returns the correct result, this is essentially what the tests are checking.
• Support for generic types (although the resolution in TastReflect.Assembly is clunky and doesn't work correctly for Multiple type parameters also breaks when there is Assembly Qualified type parameters. I have a better type name parser but it isn't quiet finished and is what I'm hoping to finish up tonight)
• Unions (although not tested generic unions)

Off the top of my head the following things are missing or I have no idea about the current behavior

• Support of events?
• No testing has been done on whether types from external assemblies work.
• Anonymous records?
• Generic Methods - I think these should be implemented in TastReflect but I have never tried excising them.
• Class Types - not tested may just work :)
• Expression translation probably needs some additional patterns to support the new types of calls that might be needed in any type providers that use this feature. I have already done this for field get as an example to support the Serializing TP in the example.
• Provided types from other TP's? Not sure about this one, but it I have a feeling it would make TP Combinators somewhat nicer.

@TobyShaw

This comment has been minimized.

TobyShaw commented Mar 30, 2018

Thought I'd leave a comment here on a few things I've done on this.

I spent a small amount of time refactoring the test suite for this change, it should be easier to extend with more tests now: here

In terms of notable changes to the actual code:

I updated it to use the latest TPSDK, (types passed in basically ignore the target/design-time conversion process, as they always live in the target).

To get FSharp reflection working, I added the CompilationMappingAttribute to the reflected types: here

I'd hope that this would allow the use of TypeShape in type providers, but this is not the case. TypeShape is incompatible with these reflected types, due to its design. To get around this, one could use a type provider purely as a validation step before passing to a runtime-typeshape library. This would give the benefits of type-checking the type parameter, but without the code generation aspect.

I also fixed a lot of small things, like Namespace was incorrect, and the fully qualified names were also subtly wrong.

The attributeflags will need a bit of work to get exactly right. Currently I've got most flags set to sensible defaults, but eventually they will need to be set properly: here

I know that in order to get the IsNested function working, I need to synthesize an enclosing type corresponding to the Module, and report this as a top-level type's DeclaringType (IsNested is implemented in terms of DeclaringType).

On another thread, I also had a go at implementing passing of type variables. Currently it only works with bound type variables passed into provided methods, but there's a few things you could still theoretically do with it so far: here

It's not that much, and I'd hoped to have more done at this point. On the flipside, exams are finally over, and I spent a lot of time understanding various aspects of the compiler and authoring providers. I hope to make lots more progress in the next month.

@TobyShaw

This comment has been minimized.

TobyShaw commented Apr 2, 2018

So, with the option to pass in types, there comes a few questions about how exactly they get passed in.
When a type variable is supplied to a type provider, as in, MyTP<'a>, there's two options for how this could be received on the TP end.

  1. The type provider could only be activated when the variable is instantiated to a concrete type.
  2. The type provider is activated immediately, passing in the type variable 'a

I think both have their usecases, and I think we should be able to do both, depending on whether the type variable exists in the scope of a type-level function or a value-level function.

For 1. it would look like:
type MyTypeFunction<'a> = MyTP<MyOtherTP<'a>>
One would be able to call MyTypeFunction<int> and expect a type back which is specialised to int. These type functions would solve the problem of type provider composition.
To make disambiguation easier, could also use a different keyword, like:
provider MyTypeFunction<'a> = MyTP<MyOtherTP<'a>>, but I'm not sure if F# has support for context-sensitive keywords, otherwise it'd be a breaking change.

For 2. it would look like:
let myValueIdentityFunction (x : 'a) = Identity<'a>.Invoke x
Calling this function myValueIdentityFunction 1, one would know that the int returned has not been modified, as Identity only received a transparent reference to a type variable.

What do you think of this idea?

@jrmoreno1

This comment has been minimized.

jrmoreno1 commented Apr 29, 2018

@colinbull : does it support exposing private methods/properties of a class?

@colinbull

This comment has been minimized.

colinbull commented Apr 29, 2018

@TobyShaw

This comment has been minimized.

TobyShaw commented Apr 29, 2018

Just confirming that what Colin said is accurate.

@jrmoreno1

This comment has been minimized.

jrmoreno1 commented Apr 29, 2018

@colinbull, @TobyShaw : I was thinking of introducing/using F# for unit testing, being able to do so with some kind of type safety for private methods would be great. So, I'd naturally want them to be invokable.

@TobyShaw

This comment has been minimized.

TobyShaw commented Apr 30, 2018

It depends what you mean by invokable.

If you mean: The type provider is able to invoke the method at compile time, and inspect the result.

Then this will not be possible.

If instead you mean: The type provider is able to provide a method which invokes a private method, despite being in a context where that method is not visible.

Then this is another story, I think the answer is yes. I haven't really touched this aspect of the PR, so @colinbull could give a more convincing answer.

@danyx23

This comment has been minimized.

danyx23 commented Jun 6, 2018

I am interested in this feature because I think it could be useful as a way of metaprogramming somewhat similar to what Haskell has with generics via generic deriving. To give a concrete motivating example that I am currently facing in real world F#:

I have a code base of a few thousand lines of F# code that used to be a Suave only backend application. After some back and forth, we arrived at FSharp.Json as a nice way to deal with json serialization that works via reflection and allows customization of the serialization process via Attributes. This worked nicely and required no additional code to actually serialize/deserialize all the fields in all the records/DUs.

We are now moving some parts of this application to the SAFE stack and want to share the many type definitions between server and client (F# transpiled via Fable to JS). We are now switching to the Thoth library for Json serialization as it exists for both F# on the .NET runtime and as a Fable library and allows us to control serialization in the same way on both runtimes. However, this also means writing by hand a nontrivial amount of Json Encoders/Decoders. Reflection is of course not an option as it does not exist in full form in Fable atm. A code generator is being worked on by the Thoth author but a metaprogramming facility a la Haskells derive generic would be a nicer solution IMHO as it could not come out of sync with the type definitions.

Am I right to hope that an F# type provider capable of generating types from types could be built if this feature were to land that would automate the creation of the Encoders/Decoders for the various types?

To express it in code, could something like this:

type Api =
  {
    Title: string
    Version: string
    Description: string
    RecordName: string option
    PluralRecordName: string option
  }
  static member Decoder token =
    result {
      let! title = Decode.field "title" Decode.string token
      let! version = Decode.field "version" Decode.string token
      let! description = Decode.field "description" Decode.string token
      let! recordName = Decode.option (Decode.field "recordName" Decode.string) token
      let! pluralRecordName = Decode.option (Decode.field "pluralRecordName" Decode.string) token

      return {
        Title = title
        Version = version
        Description = description
        RecordName = recordName
        PluralRecordName = pluralRecordName
      }
    }

  static member Encoder (api : Api) =
    Encode.object
      [ "title", Encode.string api.Title
        "version", Encode.string api.Version
        "description", Encode.string api.Description
        "recordName", Encode.option Encode.string api.RecordName
        "pluralRecordName", Encode.option Encode.string api.PluralRecordName
      ]

let decoded = Thoth.Json.Net.Decode.decodeString Api.Decoder jsonStringValue

be replaced with something like this?

type Api =
  {
    Title: string
    Version: string
    Description: string
    RecordName: string option
    PluralRecordName: string option
  }

type ApiJsonCoders = ThothTypeProvider<Api>()

let decoded = Thoth.Json.Net.Decode.decodeString ApiJsonCoders.Decoder jsonStringValue
@TobyShaw

This comment has been minimized.

TobyShaw commented Jun 6, 2018

This will definitely be enabled by this language feature.

@7sharp9

This comment has been minimized.

Member

7sharp9 commented Jul 17, 2018

I just wanted to reiterate again
#125 (comment) that this RFC deals with types as input to a type provider, it does not enable type providers to output F# constructs like unions and records. To do that an RFC would have to be written, and then implemented for this language suggestion:
fsharp/fslang-suggestions#154

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment