Support XSD in XmlProvider

[ovatsus]

No description provided.

[rojepp]

+1 This seems to be a more robust option than guessing types from a sample.

[JonnyBoats]

+1 I could really use this.

[forki]

Creating the types is easy once you inferred the schema. The problem here is we need something that can understand XSD files - and works on mono.

I think xsd.exe (http://msdn.microsoft.com/en-us/library/x6c1kb0s(v=vs.80).aspx) is probably not the way to go.

[panesofglass]

👍 Thanks, @tpetricek. I looked for this but couldn't find it.

[tpetricek]

It would be great to have this feature - I think this is something that can be done without messing too much with the details of the XML type provider, so it is a perfect project for a new contributor :-). I'm happy to help anyone who is interested in looking into this (@panesofglass ;-)).

The easiest way to support this would be to parse the XSD file and build a value of InferedType (see here) that represents the document. I think this should be fairly straightforward (but of course, XSD is a long W3C standard, so...)

The XML provider does not support all InferedType values that you may construct - you can explore how types of various XML documents look using the following script (it just calls the inference and prints the inferred type nicely):

#r "System.Xml.Linq.dll"
#r @"C:\Tomas\Projects\FSharp.Data\bin\FSharp.Data.DesignTime.dll"
open System.Xml.Linq
open ProviderImplementation
open FSharp.Data.RuntimeImplementation.StructuralTypes

fsi.AddPrinter(fun (t:System.Type) -> t.Name)

let doc = XDocument.Parse("""<root>
    <attrs id="1" />
    <text>hello</text>
    <tricky>3.14</tricky>
    <tricky>true</tricky>
  </root>""")
let culture = System.Globalization.CultureInfo.InvariantCulture

XmlInference.inferType culture false doc.Root

For the example, you get:

Record                  // An XML element is always represented as Record
  (Some "root",         // This is the name of the record
    [{Name = "";        // A list of attributes/children - the name "" is
      Optional = false; //    special and means the body of the element
      Type =            //    (optional means it may/may not be there)
      Collection        // Collection represents multiple children 
        (map
            [(Record (Some "attrs"), // This is InferedTypeTag - it should be              
              (Single,               // basically the name of the element
              Record (Some "attrs",[{Name = "id";  // Attribute 'id' of type 'int'
                                      Optional = false; // .. that's not optional
                                      Type = Primitive (Int32,null);}])));
            (Record (Some "text"),
              (Single,               // Single vs. Multiple - how many times 
                                     // can the child element appear?
              Record (Some "text",[{Name = "";  // Body is non-optional
                                    Optional = false; // .. string
                                    Type = Primitive (String,null);}])));
            (Record (Some "tricky"),
              (Multiple,
              Record
                (Some "tricky",
                  [{Name = "";
                    Optional = false;
                    Type =         // This represents type that is a choice
                    Heterogeneous  // between multiple possible options - 
                      (map         // here number of boolean
                          [(Number, Primitive (Decimal,null));
                          (Boolean, Primitive (Boolean,null))]);}])))]);}])```

[panesofglass]

This may be a good opportunity to finally learn how type providers work. I'm interested. That said, you can look around and see just how quick I am to finish things. :)

Are you thinking a custom XSD parser? What are the limitations of using other, external libraries? I forget.

[tpetricek]

If there are some external libraries that we can easily rely on, that's probably fine - I'm not sure how difficult would it be to just read XSD as a XML document and work with that...

I'll be certainly happy to help (and maybe I'll recruit more people :-)) but I'm probably not able to lead the effort.

[rojepp]

I'm not sure reading Xsd's as plain Xml documents is a good way forward. Xsd's are complicated beasts. I started experimenting with this using classes in System.Xml.Schema but I never really got anywhere. Lack of time/ran out of energy.. You should probably get some tricky Xsd's to try out with the TP to understand why it's not trivial. Often, you'll get some complex Xsd from a third party and it would be awesome to have a robust Xsd TP for them.

[chaliy]

.NET already have build in support for parsing XSD - XmlSchema. Main problem is to convert XmlSchema model into appropriate for typeprovider model. So for example logic that says that "xsd:element with xsd:sequence of xsd:element should result as DTO" is hardcoded in xsd.exe tool. This logic is internal and so need to be reimplemented. XmlSchemaElement already have some information like IsAbstract that could help.

[tpetricek]

Cool - I did not know about System.Xml.Schema - that certainly sounds like a better way to go.

[dfaroi]

Maybe following codeplex project could help :
Open Linq To XSD => http://openlinqtoxsd.codeplex.com
forked of Linq To XSD => http://linqtoxsd.codeplex.com

[ovatsus]

@runefs, seems you're working on a XSD type provider (http://stackoverflow.com/questions/20466880/getting-compile-error-on-provided-type). Would you like to integrate it in FSharp.Data?

[runefs]

I might be able to contribute. i'm currently working on a XSD type provider. It's currently based on System.Xml.Linq but 'd look into changing that to System.Xml.Schema

[panesofglass]

Transformations would be so nice. I mentioned XQuery to @mausch in the issues for his XmlLiteralsTypeProvider. XSLT would also be interesting, but I though XQuery might be a little simpler to implement in F#.

[pver]

I'm quite new to type providers (so maybe I'm totally wrong with my question) but could such an XSD provider be used instead of providing a sample XML to the XML type provider?
So could the XSD type provider be used to get the types from an XSD file, and could that type information be used (in a second step) as an input to the XML type provider to parse an XML file? Or will it just expose the types defined in the XSD and nothing more?

[ovatsus]

@pver that's the idea, instead of giving an xml example, provide a xsd that defines the structure of the xml, but still generate types for the xml described by the xsd, not for the xsd itself

[pver]

@ovatsus thanks for the confirmation, that sounds great :)

[ovatsus]

@runefs, @rojepp, @pezipink, all of you seemed interested in contributing this, has any of you made any progress on this? There's a lot of people asking for this, it would be great to be able to pass a xsd to the schema parameter of XmlProvider and make the types match correctly

[ghost]

I've had lots of people ask me too

[runefs]

I made progress and then did nothing about it. Being Willy and not
satisfied because the testing wasn't exhaustive. I've been able to parse
the schemes I had laying around. I'll try and merge what ever changes
there's been since I work on it and then create a pull request with what I
have if that's of interest

Mvh
Rune

Den 19/03/2014 kl. 19.42 skrev Gustavo Guerra notifications@github.com:

@runefs https://github.com/runefs, @rojepp
https://github.com/rojeppyou both seemed interested in contributing
this, has any of you made any
progress on this? There's a lot of people asking for this, it would be
great to be able to pass a xsd to the schema parameter of XmlProvider and
make the types match correctly

Reply to this email directly or view it on
GitHubhttps://github.com//issues/57#issuecomment-38083347
.

[runefs]

Ok ok I'll get to it :) if nothing else I have a very working prototype I
think

Mvh
Rune

Den 19/03/2014 kl. 20.53 skrev Don Syme notifications@github.com:

I've had lots of people ask me too

Reply to this email directly or view it on
GitHubhttps://github.com//issues/57#issuecomment-38091828
.

[ovatsus]

Great! Let me know if you need help merging, there's been quite a few internal changes to FSharp.Data in the meantime. Was the existing InferedType data model enough or did you have to extend it?

[JonnyBoats]

If you need help with testing I have some rather complicated data with schema (XSD files) that I could throw at it.

[runefs]

I'm gonna need some help on this one
I have a few simple question (in the newb category)
What's the process for testing. I essentially could figure our how to add
tests for the provider. (Which has been a drag when debugging too!)

And John please send me the XSDs (preferably with sample XML as well)

Br
Rune

2014-03-19 22:15 GMT+01:00 John Tarbox notifications@github.com:

If you need help with testing I have some rather complicated data with
schema (XSD files) that I could throw at it.

Reply to this email directly or view it on GitHubhttps://github.com//issues/57#issuecomment-38107800
.

[ovatsus]

There are several kinds of tests:

• You can test that type inference directly, feeding it an xsd and asserting on the output InferedType. Example: https://github.com/fsharp/FSharp.Data/blob/master/tests/FSharp.Data.DesignTime.Tests/InferenceTests.fs#L57 (currently there's no coverage for XmlInference there, only for JsonInference and CsvInference)
• You can test the output of the type provider, i.e, the types created and the code generated for each member. This is done in https://github.com/fsharp/FSharp.Data/blob/master/tests/FSharp.Data.DesignTime.Tests/SignatureTests.fs. You just need to add test cases to https://github.com/fsharp/FSharp.Data/blob/master/tests/FSharp.Data.DesignTime.Tests/SignatureTestCases.config and the tests will compare the output to what's in https://github.com/fsharp/FSharp.Data/tree/master/tests/FSharp.Data.DesignTime.Tests/expected. You can see how those lines are parsed here: https://github.com/fsharp/FSharp.Data/blob/master/src/TypeProviderInstantiation.fs#L160. Most of the times my main tests are regenerating the expected files and doing a diff with what's checked in to compare the impact of the changes. You can do that by Alt+Entering the whole of SignatureTests.fs, and then calling generateAllExpected in FSI. Alternatively, if you want to debug, you can do it all without having to use the test solution. Rename https://github.com/fsharp/FSharp.Data/blob/master/src/Test.fsx to Test.fs, change the type of the FSharp.Data.DesignTime project to Console Application and run. The test file currently tries to generate one example of each kind of provider in PCL profile 47 (https://github.com/fsharp/FSharp.Data/blob/master/src/Test.fsx#L41) and then generates all test cases in .NET 4.0 (https://github.com/fsharp/FSharp.Data/blob/master/src/Test.fsx#L71). By running you will both print the output and overwrite the expected files.
• Finally, you can test the runtime behaviour (https://github.com/fsharp/FSharp.Data/blob/master/tests/FSharp.Data.Tests/XmlProvider.fs). This will exercise that whatever the XmlRuntime.XXX calls that are shown in the signature tests actually work as expected at runtime. We should actually also have a couple of tests for those runtime methods individually, but we're only doing that in a handful of cases

A bit of this is on http://fsharp.github.io/FSharp.Data/contributing.html, but not much. It would be great if you could improve it based on your experience. Let me know if you need more help. You can also push to your repo and I can have a look if you want

[pezipink]

Just to add to this - I don't think it matters if your XSD provider does not work perfectly (or at all) with John's complex schema(s). What's important is a base we can work from that covers the very fundamental blocks of XSD, and we can then work on it together from there.

[ovatsus]

no problem

[runefs]

So I got the code working with compiled schema sets and also with what I
find to be much cleaner code. However I have an issue in the travis build
that I would very much appreciate a little insight into. Being pre-newb on
travis and having very little experience with anything ending in .sh. The
error on travis is very much consistent with an error that I previously had
in the code which is now fixed and works locally and on AppV. So either
travis is using an older version (from the previous pull request) which
does semm that likely or I would like suggestion to why the build on
travis could result in a stack overflow when that's not the case locally or
on AppV. ANy information/suggestions would be highly appreciated

2015-08-11 15:56 GMT+02:00 Gustavo Guerra notifications@github.com:

no problem

—
Reply to this email directly or view it on GitHub
#57 (comment).

[giacomociti]

Glad to see @runefs progress, so I better quit experimenting on my own.
I may help a bit in refining and testing the code once it get merged.
Unfortunately I cannot really help with trevis and build related stuff in general.

[runefs]

@giacomociti I don't think it as such is caused by the build. It seems to
be either Mono or *nix that causes it to work differently. I'm trying to
create a VM where I can recreate the problem so that I can debug it

2015-08-27 22:02 GMT+02:00 Giacomo Citi notifications@github.com:

Glad to see @runefs https://github.com/runefs progress, so I better
quit experimenting on my own.
I may help a bit in refining and testing the code once it get merged.
Unfortunately I cannot really help with trevis and build related stuff in
general.

—
Reply to this email directly or view it on GitHub
#57 (comment).

[ckpearson]

Hi, I'm working on a project at the moment that requires us to produce an XML document based on quite a strict and large schema and I instantly thought "F# type providers!". Did you manage to get this working at all?

[runefs]

Currently there's a cross compilation bug. System.Xml behaves differently
in the mono binding and MS binding.

2015-09-08 17:50 GMT+02:00 Clint Pearson notifications@github.com:

Hi, I'm working on a project at the moment that requires us to produce an
XML document based on quite a strict and large schema and I instantly
thought "F# type providers!". Did you manage to get this working at all?

—
Reply to this email directly or view it on GitHub
#57 (comment).

[ckpearson]

I see, is there a place I can grab it for targeting MS .NET? I'll only be running on Windows anyway.

[CumpsD]

@ckpearson this seems to be the place to grab it: https://github.com/fsharp/FSharp.Data/tree/XsdProvider

I'm looking forward to this too, now I'm trying to come up with XML examples to feed the TP to cover all properties I need to write out other XML, would be nicer to define it in xsd

[giacomociti]

@CumpsD you may check out my advice to create samples automatically. Looking forward to the XsdProvider we haven't heard from @runefs for a while. I hope there is some workaround to make it work on mono.

[CumpsD]

@giacomociti thanks for the advice!

[runefs]

I've spent some time on the XsdProvider again however after fetching the
latest from the master I have some unexplained problems compiling the
JsonProvider test cases, which is where I'm stuck for now. Due to this I
haven't gotten to a state where I've completed all unit test. However some
of the bugs from previous versions are now fixed. So I'll see if I can
figure out why the compiler complains that it can't load the appropriate
type for the JsonProvider

2015-12-03 9:47 GMT+01:00 Giacomo Citi notifications@github.com:

@CumpsD https://github.com/CumpsD you may check out my advice
#890 (comment)
to create samples automatically. Looking forward to the XsdProvider we
haven't heard from @runefs https://github.com/runefs for a while. I
hope there is some workaround to make it work on mono.

—
Reply to this email directly or view it on GitHub
#57 (comment).

[runefs]

I'm currently stuck with this error message

"....FSharp.Data.Tests/JsonProvider.fs(22,22): Error FS1109: A reference to
the type 'FSharp.Data.JsonProvider,Sample=" [ {"a":12.3}, {"a":1.23,
"b":1999.0} ] ",SampleIsList="True"' in assembly 'FSharp.Data' was found,
but the type could not be found in that assembly (FS1109)
(FSharp.Data.Tests)"

Since I haven't changed anything (willingly at least) with regards to the
JsonProvider I'm at a loss. Any ideas to what could cause this would be
very welcome :)

2015-12-09 19:42 GMT+01:00 rune funch søltoft rune@funch.dk:

I've spent some time on the XsdProvider again however after fetching the
latest from the master I have some unexplained problems compiling the
JsonProvider test cases, which is where I'm stuck for now. Due to this I
haven't gotten to a state where I've completed all unit test. However some
of the bugs from previous versions are now fixed. So I'll see if I can
figure out why the compiler complains that it can't load the appropriate
type for the JsonProvider

2015-12-03 9:47 GMT+01:00 Giacomo Citi notifications@github.com:

@CumpsD https://github.com/CumpsD you may check out my advice
#890 (comment)
to create samples automatically. Looking forward to the XsdProvider we
haven't heard from @runefs https://github.com/runefs for a while. I
hope there is some workaround to make it work on mono.

—
Reply to this email directly or view it on GitHub
#57 (comment).

[Kazark]

👍

[giacomociti]

I've been playing again with my implementation and released it as a separate project.

I just referenced most source code of FSharp.Data, added one file (XsdInference.fs) and slightly
modified XmlProvider.fs adding a Schema parameter.

Of course I just could have made a PR to FSharp.Data. But, first, we already have an ongoing attempt.
And I'm also afraid that my stuff may have more or less the same issues when it comes to integration and making things work across multiple platforms and versions.

So I'm glad if someone is willing to experiment with FSharp.Data.Xsd but be warned about this kind of issues and its overall maturity.

[TonyHenrique]

I have a complex XSD Schema (multiple files). How can I use F# to read it so I can use this to generate an output XML, with my data on that XSD structure?

nfe_v4.00.xsd and xmldsig-core-schema_v1.01.xsd

http://www.nfe.fazenda.gov.br/portal/exibirArquivo.aspx?conteudo=BALgvpK9Jvo=

In C# I could do

xsd.exe nfe_v4.00.xsd xmldsig-core-schema_v1.01.xsd /c /edb

How can I do this in F# (Code Generated F# Types), or, can I read these two XSD directly on F# (F# Type Provider)

[giacomociti]

hi @TonyHenrique, xsd is not supported yet in F# Data. The XsdProvider branch is not merged, nor is my PR #1004
If you are willing to use FSharp.Data.Xsd instead, I think your use case is easily supported with the ResolutionFolder parameter, as explained here

[ovatsus]

I'll have a look at trying to merge that this month. Should I consider that PR as a replacement for the old branch?

[giacomociti]

Yes, the PR is an alternative to the existing branch

[TonyHenrique]

@giacomociti After adding ResolutionFolder, it can read the XML using the 2 XSD definitions.
Now, how can I populate the F# type with Data and generate a XML? is that supported? Can you post a link describing how to do this?

[giacomociti]

@TonyHenrique type providers are more geared towards reading, but writing is supported simply using the constructors on the types generated by the XmlProvider, as explained in the Transforming XML section towards the end of the XmlProvider documentation

[brettrowberry]

Has this been resolved?

[toburger]

I'm already using the separate FSharp.Data.Xsd Type Provider, but it would be awesome if the excellent work of @giacomociti could be merged into FSharp.Data.

[ovatsus]

I also really want this, it's on the queue for review after the .net standard PR is finished On 22 Mar 2018 10:37, Tobias Burger <notifications@github.com> wrote: I'm already using the separate FSharp.Data.Xsd Type Provider, but it would be awesome if the awesome work of @giacomociti<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgiacomociti&data=02%7C01%7C%7C3b556186d0fb4e04094708d58fe0f202%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636573118689298244&sdata=U7C85i8Ox877AlNCITvLIr5sjw13vm2f5ZtI2vDcOKc%3D&reserved=0> could be merged into FSharp.Data. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffsharp%2FFSharp.Data%2Fissues%2F57%23issuecomment-375257758&data=02%7C01%7C%7C3b556186d0fb4e04094708d58fe0f202%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636573118689298244&sdata=1OyxVkusO7jhe4Tq7qqAVrd2SRxzrWtundYEQ1K984c%3D&reserved=0>, or mute the thread<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAtFyQMWi_MDbuJ5SMx8-oWLjLh0uwC3ks5tg370gaJpZM4AZo08&data=02%7C01%7C%7C3b556186d0fb4e04094708d58fe0f202%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636573118689298244&sdata=oPByqJ4o4A8N5jntMkrnxCqiUuSNKq5hZb3CRNbCCzo%3D&reserved=0>.