Sharepoint metadata does not convert to correct JayData context

[PeterAronZentai]

According to Rainer's report at https://gist.github.com/2853140 in issue #9

[RainerAtSpirit]

Can't test at the moment with the latest version of jaySvcUtil as the option to use a filename for the -mswitch is broken.
But taking the dynamic nature of SharePoint into account, where the listdata.svc runs in the context of the current site, it doesn't feel right to rely on an command line utility for creating Entities. Luckily jaySvcUtil internally uses XSLT for the heavy lifting so it shouldn't be too hard to come up with a solution.

[PeterAronZentai]

Ooops, indeed the -m option for local files has been broken. Thanks for finding this bug. I create a separate issue for it, and starting fixing - its a good luck I have it still open :)

[PeterAronZentai]

Regarding dynamic usage: the XSLT that processes the metadata is sadly Microsoft bound because of the use of msxml:node-set. This is a rather easy to resolve thing as standards based environments support node-set() instead. What I am not sure if widely supported is the extension methods and functions that is part of XSLT standard - but a couple of years ago it wasn't implemented commonly. Maybe we just have to remove that functionality from the translator...

What my biggest question is: how one could support a dynamic SharePoint scenario with changing list schemas from a client point of view? What I am trying to point out is that unless the UI is completely dynamic the code will not change, so there must be some update to the client if it wants to utilize new sharepoint fields appearing in the list or a new list in is entirety... We are into that way but far from it... Are you pursuing such a scenario (like dynamic data was for the server side)?

[PeterAronZentai]

It has been fixed with the last commit to the JayData/JaySvcUtil project where the tool lives now. I have the updated binaries and now update the codeplex download page... NuGet will be updated tomorrow :( i think

[PeterAronZentai]

this issue is about the context generated from sharepoint and not the -m option :) so I close that one and leave this open until further info comes in... I have successfully generated a sharepoint context js, you can find it here: https://gist.github.com/2883025

Note, that I had to edit the serviceUri in the default generated instance as it pointed to the local file (another sortof issue)
Also note that I had to save the SharePoint metadata in UTF-8 in order for the funny parts to be funny.

[RainerAtSpirit]

Technically it should be possible to traverse SharePoint sites in a SPA (single page application) and creating the site context on the fly when selecting a site (e.g. server side $metadata / XSLT -> inline script). I'm just not sure how that would add any real value as end user can traverse SharePoint already.

On the other side I see value in enterprise point solutions (-> user is authenticated) that uses SharePoint data via OData.
Backbone and Backbone.SharePoint https://github.com/lstak/Backbone.SharePoint or KendoUI.data.odata are already doing a good job. Depending on what solution you are looking for that might be already all you need. See http://www.spirit.de/demos/metro/ServiceFiles/DynMetroStyle.aspx for an example of combining the OOTB DataForm web part to build dynamics tiles with a KendoUI grid to dynamically display the information.

For other kind of solutions it might be beneficial if the solution could react dynamically to changes in the structure of (certain) underlying lists and libraries. That's where I see the biggest advantage of JayData at the moment. Now I only need to find some spare time to come up with a nice POC to either proof me wrong or wright ;-).
Starting point would be here: http://www.spirit.de/demos/metro/ServiceFiles/JayData.aspx

[PeterAronZentai]

Okay, I see what you mean. We will address this in 1.0.6 for the browsers that support XSLT. Do you think it is a viable set of clients? Hope you'll be able to build on it in the blog sooner then the release of 1.0.6

JayData supports discovering and traversing it's metadata dynamically so building a solution that binds itself to the dynamic structure of a JayData SharePoint context is easy to do.

JayData reflection is accessible through the type.memberDefinitions array (from an instance get it with instance.getType().memberDefinitions) where you can see all the members of a type and the base that defined that member. Association and navigation properties are detectable too.

[RainerAtSpirit]

Nope, guess there's a misunderstanding here. XML->XSLT->JS will be done at the server side, browsers XSLT isn't reliable enough. A simple DataFormWebPart should be good enough to start with. I stripped out the function calls in your XSLT https://gist.github.com/2883277 and at a first glance that produces the same output as when running through JaysvcUtil.

[PeterAronZentai]

Oh I see, nice idea :) Function calls are only used to make the context definition parametric so that you can provide your own base classes, etc. These are not required for the core conversion of the metadata to JavaScript classes - just a sugar on top feature.

[RainerAtSpirit]

Ok making progress. Go here: http://www.spirit.de/demos/metro/ServiceFiles/JayData.aspx and make sure to log in. Open Firebug to inspect JayDataTest. If you run something like JayDataTest.Announcements.toArray() you see that the correct request is made, but then you get an JSON.parse error in JayData.js 16832.
Either an issue with the format of http://www.spirit.de/demos/metro/ServiceFiles/JS/StandAlone.js, which is a static version of the $metadata XSLT conversion or an issue caused by the OData returned by SharePoint.

BTW: At a first glance I couldn't find a getType method on JayData. Probably I misunderstood your comment about .getType().memberDefinitions. Would be great if you could provide a small example.

[RainerAtSpirit]

a) looks like part of the issue comes from list that has German Umlauts. As these are allowed chars in SharePoint at some point this needs to be tackled.
b) renamed the list in standalone.js, but the example is still broken

[PeterAronZentai]

It is strange as FF is capable of getting and displaying the Tasks list, other lists show the invalid character issue. Other browsers can not event access the Tasks list with the same issue. My first finding is that js scripts and data is transferred in ISO-59xx charset and not in UTF-8 even though the page has a correct meta charset utf-8.

Also I found that if I project the value of the result set with JayDataTest.Tasks.map(function(t) { return t.ContentType }).toArray(); then it's OK. so we are safe to assume it is only the special characters that cause the issue.

Diving deeper stay tuned :)

Regarding getType:
with an instance -> JeyDataTest.getType().memberDefinitions
with a type: $data.EntitySet.memberDefinitions

[RainerAtSpirit]

Thanks for looking into it. I'll look into the charset issue as well. Maybe something is broken in the process of converting the $metadata with the standalone XSLT. I can provide you write access to the demo if that helps your diving :).

[RainerAtSpirit]

I just used the latest available version of jaysvcutil to produce the .js file, which at least seems to solve the charset issue.
http://www.spirit.de/demos/metro/ServiceFiles/jaysvcutil.aspx.

I then rerun the standalone XSLT conversion in Altova http://www.spirit.de/demos/metro/ServiceFiles/altova.aspx and this produces the same result. Not sure what went wrong yesterday.

Both version still show the JSON.parse error though

[PeterAronZentai]

We are on it, my colleague, Viktor Lazar joins this topic too. And yes, I guess some write access could help. Could we somehow fiddle with the page itself? Viktor has an idea that setting charset attribs on the script block might help. In the mean time I try to reproduce this use case in local environment without sharepoint. Our language has also special chars (őŐűŰüÜ) that will produce the same issue i guess..

[RainerAtSpirit]

Great. I go ahead and create a new site collection for further testing and convert the example to a static html version, so that it's easier to fiddle around with it. Ping me via Email so that we can exchange access data.

[PeterAronZentai]

We have the bug. Viktor - as a fresh eye immediately spotted the mistake. It is a bug in JayData regarding the '$$unbounded' navigation property. So it's not a character issue, no need to ghostchase that. To our estimate this fix will be ready by the end of day, you'll get it first it for internal testing then we ammend it to our release if that suits you. Thank you for your valuable help so far and in the future too.

How can we contact you in an email? can not get your address.

[RainerAtSpirit]

@lazarv good job!
@PeterAronZentai No work required on my end sounds good :). Re Email: I'm registered on your site so you might have my data already. Otherwise simply use our companies info@ address, which will eventually find it's way to me.

[PeterAronZentai]

Quickrep: the case is solved, we could repro it independent of SharePoint and now we create the unit tests for it. Today or maybe tomorrow morning we'll have a new JayData.js that has this issue sorted out.

[PeterAronZentai]

The code that fixes this bug has been pushed to GitHub/jaydata/jaydata/development just about today morning. We will not make a release for this, mainly because 1.0.6 is on next friday with tons of other fixes (large part of them reported by @RainerAtSpirit) plus Any() and All() support in subqueries for OData V3 (this one is a superb new feature of OData I dont know how we used it without it)...