Add method to extract external schema urls from used $ref's

[Bartvds]

Optionally only the ones that aren't loaded yet. This methods could then be used to simplify preloading the schema's for tv4.addSchema() and synchronous validation.

I could have a got at this myself but a complications will be to find a way to resolve nested or recursive references. Do you got an idea on how to approach this?

[geraintluff]

That would be useful, yeah.

When adding a schema, tv4 performs a normalisation pass (to convert "id" and "$ref" values into absolute URIs) (function source, called from code).

A tidy way to collect $refs might be to move that function to inside the ValidatorContext class, so it can collect the $ref values as it goes along.

I'm not quite sure what the issue would be with nested/recursive references, though - if ValidatorContexts have a list of all $refs in schemas that have been added with then, then you should be able to repeatedly check for $refs that haven't been added using addSchema(), and add the missing ones. Or do you mean the case when a $ref points to itself?

[Bartvds]

With nested/recursive references I meant a schema that refers to an external schema that refers to another external schema (etc etc).

To get at the urls of all of them it would need multiple passes: check a schema and get the urls, load and add the sub schemas, check again for new missing ones, load them, and repeat this until they're all cached.

Now I know tv4 will collect the mappings it could have an api like this:

//add our local root schema with no uri (if a uri is set in the data then auto-register it)
//maybe this should clear the cache as well?
tv4.initSchema(schema);  

//could also be an addSchema() overload without the uri
// this could allow to preset in a bunch of schema's from local .json without fiddling around to extract their own uri's
tv4.addSchema(schema);  

//then grab the list of know ref uri's
var uriList = tv4.getRefUriList(ntoloadedOnly);

//or maybe a bit more reusable in an alternate object style;
//by mapping uri's to schema (with falsy for non-yet-loaded schemas)
var uriMap = tv4.getRefUriMap();

This can then be repeated until no empty uri's show up, after which you can validate synchronously by anyone of these schema.

What do you think?

[geraintluff]

Sounds about right, yeah. :)

[Bartvds]

Ok, cool. Do you feel a need to implement this yourself? Otherwise I'll have a go at it.

If I take this I'll create a develop branch so it won't interfere with master until ready.

[geraintluff]

Just to check - what's going into tv4 would just be some methods/variables like tv4.referenceUrls and tv4.getMissingReferences(), and the actual fetch/add recursion loop would be a separate thing to be written using those?

I don't have a particular need to write everything myself, so by all means go for it.

[Bartvds]

Yes, only the basic accessor methods like tv4.referenceUrls, tv4.getMissingReferences() and the slightly modified initial tv.addSchema() should be in the tv4 core (I like those names btw).

They would just be a clean way to get at the reference data so we can do stuff with it (like loading, but also for debugging). The actual fetching and adding would be external.

Once that is stabilised and used in practice we could look at hammering the fetchers I'd write for my own projects into a plugin/addon type of thing.

[geraintluff]

Sounds great - look forward to seeing it. )

[Bartvds]

Just now I committed my edits for this to a develop-branch. It's an ugly commit; I tried to clean it up but only got worse: had made the mistake of modifying stuff while I was implementing this (so we learn :)

One thing I added besides implementing Issue was source-map support via grunt-concat-sourcemap/sourcemap-support: now node.js stack traces of the core and tests display path to the actual fragment file (instead of the concatenated file). Pretty neat, also works with in Chrome Debugger if you enable it (also breakpoints etc :)

Also I cleaned some stuff in the tests.

[Bartvds]

@geraintluff I'm mergin this with your recent edits and pushing it to /master