Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recursive schema (tree-like) validation #379

Closed
Marsup opened this issue Jul 10, 2014 · 30 comments
Closed

Recursive schema (tree-like) validation #379

Marsup opened this issue Jul 10, 2014 · 30 comments
Assignees
Labels
feature New functionality or improvement
Milestone

Comments

@Marsup
Copy link
Collaborator

Marsup commented Jul 10, 2014

Hello,

I'm wondering what's the best way to accomplish recursive schema validation.
Let's say I have an object that contains a property "children" holding an array of objects of the same kind.
The only way I see is to create the 1st layer of validation to get a joi object without "children", then call .keys again on it like schema = schema.keys({ children: Joi.array().includes(schema) }) but that only gets me to validate the 1st level, not recursively.
Any tips ?

@Marsup
Copy link
Collaborator Author

Marsup commented Aug 1, 2014

@hueniverse you mentioned in #154 it was possible, would you mind making a short snippet for that ?

@hueniverse
Copy link
Contributor

That particular example was not full recursion. I don't see how this can be done without special support for it.

@krisb
Copy link

krisb commented Sep 16, 2014

I would like to be able to handle a recursive schema too.

@hueniverse - I'd be happy to put a PR together if you have some guidance on how you would like this done.

@Marsup
Copy link
Collaborator Author

Marsup commented Sep 30, 2014

Hey @hueniverse, I've been thinking a lot about this one, it seems that the only thing keeping it from working in the current state is that both object.keys and array.includes/excludes are making clones, if current objects were modified it would work. Would you be open to new alternative methods to these that would do that ?

@hueniverse
Copy link
Contributor

@Marsup what would that look like? Show me a code example of how you would use it.

@Marsup
Copy link
Collaborator Author

Marsup commented Oct 14, 2014

@hueniverse Considering the OP example :

var schema = Joi.object({
  name: Joi.string()
});
schema.alterKeys({ children: Joi.array().includes(schema) });

It's the same API but without the inside clone, no need to grab the return value either since schema is modified.
Same way for arrays.

@markstewart
Copy link

I was in need of this feature myself and have implemented something for discussion in the pull request above.

Example:

var tree = Joi.object({
    name: Joi.string(),
    children: Joi.array().includes(Joi.recurse())
}).recursive();

It does maintain the immutable property by creating a marker recurse object that gets replaced with the type marked with recursive() at validation time.

@Marsup
Copy link
Collaborator Author

Marsup commented Oct 14, 2014

This might prove easier to describe having a concrete type.
You're missing recursive arrays though.

@markstewart
Copy link

@Marsup Can also use with recursive array types so for example:

var array = Joi.array().includes(Joi.number(), Joi.recurse()).recursive();
array.validate([[1, 2, 3, [3, [4, 5]]]], function (err, value) {});

I tried to stay away from mutating the schema objects. Not sure of the implications but it seemed an important property to maintain.

This could also be solved by introducing named types so maybe something like any.typeName(string) and Joi.namedType(string) to reference the previously marked type instead of recurse()?

@Marsup
Copy link
Collaborator Author

Marsup commented Oct 14, 2014

You're missing tests on it then :)
I thought it would be doable with existing Joi.ref but I don't have time to investigate right now.
Will it still work with 2 recursive calls ? (ie. recursive array inside a recursive object)

@markstewart
Copy link

@Marsup My proposal won't work with some nesting scenarios since it doesn't allow a way to reference the outer type from within the inner one. Will need to try something different to support that.

@martinheidegger
Copy link
Contributor

Just figured out that this is very important to one of my tasks. +1 need it too, right now badly.

@Marsup
Copy link
Collaborator Author

Marsup commented Mar 9, 2015

I need it as well but I'm still searching for a decent API to represent that.

@Lotes
Copy link

Lotes commented Mar 27, 2015

I think you need a kind of "defs" section like in SVG - a region of predefined blocks for later reuse. Using a lazy lookup function you could realize the recursive aspect.

var defs = Joi.definitions(function() {
  //where "this" is the "definitions" object
  return {
    //name: type pairs
    tree: Joi.object.keys({
      value: Joi.string(),
      children: Joi.array().items(this.lookup('tree')) 
      //lazy lookup call (returning a function that will be evaluated on validation)
    }),
    anotherType: Joi.number(),
    //...
  };
});
var treeSchema = defs.lookup('tree');

Is that doable?

@DavidTPate
Copy link
Contributor

Depending on the use case, I could see a representation similar to JSON schema's "additionalPropteries" or "patternProperties" working.

I think the best solution though would be to be able to reference another schema without making a clone of it.

@tjmehta
Copy link

tjmehta commented Dec 1, 2015

Ran across this problem today. Would love to send a PR if anyone has implementation ideas.

@Marsup
Copy link
Collaborator Author

Marsup commented Feb 13, 2016

Quick poll, what does everyone think about a Joi.lazy primitive that would be used as follow :

const schema = Joi.object({
  children: Joi.array().items(Joi.lazy(() => schema))
}) 

With probably an additional string argument or something for description.

@Eirik81
Copy link

Eirik81 commented Mar 17, 2016

I also need to validate recursive objects. I think the Joi.lazy primitive seems like a good solution.

@AdriVanHoudt
Copy link
Contributor

just an idea: Joi.lookup()?

@Marsup
Copy link
Collaborator Author

Marsup commented Mar 17, 2016

With the same behavior or something else entirely ?

@AdriVanHoudt
Copy link
Contributor

same behaviour, just another name that popped in my head

@myndzi
Copy link

myndzi commented Mar 26, 2016

I was just looking into this to, amusingly, validate the output of schema.describe such that all items in the schema have defaults. I was looking for something like Joi.ref() but rather than referring to a value it would refer to a schema. I'm not sure how I feel about lazy where the schema is specified by some callback function. There's already support for resolving dependencies for .ref, but in this case since it would be recursive, I'm not sure that works. It seems to me that referencing the schema to recurse by its string key is more in keeping with the Joi api as is, and simpler, too... though the syntax would have to be expanded to allow support for root references. Maybe something like this:

const schema = Joi.object().keys({
  children: Joi.object().pattern(/./, Joi.recurse('^'))
});

Or,

const schema = Joi.object().keys({
  foo: Joi.string(),
  bar: Joi.object().pattern(/./, Joi.recurse('^bar')
});

@ghost
Copy link

ghost commented Apr 4, 2016

Searching for recursive validation i just fall in this topic, then playing with the node shell i find that solution

var obj = Joi.object().keys({b: Joi.any()})
obj = obj.keys({a: obj}) //add the reference extending the base object
obj = obj.keys({a: obj}) //add recursion

//try it
obj.validate({ a: { a: { b: 3 }, b: 4 }, b: 3 }, (err, result) => {if(err) throw err; console.log(result);})
/* prints "{ a: { a: { b: 3 }, b: 4 }, b: 3 }" */
obj.validate({ a: { a: { b: 3 }, b: 4, c: 5 }, b: 3 }, (err, result) => {if(err) throw err; console.log(result);})
//throws ValidationError: child "a" fails because ["c" is not allowed]

despite this solution I like the proposal for

const schema = Joi.object().keys({
  foo: Joi.string(),
  bar: Joi.object().pattern(/./, Joi.recurse('^bar')
});

@myndzi
Copy link

myndzi commented Apr 9, 2016

By my understanding from looking into it the other day, this will only actually work to the depth that you repeat line 3. That is, it's not recursing, it's just explicitly specifying another level of depth each time you do obj.keys({ a: obj }). In your example, this fails:

obj.validate({ a: { a: { a: { b: 3 }, b: 3 }, b: 3 }, b: 3 })

@MarkHerhold
Copy link

Does anyone have a hack solution to this or a PR in-progress?

I think one solution would be to recurse the tree yourself, validating direct children and propagating validation errors up the call stack. Another would be to use another validator for the time being (like ajv which seems to have solved this). 😞

@jdarling
Copy link

jdarling commented May 25, 2016

Joi.custom(function) seems like it might be useful. Though the name may not be so good.

Basic idea:

const validateTaskOrSubTask = (value)=>{
  if(value instanceof Task){
    return Joi.validate(value, Task);
  }
  if(Array.isArray(value)){
    return value.map(validateTaskOrSubTask);
  }
  return Boom.invalid('Unknown or invalid value'); // Yeah, I don't know what this should really look like...
};

const Task = Joi.object().keys({
  type: Joi.string(),
});

const Job = Joi.object().keys({
  tasks: Joi.custom(validateTaskOrSubTask)
});

Excuse my typos, but its a proposal so hopefully they are allowed :D

@Eirik81
Copy link

Eirik81 commented Jul 14, 2016

I'm using hapi-swagger 6.1.0 and joi 9.0.0. When using Joi.lazy for response validation loading the /docs webpage fails with this message:

500 : {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"} 
http://localhost:9999/swagger.json

In the console I get this error:

160714/104936.718, [error], message: Uncaught error: Cannot read property 'type' of undefined stack: TypeError: Uncaught error: Cannot read property 'type' of undefined
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:144:24)
    at Object.internals.parseArray (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:396:40)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:181:30)
    at joiObj.forEach (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:93:46)
    at Array.forEach (native)
    at Object.properties.parseProperties (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:84:16)
    at Object.properties.toParameters (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:40:36)
    at Object.definitions.appendJoi (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/definitions.js:36:41)
    at Object.internals.parseObject (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:333:74)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:176:30)
Debug: internal, implementation, error 
    TypeError: Uncaught error: Cannot read property 'type' of undefined
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:144:24)
    at Object.internals.parseArray (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:396:40)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:181:30)
    at joiObj.forEach (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:93:46)
    at Array.forEach (native)
    at Object.properties.parseProperties (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:84:16)
    at Object.properties.toParameters (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:40:36)
    at Object.definitions.appendJoi (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/definitions.js:36:41)
    at Object.internals.parseObject (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:333:74)
    at Object.properties.parseProperty (/home/eirik/Repos/data-store-mockmagic/node_modules/hapi-swagger/lib/properties.js:176:30)

@Marsup
Copy link
Collaborator Author

Marsup commented Jul 14, 2016

And it's clearly an error in hapi-swagger, how does this concern me ?

@Eirik81
Copy link

Eirik81 commented Jul 14, 2016

I posted it here since the error only occurs when using Joi.lazy, so I thought there might be a compability issue between hapi-swagger and the joi.lazy method. But I'll post the issue at the hapi-swagger project as well.

@lock
Copy link

lock bot commented Jan 9, 2020

This thread has been automatically locked due to inactivity. Please open a new issue for related bugs or questions following the new issue template instructions.

@lock lock bot locked as resolved and limited conversation to collaborators Jan 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature New functionality or improvement
Projects
None yet
Development

No branches or pull requests