Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tv4.validate not finding problems #3173

Closed
tlemons opened this issue Jun 16, 2017 · 23 comments

Comments

Projects
None yet
9 participants
@tlemons
Copy link

commented Jun 16, 2017

App Details:

Postman for Windows
Version 5.0.0
win32 10.0.10586 / x64

Hi - over the last few days, I've begun experimenting with Tiny Validator snippet functionality in the Postman test facility. This doesn't SEEM to be working, in that it doesn't find problems/bugs that I introduce into the schema

Here is a JSON response from one of our REST APIs (which, right now, doesn't show much information, and that is intentional in this public setting):

{
"_embedded": {
"clients": []
},
"_links": {
"self": {
"href": "https://10.7.104.144/elg/rest-api/clients"
},
"profile": {
"href": "https://10.7.104.144/elg/rest-api/profile/clients"
}
},
"page": {
"size": 20,
"totalElements": 0,
"totalPages": 0,
"number": 0
}
}

I copied this JSON data to https://jsonschema.net/#/editor, and generated the following JSON Schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"definitions": {},
"id": "http://example.com/example.json",
"properties": {
"_embedded": {
"id": "/properties/_embedded",
"properties": {
"clients": {
"id": "/properties/_embedded/properties/clients",
"items": {},
"type": "array"
}
},
"type": "object"
},
"_links": {
"id": "/properties/_links",
"properties": {
"profile": {
"id": "/properties/_links/properties/profile",
"properties": {
"href": {
"id": "/properties/_links/properties/profile/properties/href",
"type": "string"
}
},
"type": "object"
},
"self": {
"id": "/properties/_links/properties/self",
"properties": {
"href": {
"id": "/properties/_links/properties/self/properties/href",
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"page": {
"id": "/properties/page",
"properties": {
"number": {
"id": "/properties/page/properties/number",
"type": "integer"
},
"size": {
"id": "/properties/page/properties/size",
"type": "integer"
},
"totalElements": {
"id": "/properties/page/properties/totalElements",
"type": "integer"
},
"totalPages": {
"id": "/properties/page/properties/totalPages",
"type": "integer"
}
},
"type": "object"
}
},
"type": "object"
}

I used this schema to prepare this Postman test:
var jsonData = JSON.parse(responseBody);
var schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"definitions": {},
"id": "http://example.com/example.json",
"properties": {
"_embedded": {
"id": "/properties/_embedded",
"properties": {
"clients": {
"id": "/properties/_embedded/properties/clients",
"items": {},
"type": "array"
}
},
"type": "object"
},
"_links": {
"id": "/properties/_links",
"properties": {
"profile": {
"id": "/properties/_links/properties/profile",
"properties": {
"href": {
"id": "/properties/_links/properties/profile/properties/href",
"type": "string"
}
},
"type": "object"
},
"self": {
"id": "/properties/_links/properties/self",
"properties": {
"href": {
"id": "/properties/_links/properties/self/properties/href",
"type": "string"
}
},
"type": "object"
}
},
"type": "object"
},
"page": {
"id": "/properties/page",
"properties": {
"number": {
"id": "/properties/page/properties/number",
"type": "integer"
},
"size": {
"id": "/properties/page/properties/size",
"type": "integer"
},
"totalElements": {
"id": "/properties/page/properties/totalElements",
"type": "integer"
},
"totalPages": {
"id": "/properties/page/properties/totalPages",
"type": "integer"
}
},
"type": "object"
}
},
"type": "object"
};

tests["Validate JSON return"] = tv4.validate(jsonData, schema);
console.log("JSON validation failed: ", tv4.error);

The problem that I see is, if I modify the schema to induce a problem, tv4 doesn't detect this. For instance, in the schema, if I change "page" to "XXpage" and execute the test, tv4.validate still returns null (=success).

I'm doing this testing to ensure that, if my JSON output ever changes, the differences between it and the schema will be detected.

Is the tv4 code wrong, or is my schema wrong, or something else?

@madebysid

This comment has been minimized.

Copy link

commented Jun 19, 2017

Hi @tlemons, we use the tv4 library as-is. Can you try testing your code over at https://npm.runkit.com/tv4 and see if you're still seeing this problem?

@madebysid madebysid self-assigned this Jun 19, 2017

@tlemons

This comment has been minimized.

Copy link
Author

commented Jun 19, 2017

Hi - thanks very much for the reply. And thanks for the pointer to runkit. I tried it there, and see the same problem; if I purposely cause a mismatch between the schema and the json data, tv4 does not detect it. I'm using the JSON schema that was generated from my JSON data at https://jsonschema.net/#/editor; have there been any reports of problems with using JSON schema generated by that site?

Thanks!
tl

@madebysid

This comment has been minimized.

Copy link

commented Jun 20, 2017

Can you also create an issue here: https://github.com/geraintluff/tv4 since this looks like a TV4 issue?
I'm closing this, but please let us know here when this gets fixed there and we'll bump up our version to include the fix :)

@tlemons

This comment has been minimized.

Copy link
Author

commented Jun 27, 2017

As you suggested, I opened a TV4 bug at geraintluff/tv4#257.

@tlemons

This comment has been minimized.

Copy link
Author

commented Jun 30, 2017

I have not received an response to the TV4 bug that I posted, and I see dozens of issues at https://github.com/geraintluff/tv4/issues, some of which have been open for years.

I really want to do JSON return body checking in Postman, and like to idea of using the JSON Schema for this. If TV4 doesn't work and isn't supported, have you considered supporting a different JSON Schema-compliant checker in Postman?

Thanks
tl

@sanamvarma

This comment has been minimized.

Copy link

commented Nov 13, 2017

Hi,
I am facing a similar issue:
my response schema is

const resSchema =
{
"$schema": "http://json-schema.org/draft-04/schema#",
"required": ["T1","T2"] ,
"properties":
{
"topicId": { "type": "integer" },
"topicNamesssss": { "type": "string" },
"topicNote": { "type": "string" },
"grade": { "type": "integer", },
}

};

My response is :
{
"topicId": "1",
"topicName": "2",
"topicNote": "Area: Concept & informal units",

}

pm.expect(tv4.validate(response1, resSchema)).to.be.true; is always returning true.

Even if i am passing my response as "", tv4.validate is returning true.

I validated the same in a JSONvalidator and that correctly shows me the discrepancies between the schema and response. PFA screenshot for it.

schema validation

Any help would be appreciated.

Thanks,
Sanam

@rmlira

This comment has been minimized.

Copy link

commented Nov 28, 2017

I have tried with the banUnknownProperties (https://github.com/geraintluff/tv4#the-banunknownproperties-flag) and it works.

Just do:

tv4.validate(JsonData, schema, false, true);

@arjunUnniumbath

This comment has been minimized.

Copy link

commented Dec 20, 2017

is this issue fixed?

@rmlira

This comment has been minimized.

Copy link

commented Dec 25, 2017

Yes, with the additional parameters it works properly.

@mverma-va

This comment has been minimized.

Copy link

commented Mar 15, 2018

It works with additional parameter but only if the variable is defined in testscript and not when it is get from environment variables.
This WORKS:
pm.test('User schema is valid', function() { pm.expect(tv4.validate(allData, user_schema, false, true)).to.be.true});
This Does NOT WORKS:
pm.test('User schema is valid', function() { pm.expect(tv4.validate(allData, pm.environment.get("user_schema"), false, true)).to.be.true});
Any suggestions to fix it???

@rmlira

This comment has been minimized.

Copy link

commented Mar 17, 2018

I did a simple test and it worked. In prerequest scripts:

const schema = {
    type:"object",
    properties:{
        "ip": {
            "type":"string"
        }
    }
};

pm.environment.set("SCHEMA", schema);

then in tests:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), pm.environment.get("SCHEMA"), false, true), tv4.error).to.be.true;
})

in a Get request to https://postman-echo.com/ip

Try this way, if not work can you show your scripts?

@mverma-va

This comment has been minimized.

Copy link

commented Mar 19, 2018

@rmlira There is still one difference (they way you did works for me too), but I did not set the variable using pm.environment.set, I defined the variable itself in "Manage Environment" and then it does not work, can you try that.
Just remove your set statement and set the variable first manually in your "Manage env" section.

Also noticed one more thing that, even if I set it in one request and try to test this in another request using "get" it does not work (always pass and not fail on valid failure)

Tried the same scenario with "Get request https://postman-echo.com/ip", same issue still

@rmlira

This comment has been minimized.

Copy link

commented Mar 19, 2018

Hi @mverma-va . I see your point, sorry, I had not understood before. I made this test:

On Postman > Manage Environments > Add a new environment > create in this new environment a variable 'schema' with the content:

schema:{type:"object",properties:{"ip": {"type":"string"}}}

(bulk edit)

Then in request, on Tests I remove the prerequest scripts and keep the schema test as follows:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), pm.environment.get("schema"), false, true), tv4.error).to.be.true;
})

And it doesn't work properly, I mean it pass the test but if I change the 'ip' type to 'number' in environment it still passing the test when it should not.

I noticed that when you store schema in environment using 'pm.environment.set' it stores as an object ([Object object]), when you store it manually it will be stored as a string. So to make it works properly I change the schema stored in environment to:

schema:{"type":"object","properties":{"ip":{"type":"string"}}}
(bulk edit)
just adding a few quotation marks...

Then in test script i made a JSON parse from this stored schema as follows:

pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true), tv4.error).to.be.true;
})

Converting from string to proper object.

Now it works properly. Can you check if this solves your problem? If not can you post your schema too?

@mverma-va

This comment has been minimized.

Copy link

commented Mar 20, 2018

Hi @rmlira That worked for me properly. My variable already had quotes, it is basically JSON.parse() which solved the problem, thanks a lot!

@radauto

This comment has been minimized.

Copy link

commented Jun 26, 2018

@mverma-va - could you please help , i get below error while running my scripts after last changes you have mentioned.
schema | JSONError: Unexpected token u in JSON at position 0

pm.test("schema", function()
{
pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true),tv4.error).to.be.true;
})

@radauto

This comment has been minimized.

Copy link

commented Jun 26, 2018

@rmlira - could you please help , i get below error while running my scripts after last changes you have mentioned.
schema | JSONError: Unexpected token u in JSON at position 0

pm.test("schema", function()
{
pm.expect(tv4.validate(pm.response.json(), JSON.parse(pm.environment.get('schema')), false, true),tv4.error).to.be.true;
})

@mverma-va

This comment has been minimized.

Copy link

commented Jun 26, 2018

@radauto not really sure, maybe you are missing some quotes or braces like "{" in your schema json object

@radauto

This comment has been minimized.

Copy link

commented Jun 26, 2018

@mverma-va Thanks for your immediate response , i did some changes and now i get
schema | AssertionError: ValidationError: Invalid type: array (expected object): expected false to be true

  1. Defined schema in pre request script
    2.my test has
    pm.test("schema", function() {
    pm.expect(tv4.validate(pm.response.json(), pm.environment.get("SCHEMA"), false, true), tv4.error).to.be.true;
    })

if i send you sample schema will you be able to help ?

@mverma-va

This comment has been minimized.

Copy link

commented Jun 26, 2018

@radauto you can use these two tools to 1.create and 2.validate your schema.

  1. https://jsonschema.net/#/editor
  2. https://www.jsonschemavalidator.net/
@radauto

This comment has been minimized.

Copy link

commented Jun 26, 2018

@mverma-va Thank you. i have corrected and its working fine.
but , my console shows as
Schema Validation | AssertionError: expected false to be true , when my schema is not matching response which is correct.

But is there a way , we can get actual Variable which is wrong as per schema ?

@nsharma0101

This comment has been minimized.

Copy link

commented May 14, 2019

@mverma-va @radauto My test is working fine and getting proper message.
Schema Validation | AssertionError: expected false to be true ,
Is there a way to capture exact object or property which is missing or incorrect. could you please share if you found the solution.

@codenirvana

This comment has been minimized.

Copy link
Member

commented May 14, 2019

@nsharma0101 tv4 is deprecated in favor of Ajv.
Refer #3300 (comment) for debugging schema validation errors using Ajv.

@rmlira

This comment has been minimized.

Copy link

commented May 14, 2019

@nsharma0101
you can check tv4 documentation here https://geraintluff.github.io/tv4/.

The method tv4.validate returns false or true if validation is succeeded or not and stores and object with error details which you can access calling tv4.error. Or you can use tv4.validateMultiple instead of tv4.validate. The result of this call will be an object with 'errors' property containing detailed causes of failure. The advantage is that you will have all schema errors reported in one call, not only the first error as returned by tv4.validate.

example:

let schemaValidation = tv4.validateMultiple(pm.response.json(), JSON_SCHEMA, false, true);

Now schemaValidation variable grabs result of validation with all errors that could be found.

To validate postman test you can do:

pm.expect(schemaValidation.valid, schemaValidation.errors).to.be.true;

in case of error all messages in schemaValidation.errors will be listed as result of pm.expect call.

Anyway now will be better if you migrate to Ajv as suggested by @codenirvana

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.