Performance bad for big JSON files/instances

[abentele]

Because the validation is based on an in-memory representation of the file, I get an OutOfMemoryError if I want to parse a big file with > 1.000.000 objects. Even if the Json is smaller, the memory consumption may be critical for production use.

The validator should be a streaming parser with almost constant memory consumption.

[fge]

Too late for 2.x...

But yeah, I did encounter a problem already with poor performance. Not sure how error messages will be collected however, since it means keywords which check for structural validity will have to be postponed.

It's doable anyway, but that will be for 3.x.

[pat-humphreys]

@abentele Did you end up using another Library for this? As I am getting the same issue over 1M entries the process is using x4 times the memory of the file to validate it and not releasing the memory, even though I am streaming the file.

[abentele]

@pat-humphreys: no. I didn't find any library that solves this problem (in Feb/2015); I did it just without validation. If the current lib would offer streaming validation with little memory usage, I still would use it.

[pat-humphreys]

@abentele I just did some testing with networknt and it takes 20.681 Secs to validate 6.2M records, where as using json-schema-validator I killed it after 24Mins (+it used 6.5GB of RAM) and it only validated 1.4M of the records. So until this major performance issue is sorted with this library I would advise not using it