New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doesn't work with large JSON files #11
Comments
Also its really slow for large files |
Patches are welcome! ;-) |
My file is only 854K, but it seems to running forever, going to leave it overnight to see if it can finish... |
I think it's working more better in terminal mode than js implementation in browser. But I have not largest files to confirm. |
Any news regarding this? |
I don't get an out of memory exception with my 60MB json files. It just never finishes and my job agents kills it after an hour, I left it running longer than that locally but after most of a day I called it quits. Performance seems to degrade exponentially with file-size as my 7MB files completes "nicely" after about 150 seconds comparatively. Would be nice if this performed at linear speed with file size. |
@philipborg I had the same issue.. it was working before I made changes to my tsconfig. It just hangs. @andreyvit I forget exactly what changes I made but it was something like changing my tsconfig to target: "esnext" |
I have found a workaround for my needs I thought I'd share. It seems that comparing large arrays of objects is the issue. Replacing the array with an object, so it only compares key matches, solved it for me. Now it goes decently fast even with 60MiB files. My objects themself contains arrays of strings which doesn't seem to cause any problems. So instead of [
{
"field1": 2,
"field2": "foo",
[...]
},
{
"field1": 1337,
"field2": "bar",
[...]
}
] I used something more akin to {
"MyIdentifier1": {
"field1": 2,
"field2": "foo",
[...]
},
"MyIdentifier2": {
"field1": 1337,
"field2": "bar",
[...]
}
} This only works if you can find an unique identifier in the data to use as the object key. In my case I had to do a composite key and escape my separators. It worsens the comparison and requires you to mutate the json before comparing, so it's just a workaround and not an issue solution. Essentially it restricts the search scope for what it will compare objects with as now it only compares objects with matching keys. |
Trying to compare two files of size 701MB results in
The text was updated successfully, but these errors were encountered: