-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adapt DefiantJS for Node enviroment #4
Comments
It would be great, just tried to run it with node.js => window is not defined |
In order to make this work, suitable dependencies are needed. I've looking in to jsdom, xmldom, etc. Though some of these libs opens unexpected can of worms that requires some adjustments to those libs. Which is not desirable...I'll continue dissecting the problem. Meanwhile, suggestions are welcome |
you are a hard worker :) |
Hello hbi, is there any news on the Node implementation ? |
I have a few private project which I juggle between and currently I'm working on writing Longscript, desktop application, with which one can produce animations like these; http://longscript.com/gallery.php |
Has there been any additional effort on this? I can see a potential use case for my project. |
@Mctalian Yeah...a little bit. Though, I realized fast that this "path" requires dependencies and I'm looking for a way to do it with fewest dependencies. |
@hbi99 I see. The main thing I've noticed is that utilizing the Snapshot feature causes my webpage to lock up for 0.5s to 2.0s depending. I figured if this was adapted for Node it would be non-blocking as well, and may assist in this area. This is a minor thing. I'm more curious about being able to move some things server-side for some enhancements I've thought of. Not sure how much I can help, but if there's anything I can do, let me know. |
Ok...thanks. Will do 👍 0.5-2 seconds...it sounds like you have a large JSON file...how big is it? |
Pretty big. I think there's something like 430 nodes total. And it's only going to get bigger over time! |
@Mctalian I might have a solution for improving the performance even more...and cut down the time it takes to create a snapshot. I'll get back regarding this. Thanks for the JSON data and feedback |
I will make a new issue for ya. |
I'm also looking for this. Do you happen to know if there are any existing alternatives out there that provide same/similar functionality for NodeJS? (also, great job on this project) |
@iojohnso I think there are a few repo's doing similar stuff, even though I haven't tested them myself; Here are a few other matches; |
Would you consider providing this as an NPM package to add to my dependency list? Not the same as working within Node, but close enough for me not to create a new request =) |
Is it possible to make the snapshot feature available for pre-processing large static JSON files (around 5 - 7mb) that will be used for searching later? I've tried doing it myself but x10 is a dependency of your JSON.toXML function and x10 requires a window object. I understand x10 is used simply to boost speed in processing using web workers but I am having trouble trying to do it myself. Just to clarify, I will be using this JSON data later in the browser and searching using Defiant.js so instead of creating a snapshot every time (it's the same file) I'd rather have it pre-optimized. |
@hbi99 Defiant.js does NOT depend on any browser-related libraries, right? |
Hi @anhldbk, @yehya, It is the right way to do...creating your snapshot once, instead of every time. In your scenario, the creation of the snapshot might take some time and this is where x10 does its magic. It can create the snapshot in the "background", hence not blocking UI thread....meaning that you parallell script run meanwhile the snapshot is being created. Cheers |
damn. need it for npm |
Needs to be on npm - these days, if it is not on npm, it does not exist. |
This would be sooo nice! @hbi99 Did you have any progress on making it node-compatible? |
Hey any progress @hbi99 |
I am experimenting with some approaches |
So I tried with JSONPath plugin (https://www.npmjs.com/package/JSONPath), the filtering is alright but I'm trying to process 4mb json data so I would need the snapshot method, that you provide in defiant.js. So please keep me in loop |
@UtsavPreet - not sure how to interpret your request; do you mean that the snapshot feature of defiant.js should be added to JSONPath library? JSONPath library approaches this problem in totally different way and it is not appealing to me to contribute to that library since I do not find that solution to be elegant. The reason I've chosen to wait making defiant.js be available in Node env is because it would've required a dependency list that I personally didn't want to get behind of. Though I've just started testing Chromium + Puppeteer with defiant.js will try to offer a solution with a minimum dependencies. I'll get back to this issue/comments when I'm solved it according to my initial requirements list. |
Update: I have made it to work in Node env but current solution requires promises and the code is a bit quirky but I'll clean it up and commit a feature branch for you to test as soon as I can. |
@hbi99 so I meant that I was using JSONPath library in my nodejs app, but since my data set is very large I would want the snapshot feature + search that your library provides, in short waiting for your npm package. I'll check out the latest version and then come back to you if there are any issues. |
@Snoop @Mctalian @iojohnso @ehorodyski @yehya Can I have your opinions; In the first version of this library, Defiant extended the JSON object, enabling such interface: var result = JSON.search(data, '//*'); In the upcoming version (v2.0) and in Node environment, it will be possible to do the same thing like this: var result = await JSON.search(data, '//*'); Now, my question is this; should continue to extend the JSON object or just export an object that exposes the search method?...like this: var defiant = require('defiant.js');
var result = await defiant.search(data, '//*'); |
@hbi99 is there a version of nodejs which i can take look. ? For Nodejs my suggestion will be
|
Hi @tomalex0,
I will put the solution on a feature branch later this evening.
/hbi
…Sent from my iPhone
On 11 Jun 2018, at 09:23, Thomas Alexander <notifications@github.com<mailto:notifications@github.com>> wrote:
@hbi99<https://github.com/hbi99> is there a version of nodejs which i can take look. ?
For Nodejs my suggestion will be
var defiant = require('defiant.js');
var result = await defiant.search(data, '//*');
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#4 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAm7H6sIOYKyxO8jMeYZpRUCNwcqgvkNks5t7hrpgaJpZM4BbIu2>.
|
Hello again @tomalex0, Having said that...here is the instructions. Install the alpha version like this:
Once installed, you can create a "test.js" with this content: var defiant = require('defiant.js');
var data = {
"store": {
"book": [
{
"title": "Sword of Honour",
"category": "fiction",
"author": "Evelyn Waugh",
"@price": 12.99
},
{
"title": "Moby Dick",
"category": "fiction",
"author": "Herman Melville",
"isbn": "0-553-21311-3",
"@price": 8.99
},
{
"title": "Sayings of the Century",
"category": "reference",
"author": "Nigel Rees",
"@price": 8.95
},
{
"title": "The Lord of the Rings",
"category": "fiction",
"author": "J. R. R. Tolkien",
"isbn": "0-395-19395-8",
"@price": 22.99
}
],
"bicycle": {
"brand": "Cannondale",
"color": "red",
"@price": 19.95
}
}
};
async function test_search() {
var result = await defiant.search(data, '//book[position() <= 2]');
console.log(result);
}
test_search(); After that, execute your test-script with |
Awesome @hbi99 , thanks a lot for the quick response, will give it a try and let you know |
Hi, what's the status of this? Any progress with node? |
To all...I've uploaded a first version with node-support. Documentation is coming. Cheers |
This works fine so far, but my process never exits. I'm working on a CLI and if I call await defiant.search(), it returns the results, but the process just keeps running. Without defiant, the process exits normally. Is this bug? Or do I have to call something explicitly after the query? |
@hbi99 Please see my comment above I also tried the example code you posted, same result. |
Its not a bug - here is how v2 works: I think this setup is viable option - unless anyone else has a better idea. Meanwhile - I've added a few other features that is not committed to master branch. And that is the ability to make facet search on JSON structure. Which I think might be of interest :-) |
Thanks for the response! OK now I see why this is happening. Great news about he new features! :) Cool! When can we expect these changes to be published? |
Yeah..the "slowness" is due to the initial procedure of Puppeteer. To overcome it, I've been experimenting with the following set up; there is a new method named const snapshot_id = await defiant.create_snapshot(data)
// returns something like this: snapshot_1543686684427
const test = await defiant.search(snapshot_id, '//book[position() <= 2]')
// returns array with matches Now the good news is that this unique identifier can be used when making the facet search as well. In such scenario, making a facet search will look like this: await defiant.get_facets(snapshot, {
'authors': {group: 'author', key: 'name'},
'categories': {group: 'category', key: 'name'}
});
// returns an object like this
{
authors: {
"Arthus C. Clarke": 31,
"J. R. R. Tolkien": 7,
...
},
categories: {
"Science Fiction": 15,
"Fantasy": 26,
....
}
} ...regardless of the depths of the dataset - as long as they are recurring per book. |
Regarding the exit code - I'm seriously asking for suggestions. That thing bugs me as well so I really appreciate suggestions on how to solve it better :-) I'll close this as issue but welcome new issues instead of this long thread. Cheers |
@hbi99 For example: defiant.init({exitOnFinish: true}) I mean, I'm new to this lib, but it seems to me that in the typical NodeJS app it's necessary to init the page only once. This could be done at app startup time, for example. And then the page can be reused as many times as needed. If this is correct and we have no reason to recreate the page, the default behaviour could be to exit normally when the app's job is finished. And if developers want to change this behaviour, they could pass a flag to init() or something. |
@gtamas |
No description provided.
The text was updated successfully, but these errors were encountered: