Adapt DefiantJS for Node enviroment

[hbi99]

No description provided.

[Snoop]

It would be great, just tried to run it with node.js => window is not defined

[hbi99]

In order to make this work, suitable dependencies are needed. I've looking in to jsdom, xmldom, etc. Though some of these libs opens unexpected can of worms that requires some adjustments to those libs. Which is not desirable...I'll continue dissecting the problem. Meanwhile, suggestions are welcome

[Snoop]

you are a hard worker :)

[Snoop]

Hello hbi, is there any news on the Node implementation ?
Kind regards!

[hbi99]

I have a few private project which I juggle between and currently I'm working on writing Longscript, desktop application, with which one can produce animations like these; http://longscript.com/gallery.php

[Mctalian]

Has there been any additional effort on this? I can see a potential use case for my project.

[hbi99]

@Mctalian Yeah...a little bit. Though, I realized fast that this "path" requires dependencies and I'm looking for a way to do it with fewest dependencies.

[Mctalian]

@hbi99 I see. The main thing I've noticed is that utilizing the Snapshot feature causes my webpage to lock up for 0.5s to 2.0s depending. I figured if this was adapted for Node it would be non-blocking as well, and may assist in this area. This is a minor thing. I'm more curious about being able to move some things server-side for some enhancements I've thought of.

Not sure how much I can help, but if there's anything I can do, let me know.

[hbi99]

Ok...thanks. Will do 👍

0.5-2 seconds...it sounds like you have a large JSON file...how big is it?

[Mctalian]

Pretty big. I think there's something like 430 nodes total. And it's only going to get bigger over time!

[hbi99]

@Mctalian I might have a solution for improving the performance even more...and cut down the time it takes to create a snapshot. I'll get back regarding this.

Thanks for the JSON data and feedback

[Mctalian]

I will make a new issue for ya.

[iojohnso]

I'm also looking for this. Do you happen to know if there are any existing alternatives out there that provide same/similar functionality for NodeJS?

(also, great job on this project)

[hbi99]

@iojohnso
Sorry for the delayed response....and thanks

I think there are a few repo's doing similar stuff, even though I haven't tested them myself;
https://github.com/ajaxorg/node-o3-xml

Here are a few other matches;
https://github.com/search?o=desc&q=xpath+node&ref=searchresults&s=stars&type=Repositories&utf8=%E2%9C%93

[ehorodyski]

Would you consider providing this as an NPM package to add to my dependency list? Not the same as working within Node, but close enough for me not to create a new request =)

[yehya]

Is it possible to make the snapshot feature available for pre-processing large static JSON files (around 5 - 7mb) that will be used for searching later? I've tried doing it myself but x10 is a dependency of your JSON.toXML function and x10 requires a window object. I understand x10 is used simply to boost speed in processing using web workers but I am having trouble trying to do it myself.

Just to clarify, I will be using this JSON data later in the browser and searching using Defiant.js so instead of creating a snapshot every time (it's the same file) I'd rather have it pre-optimized.

[anhldbk]

@hbi99 Defiant.js does NOT depend on any browser-related libraries, right?

[hbi99]

Hi @anhldbk,
It does not depend on any browser related library...though it does depend on the browser environment.

@yehya,
See my answer above, also regarding the dependency of x10 - this is also my library and I separated these two libraries because the functionality they both bring are just too different. Beside the speed, x10 brings a simple interface to web workers (IMHO, simpler than other libraries).

It is the right way to do...creating your snapshot once, instead of every time. In your scenario, the creation of the snapshot might take some time and this is where x10 does its magic. It can create the snapshot in the "background", hence not blocking UI thread....meaning that you parallell script run meanwhile the snapshot is being created.

Cheers

[askucher]

damn. need it for npm

[mmc41]

Needs to be on npm - these days, if it is not on npm, it does not exist.

[devdavidkarlsson]

This would be sooo nice! @hbi99 Did you have any progress on making it node-compatible?
Thanks...

[UtsavPreet]

Hey any progress @hbi99
or any possible alternatives ?

[hbi99]

I am experimenting with some approaches

[UtsavPreet]

So I tried with JSONPath plugin (https://www.npmjs.com/package/JSONPath), the filtering is alright but I'm trying to process 4mb json data so I would need the snapshot method, that you provide in defiant.js. So please keep me in loop
Thanks

[hbi99]

@UtsavPreet - not sure how to interpret your request; do you mean that the snapshot feature of defiant.js should be added to JSONPath library? JSONPath library approaches this problem in totally different way and it is not appealing to me to contribute to that library since I do not find that solution to be elegant.
Aside from that, JSONPath is a library runnable in Node environment - defiant.js is not (currently)

The reason I've chosen to wait making defiant.js be available in Node env is because it would've required a dependency list that I personally didn't want to get behind of. Though I've just started testing Chromium + Puppeteer with defiant.js will try to offer a solution with a minimum dependencies. I'll get back to this issue/comments when I'm solved it according to my initial requirements list.

[hbi99]

Update: I have made it to work in Node env but current solution requires promises and the code is a bit quirky but I'll clean it up and commit a feature branch for you to test as soon as I can.

[UtsavPreet]

@hbi99 so I meant that I was using JSONPath library in my nodejs app, but since my data set is very large I would want the snapshot feature + search that your library provides, in short waiting for your npm package.

I'll check out the latest version and then come back to you if there are any issues.
Thank You so much :)

[hbi99]

@Snoop @Mctalian @iojohnso @ehorodyski @yehya

Can I have your opinions; In the first version of this library, Defiant extended the JSON object, enabling such interface:

var result = JSON.search(data, '//*');

In the upcoming version (v2.0) and in Node environment, it will be possible to do the same thing like this:

var result = await JSON.search(data, '//*');

Now, my question is this; should continue to extend the JSON object or just export an object that exposes the search method?...like this:

var defiant = require('defiant.js');

var result = await defiant.search(data, '//*');

[tomalex0]

@hbi99 is there a version of nodejs which i can take look. ?

For Nodejs my suggestion will be

var defiant = require('defiant.js');

var result = await defiant.search(data, '//*');

[hbi99]

Hi @tomalex0, I will put the solution on a feature branch later this evening. /hbi

…

Sent from my iPhone On 11 Jun 2018, at 09:23, Thomas Alexander <notifications@github.com<mailto:notifications@github.com>> wrote: @hbi99<https://github.com/hbi99> is there a version of nodejs which i can take look. ? For Nodejs my suggestion will be var defiant = require('defiant.js'); var result = await defiant.search(data, '//*'); — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAm7H6sIOYKyxO8jMeYZpRUCNwcqgvkNks5t7hrpgaJpZM4BbIu2>.

[hbi99]

Hello again @tomalex0,
First of all, I want to declare that I have intentions to make the upcoming version 2.0.0 to be much more powerful than right now. This version is really alpha with emphasis on alpha. With this version I am hoping to get feedback as well as make optimisations along the way. I've also worked with node version 9.2.0 - if you don't havenvm, please install it from here:
https://github.com/creationix/nvm

Having said that...here is the instructions. Install the alpha version like this:

npm install defiant.js@alpha

Once installed, you can create a "test.js" with this content:

var defiant = require('defiant.js');

var data = {
   "store": {
      "book": [
         {
            "title": "Sword of Honour",
            "category": "fiction",
            "author": "Evelyn Waugh",
            "@price": 12.99
         },
         {
            "title": "Moby Dick",
            "category": "fiction",
            "author": "Herman Melville",
            "isbn": "0-553-21311-3",
            "@price": 8.99
         },
         {
            "title": "Sayings of the Century",
            "category": "reference",
            "author": "Nigel Rees",
            "@price": 8.95
         },
         {
            "title": "The Lord of the Rings",
            "category": "fiction",
            "author": "J. R. R. Tolkien",
            "isbn": "0-395-19395-8",
            "@price": 22.99
         }
      ],
      "bicycle": {
         "brand": "Cannondale",
         "color": "red",
         "@price": 19.95
      }
   }
};

async function test_search() {
	var result = await defiant.search(data, '//book[position() <= 2]');
	console.log(result);
}

test_search();

After that, execute your test-script with node test.js

[tomalex0]

Awesome @hbi99 , thanks a lot for the quick response, will give it a try and let you know

[gtamas]

Hi, what's the status of this? Any progress with node?

[hbi99]

To all...I've uploaded a first version with node-support. Documentation is coming.
Please use with caution and test - I think I will add functionality and fix bugs - and mostly, I want to make performance improvements nearest future.

Cheers

[gtamas]

This works fine so far, but my process never exits. I'm working on a CLI and if I call await defiant.search(), it returns the results, but the process just keeps running.

Without defiant, the process exits normally.

Is this bug? Or do I have to call something explicitly after the query?

[gtamas]

@hbi99 Please see my comment above

I also tried the example code you posted, same result.

[hbi99]

Its not a bug - here is how v2 works:
The Defiant library is actually running in a browser page created with Puppeteer. To set up the page, Puppeteer needs to prepare, create "page" and evaluate Defiant library. Now this is a costly procedure and to avoid doing this every time Defiant is run, I am not "destroying" the page once Defiant has done its job. Instead, I keep the page alive for next time (if ever) Defiant is used. This is why the exit code is not "delivered".

I think this setup is viable option - unless anyone else has a better idea.

Meanwhile - I've added a few other features that is not committed to master branch. And that is the ability to make facet search on JSON structure. Which I think might be of interest :-)
Its actually fast also - on my computer it facets JSON data of less than 1MB in matters of a few hundreds of milliseconds. Larger files are facetted approximately 650ms / 1 MB

[gtamas]

Thanks for the response! OK now I see why this is happening.

Great news about he new features! :) Cool! When can we expect these changes to be published?
This would be nice indeed. The current implementation works, but it runs very slowly on my machine.

[hbi99]

Yeah..the "slowness" is due to the initial procedure of Puppeteer. To overcome it, I've been experimenting with the following set up; there is a new method named create_snapshot, which returns unique identifier (string). When making multiple searches on same dataset, it's possible to use this string to tell which snapshot to make searches on. Of course, this assumes that you are using the same dataset to make searches.

const snapshot_id = await defiant.create_snapshot(data)
// returns something like this: snapshot_1543686684427
  
const test = await defiant.search(snapshot_id, '//book[position() <= 2]')
// returns array with matches

Now the good news is that this unique identifier can be used when making the facet search as well. In such scenario, making a facet search will look like this:

await defiant.get_facets(snapshot, {
	'authors': {group: 'author', key: 'name'},
	'categories': {group: 'category', key: 'name'}
});

// returns an object like this
{
	authors: {
		"Arthus C. Clarke": 31,
		"J. R. R. Tolkien": 7,
		...
	},
	categories: {
		"Science Fiction": 15,
		"Fantasy": 26,
		....
	}
}

...regardless of the depths of the dataset - as long as they are recurring per book.

[hbi99]

Regarding the exit code - I'm seriously asking for suggestions. That thing bugs me as well so I really appreciate suggestions on how to solve it better :-)

I'll close this as issue but welcome new issues instead of this long thread.

Cheers

[gtamas]

@hbi99
Regarding the exit code: it's true that page init is indeed a very expensive call. Takes 1-2 second on my machine.
However maybe it would be better to make this behaviour optional by adding some kind of flag allowing people to enable / disable it.

For example: defiant.init({exitOnFinish: true})

I mean, I'm new to this lib, but it seems to me that in the typical NodeJS app it's necessary to init the page only once. This could be done at app startup time, for example. And then the page can be reused as many times as needed.

If this is correct and we have no reason to recreate the page, the default behaviour could be to exit normally when the app's job is finished. And if developers want to change this behaviour, they could pass a flag to init() or something.

[hbi99]

@gtamas
I like this idea a lot - I'll implement this very soon.
Thanks for the idea