compatibility with browserify

[elf-pavlik]

seams like at present we can't use this package with browserify 😢

more details in levelgraph/levelgraph-jsonld#3 (comment)

@mcollina also suggests some possible solutions on thread mentioned above 😄

[dlongley]

cc: @davidlehn

[elf-pavlik]

@davidlehn i plan to work in next weeks with JSON-LD and RDFa in browser environment... will try to investigate this issue as well!

[mcollina]

👍

[davidlehn]

The quick fix of omitting jsdom and rdfa support works at a basic level. Of course if you do that then you lose RDFa support in the jsonld.js "request" extension.

Browserify is being aggressive with the jsonld.js code. It pulls in every dependency it sees. Even without jsdom and rdfa you end up with 334k minified js! I'm guessing much of that is not needed. If the environment detection support works after browserification, the code could just use the document loader XHR browser support rather than all the libraries needed for node support.

The jsonld.js "request" extension is just node only right now. I think a similar version optimized for browsers could be made. It would use XHR to avoid the "request" library and all its dependencies. It could probably use native browser DOM features rather than bothering with jsdom. And the stdin support is not really that useful in a browser? The RDFa parsing library would be used by both node and browser versions.

I think the way forward is to start by making sure environment detection is working in node, with regular browser use, and with browserify use. Then block browserify from using node libraries like http, request, jsdom, etc and just depend on native browser features. Then refactor the "request" extension to adapt to the environment and use XHR and native DOM rather than request and jsdom libraries.

That should let jsonld.js and the request extension work in all three environments and be a bit optimized too.

Thoughts?

[mcollina]

Browserify is being aggressive with the jsonld.js code. It pulls in every dependency it sees. Even without jsdom and rdfa you end up with 334k minified js! I'm guessing much of that is not needed. If the environment detection support works after browserification, the code could just use the document loader XHR browser support rather than all the libraries needed for node support.

That's the point of browserify, it is aggressive so every node-only module works out-of-the-box, but the fact that jsonld.js is both node and browser confuses browserify.

The jsonld.js "request" extension is just node only right now. I think a similar version optimized for browsers could be made. It would use XHR to avoid the "request" library and all its dependencies. It could probably use native browser DOM features rather than bothering with jsdom. And the stdin support is not really that useful in a browser? The RDFa parsing library would be used by both node and browser versions.

Exactly.

Another solution is to better modularize the code, and do a 'release' for the browser by creating a browserify bundle. This allows to better modularize the code into multiple files, and then to use browserify 'browser' section to provide the browser variants.

Anyway, 👍 for better node, browser and browserify support :).

[elf-pavlik]

thank you @davidlehn for following up on this issue!

IMO one should have possibility to use jsonld.js both in browser and in node without using anything related to RDFa, Turtle, RDF/XML or Microdata. Especially when used as dependency in their projects which may already handle other serializations using some other tools.

I'll try to take a closer look at how jsonld.js currently work sometimes this/next week. Do you have by any chance some kind of conceptual diagram, like for example one at: http://viejs.org ? I hope to come up with some suggestions on more modular setup so everyone can use only what they really need...

I didn't look yet on how you extend request, but I've seen people using various alternatives like:

• https://github.com/visionmedia/superagent
• https://github.com/substack/hyperquest
• https://github.com/ded/reqwest
• http://api.jquery.com/category/ajax/
• etc.

So I would like to take a look at possibility of staying agnostic about how one handles requests :)

[mcollina]

I'm 👍 for using superagent as it works wonderfully both on node and on the browser.

A modular setup might be very welcome. Doing separate projects will also help adoption.

[davidlehn]

@mcollina Yeah, I understand why browserify is pulling in everything, but it seems rather heavyweight depending on what you need. I'm pretty sure the environment detection part is needed no matter what. But if you were just using the basic JSON-LD processing, there is little need for the node and "request" extension support. If you want to do things like process RDFa, then at least part of the "request" extension is needed including something to use browser DOM vs jsdom. If you are using more specialized things like passing "request" lib params to the "request" extension, then you likely need to pull in the kitchen sink, or much of it. I have no idea how to make all these options easy to use. The solution will probably involve splitting up functionality into files that can be included, excluded, or substituted as needed.

@elf-pavlik The support to handle other formats is all in the "request" extension right now. It really just handles RDFa and JSON but the theory is it could be expanded with more type negotiation and additional parsers. We don't have diagrams yet, sorry. Until now it hasn't been complex enough to need diagrams.

I know there are lots of other HTTP request libraries. At the time, one reason I choose to use "request" because it was the easiest to add support for the draft HTTP Signature spec via the related node-http-signature lib. Adding an abstraction layer was overkill. I did try to make everything work such that request lib params are passed via a specific named options object but I think the return value may be request lib specific. Making a generic one-request-API-to-rule-them-all layer one be better done in another project. I'd rather not tackle that sort of problem myself but would be happy to use it. :-)

[elf-pavlik]

@davidlehn i'll do my best to read through all the code this weekend or early next week and then start working on a pull request!

would you see possibility of shifting jsonld.js focus exclusively on JSON-LD and create dedicated project for more of a high level tool which mixes in RDFa, HTTP Signature etc? i could also check what @webr3 has in mind with https://github.com/webr3/rdf.js

i assume that whatever we do we can't break https://github.com/digitalbazaar/payswarm.js ;)
BTW wouldn't HTTP Signatures fit there better? (blind shooting ATM, please feel free to ignore!)

luckily it looks like this package still doesn't have that many dependents: https://npmjs.org/package/jsonld

[davidlehn]

If it makes sense to split out things like the "request" extension into a new package, that's fine. It was just easier to add to the main package at the time. That won't fix all the problems, and just moves some of them to the new package, but might help for some use cases.

I haven't thought about the HTTP Signature issue too much. It needs the method, path, and get/set header ability and maybe to deal with some gotchas with things like the Date header. The current system using the "request" lib is pretty nice since you just pass in your signing info for any request and the lib figures it out for you.

There are more users of the package than just the npm dependents list. It would certainly be nice not to break payswarm.js too much but that code can be updated as needed.

[elf-pavlik]

without diving into code i guess jsonld.js needs to make requests to fetch external contexts?
in a browser it might get tricky if someone doesn't http://enable-cors.org , maybe yet another reason to leave it to dependent package to provide mechanism for making requests... one could then deal with CORS issues by using JSONP, falling back to proxy or whatever fits given case!

pulling out RDFa sounds even more attractive to me and possibly more straight forward...

i will ask folks participating in http://www.w3.org/community/rdfjs for their feedback and maybe I better read the code before discussing it further 📖

thank you @davidlehn for all the clarifications, same @mcollina for all the creative suggestions 👍

[davidlehn]

Yes, the main file has XHR and node support for loading contexts. In theory, a browserified file could just use the XHR support too rather than the node support. Which is why some experimentation is needed with the environment detection. The "request" extension is headed in the direction of just giving it a URL and it figures out type negotiation, parsing, and extraction details to give you back the JSON-LD for that resource. More work is needed but that was the theory.

[dlongley]

Issues #29 and #31 should be looked over when considering any updates to how requests work. After the "request" extension was written, better support was added for external document loading to the main jsonld.js file. This support has not yet been integrated into (or used to replace parts of) the "request" extension. The jsonld.js external document loader API should probably be used in any rewrite/fixes.

[elf-pavlik]

As first step I would like to pull out all the RDFa related code somewhere else with only necessary changes to the way of handling requests. What do you think about creating another repository which would use this one as dependency and recreating there all the functionality provided currently here by this JSON-LD + RDFa love affair? 😉