Public API

[wrought]

Lots of options for this.

At the basic layer, you can let folks connect directly to the Neo4J database REST interface, but be very wary of security (sub-optimal, especially since I believe user data is stored in the same database): http://docs.neo4j.org/chunked/stable/security-server.html

At a more complex layer, a secondary app can sit on top of the database, such as a flask app, as simple as this: https://gist.github.com/coolgarifTech/5671071 Here's the full tutorial: http://www.coolgarif.com/brain-food/flask-api-over-neo4j-in-python

There are other driver options, so an app can actually be written in anything: http://www.neo4j.org/develop/drivers

Finally, while hitting the live database would give you real-time data, there may be advantages to having a redundant database exposed to the public.

[wrought]

Seems to be the appropriate file describing the data model: https://github.com/CyberCRI/KnowNodes/blob/master/DB/knownodeDB.js

[wrought]

Prototype: https://github.com/wrought/knownodes-public-api/

Will deploy to Heroku, but will need to connect to some neo4j database, also needs to be made aware of knownodes data model.

[couzic]

Hi Matt, great to have you on board !

Actually, one could argue that we already have a public API (sort of)
The web app's overall architecture is based on the decoupling of the client and the server components using a REST API. So our server provides the REST endpoint, while our client is a single-page JavaScript application which gets all the JSON data it needs from AJAX calls to the server.

So in a way we already have a public API. Here are a few examples of HTTP requests it supports :

• http://www.knownodes.com/resources/caae66e5-5b56-461b-8686-5e8f9d8046a6
• http://www.knownodes.com/connections/de2cc403-9a52-4e7b-830a-0efa0a1a4b0c
• http://www.knownodes.com/resources/Biology/searchByKeyword

Our current problem right now is that the API is too permissive. There is no Authorization system in place, and except a few measures to prevent data inconsistencies, nothing prevents a third party component (or someone making manual HTTP requests) to mess with the data using the API .

In general, I am against duplicating functionally similar components. It would also duplicate the effort in maintenance. In this case, what we want is a public API which is a subset of the full API. It makes sense then to use a single API with several authorization levels, the lowest one being "Public".

What do you think ?

[wrought]

Aha, I was wondering if you exposed the rest interface to arbitrary client requests

If I understand correctly from reading some Neo4J documentation, the desirable secure architecture is such that requests on the database are made from listed IPs on the server side and handled by (in this case) a node application. So, the Neo4J rest interface is not exposed, but all application requests are made client-side then handled through node?

Either way I see how that seems sub-optimal in terms of security, would be ideal to provide some auth, especially for POST abilities. The advantage of the distinct app was a fast way to provide sufficient control to maintain a programmatic service treating the application as a black box--meaning no knowledge required of the knownodes codebase besides the data model and some desired queries. On the other hand, providing a built-in API service makes a lot of sense to me as well, both in terms of simplicity and maintenance.

I will see what I can do to get a local version of the knownodes app running. Is there any sample or fixture data easily available? If not, I can generate something, but would be handy to have more realistic data.

[couzic]

Well, I guess it's a question of perspective. On the short term, it would definitely be faster for you to create a simple and stand-alone API. However, on the long run, maintaining a single API would be a win.

Setting up a development environment can be tricky, especially if your OS is Windows. But you can give it a try, there's even a minimal "Getting Started" page in the wiki. If you want to run the app locally, you'll need a couple of additional files that are not stored in Git (the files that contain the passwords). I'll send them to you directly by mail.

We have two separate Neo4j instances, one for production and one for development. You can mess with the development one as much as you want, however keep in mind that there might have a few minor data inconsistencies here and there.

And when you finally have a running app, then you'll have to try and find your way in the code ! :)

Finally, I have Postman collection of HTTP requests supported by the API, if you need it.

[wrought]

Hey all, poking about here now, got the local instance running.

Perhaps we should open another ticket, but it's worth investigating now the use of a Distributed Data Protocol (DDP) over a REST-ish interface. Check out the spec from Meteor https://github.com/meteor/meteor/blob/master/packages/livedata/DDP.md as well as the high-level blog post http://www.meteor.com/blog/2012/03/21/introducing-ddp. You can see PeerLibrary is using this in their public API https://github.com/peerlibrary/peerlibrary/wiki/Public-API. This seems to be the preferred node.js module https://github.com/oortcloud/node-ddp-client though there is another (hasn't been updated in a year) https://github.com/alansikora/node-ddpclient.

Thoughts about DDP vs REST vs Whatever?

I'm planning to hack and see what happens, learn my way around the knownodes codebase, seeking to get an integrated solution to eliminate external dependencies.

[wrought]

FYI notes about reason to use DDP pub/sub instead of REST from @mitar in #sustainathon on freenode:

15:04 < Mitar> so the arguments for DDP are that it provides real-time push from the server to client
15:04 < Mitar> it is a pub/sub protocol, not state-less protocol like HTTP
15:04 < Mitar> so in a pub/sub protocol server keeps state what data client has and when there is new or updated data for the query client is interested in, server sends data to the client

[wrought]

After discussing further, DDP is too experimental, despite intense enthusiasm and conviction from @mitar. Created this ticket instead: #154

As for this API, there are some things to consider:

• REST makes sense
• Already using urls to view content--harmonize with programmatic access?
  • e.g. right now URLs use hashes http://localhost:3000/map/f767cf9a-9ce3-4ff2-8c48-250445b99a90 but they could use the plain text node name (and/or ID)
• Expose only public data (e.g. no user data, passwords, etc)
• Allow POST requests
• Consider naive authentication, in order to prevent accidental or otherwise DOS
  • Or, consider auth for POST
  • Or, consider no auth

[mitar]

Don't forget PATCH.

[couzic]

Also, one thing that I find pretty cumbersome right now is the endpoint URLs.

For example, to get a specific user, we make an HTTP GET request to http://www.knownodes.com/users/36927e96-b698-4d14-9130-6483337f6911

Whereas the web app has a screen to display the information relative to a specific user, at http://www.knownodes.com/user/36927e96-b698-4d14-9130-6483337f6911

That is quite unfortunate. Both URLs are extremely similar, and it's very easy to forget the 's' and run into weird behavior.

A better URL for the endpoints would be something like http://www.knownodes.com/api/v1/users/36927e96-b698-4d14-9130-6483337f6911

That way, it would be wether we try to reach the API or the web app. Also, versioning the API is very useful.

[mitar]

@micouz: Wouldn't it be even better if screen would be displayed at http://www.knownodes.com/user/36927e96-b698-4d14-9130-6483337f6911, and API would be at http://www.knownodes.com/user/36927e96-b698-4d14-9130-6483337f6911.json?

[wrought]

Wanted to post some philosophy / food-for-thought about making your API visible, browseable, and human-readable: http://django-rest-framework.org/topics/browsable-api

In a way, the knownodes node.js app provides this, but it could be honed, harmonized, and extended to provide parallel programmatic access (based on my current understanding of the app)

[mitar]

But you want even more than just browseable endpoints, no? Wasn't the idea that human-readable interface is simply normal Knownodes interface, but you can then access it through API as well, at same or very similar URL?

[mitar]

Maybe it could be very same URL, just different HTTP accept header.

[mitar]

Should data be available as JSON-LD? A w3c standard?