Use library-based APIs instead of RESTful APIs

[frbattid]

Instead of using RESTful APIs, library-based APIs could be used to increase the performance. At least, that could be the case of HDFS, i.e. use the Hadoop API instead of WebHDFS.

In the case of MySQL, the mysqlconnector seems to be the only option... at least, calls will be made localy instead of remotely. The question is... is there a CKAN API?

The idea behind this proposal is to move Cygnus deployment from Orion side to the storage side, avoiding HTTP-based communications.

[fgalan]

There is document named "The CKAN API" (note the "The" :) at http://docs.ckan.org/en/ckan-2.2/api.html, but it seems that is the old REST API we already know, so it is not the API kind you are looking for.

CKAN relies in Postgres storate. If the way in which CKAN stores information in that DB is documented, maybe a JDBC-like interface may be use (thus avoiding HTTP-based comunication and using a wiring protocol suposed to be efficient).

[frbattid]

Good point.

Anyway, this is only a proposal that would restrict the deployment of Cygnus, in the sense that would only work if, for instance, the MySQL database and the Namenode run in the same machine (asuming you want both types of persistence).

There are other ideas on the table, like moving from text protocols (HTTP) to a binary ones.

[frbattid]

Issue created regarding HDFS: #537

[frbattid]

Once this has been created for HDFS, it seems the issue can be close since:

• OrionHDFSSink has available both binary and REST backends.
• OrionMySQLSink is already based on JDBC driver, a binary approach.
• OrionMongoSink and OrionSTHSink are already based on Mongo driver, a binary approach.
• OrionKafkaSink is already based on a Java-based client, a binary approach.
• It seems there is no binary approach for OrionCKANSink.
• Future sinks such as OrionPostgreSQLSink and OrionDynamoSink will be based on binary approaches.