ngsi2cosmos management API for FI-LAB users

[fgalan]

Currently, the ngsi2cosmos is configured by FI-LAB staff in a "static" way. So if a FI-LAB user wants to store the data he/she publishes at CB to Cosmos (as happened during Santander hackaton), he/she has to talk with FI-LAB staff to do that configuration. This is a quite unflexible and not scalable approach.

Thus, the ngs2cosmos component should provide an API so users can configure CB->Cosmos rules. The processing of the request in this API will involve all the underlying actions in CB, Cosmos and ngsi2cosmos components:

• In Cosmos: creating a dataset for the user
• In CB: set up the proper subscribeContext for the user entities
• In ngsi2cosmos: configure the logic so the notifiyContext corresponding to the above subscription goes at the end to Cosmos through its HttpFs or WebHDFS API.

[frbattid]

Trying to elaborate a bit more on the tasks related to Cosmos:

1. Does the user exist in Cosmos?
1.1. If not existing then:
- Create a user in the master node local OS. Username and password must be provided by the user.
- Create a home directory with the propper rights in HDFS (/user/newuserdir)
- Go to 2.
1.2. If existing then:
- Go to 2.
2. Is the dataset public?
2.1. If it is public then:
- Create a directory for the dataset under /user/opendata
- Go to 3.
2.2. If it is not public then:
- Create a directoy for the dataset under /user/newuserdir
- Go to 3.
3. Create a Hive table for the new dataset.

[frbattid]

When dealing with Cosmos Platform (Cosmos v0.x | x > 9), the above algorythm must change. This is because each user in the Platform will have its own private cluster, i.e. there will not be a shared cluster. Implications are:

• It is not necessary to check if the user exists in the shared cluster. ngsi2cosmos only needs to know the entry point (IP address of the Head Node of the cluster, HDFS path, other).
• There is no need to decide if the data will be public or private. Inherently, all the data stored in a private cluster will be private. If the data is aimed to be open data, then it must be stored in a public cluster.

In fact, the above algorithm is summarized as "create a Hive table for the new dataset" :)

[fgalan]

The part related strictly with management API has been moved to telefonicaid/fiware-connectors#12.

The parts related with Hive/HDFS directory creation were solved at the end in ngsi2cosmos, but the need to be also implemented in Cygnus. Thus, a new issue its created in its repository, in particular: telefonicaid/fiware-connectors#13.