Skip to content
Branch: master
Find file History
G10h4ck More fixes in JSON API
Schedule delayed async session closing properly (without recurrence)
Fix double capture of this in async method registering, that was causing
  compilation error for Android
Fix minor compiler warning in jsonapi-generator
Latest commit 2d7b855 Aug 29, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
src More fixes in JSON API Aug 28, 2019
README.adoc Implement JSON API HTTP Basic authentication Sep 19, 2018

README.adoc

RetroShare JSON API

How to use RetroShare JSON API

Look for methods marked with @jsonapi doxygen custom command into libretroshare/src/retroshare. The method path is composed by service instance pointer name like rsGxsChannels for RsGxsChannels, and the method name like createGroup and pass the input paramethers as a JSON object.

Service instance pointer in rsgxschannels.h
/**
 * Pointer to global instance of RsGxsChannels service implementation
 * @jsonapi{development}
 */
extern RsGxsChannels* rsGxsChannels;
Method declaration in rsgxschannels.h
	/**
	 * @brief Request channel creation.
	 * The action is performed asyncronously, so it could fail in a subsequent
	 * phase even after returning true.
	 * @jsonapi{development}
	 * @param[out] token Storage for RsTokenService token to track request
	 * status.
	 * @param[in] group Channel data (name, description...)
	 * @return false on error, true otherwise
	 */
	virtual bool createGroup(uint32_t& token, RsGxsChannelGroup& group) = 0;
paramethers.json
{
    "group":{
        "mMeta":{
            "mGroupName":"JSON test group",
            "mGroupFlags":4,
            "mSignFlags":520
        },
        "mDescription":"JSON test group description"
    },
    "caller_data":"Here can go any kind of JSON data (even objects) that the caller want to get back together with the response"
}
Calling the JSON API with curl on the terminal
curl -u $API_USER --data @paramethers.json http://127.0.0.1:9092/rsGxsChannels/createGroup
JSON API call result
{
    "caller_data": "Here can go any kind of JSON data (even objects) that the caller want to get back together with the response",
    "retval": true,
    "token": 3
}

Even if it is less efficient because of URL encoding HTTP GET method is supported too, so in cases where the client cannot use POST she can still use GET taking care of encoding the JSON data. With curl this can be done at least in two different ways.

Calling the JSON API with GET method with curl on the terminal
curl -u $API_USER --get --data-urlencode jsonData@paramethers.json \
	http://127.0.0.1:9092/rsGxsChannels/createGroup

Letting curl do the encoding is much more elegant but it is semantically equivalent to the following.

Calling the JSON API with GET method and pre-encoded data with curl on the terminal
curl -u $API_USER http://127.0.0.1:9092/rsGxsChannels/createGroup?jsonData=%7B%0A%20%20%20%20%22group%22%3A%7B%0A%20%20%20%20%20%20%20%20%22mMeta%22%3A%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupName%22%3A%22JSON%20test%20group%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mGroupFlags%22%3A4%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%22mSignFlags%22%3A520%0A%20%20%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%20%20%22mDescription%22%3A%22JSON%20test%20group%20description%22%0A%20%20%20%20%7D%2C%0A%20%20%20%20%22caller_data%22%3A%22Here%20can%20go%20any%20kind%20of%20JSON%20data%20%28even%20objects%29%20that%20the%20caller%20want%20to%20get%20back%20together%20with%20the%20response%22%0A%7D

Note that using GET method ?jsonData= and then the JSON data URL encoded are added after the path in the HTTP request.

JSON API authentication

Most of JSON API methods require authentication as they give access to RetroShare user data, and we don’t want any application running on the system eventually by other users be able to access private data indiscriminately. JSON API support HTTP Basic as authentication scheme, this is enough as JSON API server is intented for usage on the same system (127.0.0.1) not over an untrusted network. If you need to use JSON API over an untrusted network consider using a reverse proxy with HTTPS such as NGINX in front of JSON API server. If RetroShare login has been effectuated through the JSON API you can use your location SSLID as username and your PGP password as credential for the JSON API, but we suggests you use specific meaningful and human readable credentials for each JSON API client so the human user can have better control over which client can access the JSON API.

NewToken.json
{
	"token": "myNewUser:myNewPassword"
}
An authenticated client can authorize new tokens like this
curl -u $API_USER --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/authorizeToken
An unauthenticated JSON API client can request access with
curl --data @NewToken.json http://127.0.0.1:9092/jsonApiServer/requestNewTokenAutorization

When an unauthenticated client request his token to be authorized, JSON API server will try to ask confirmation to the human user if possible through mNewAccessRequestCallback, if it is not possible or the user didn’t authorized the token false is returned.

Offer new RetroShare services through JSON API

To offer a retroshare service through the JSON API, first of all one need find the global pointer to the service instance and document it in doxygen syntax, plus marking with the custom doxygen command @jsonapi{RS_VERSION} where RS_VERSION is the retroshare version in which this service became available with the current semantic (major changes to the service semantic, changes the meaning of the service itself, so the version should be updated in the documentation in that case).

Service instance pointer in rsgxschannels.h
/**
 * Pointer to global instance of RsGxsChannels service implementation
 * @jsonapi{development}
 */
extern RsGxsChannels* rsGxsChannels;

Once the service instance itself is known to the JSON API you need to document in doxygen syntax and mark with the custom doxygen command @jsonapi{RS_VERSION} the methods of the service that you want to make available through JSON API.

Offering RsGxsChannels::getChannelDownloadDirectory in rsgxschannels.h
	/**
	 * Get download directory for the given channel
	 * @jsonapi{development}
	 * @param[in] channelId id of the channel
	 * @param[out] directory reference to string where to store the path
	 * @return false on error, true otherwise
	 */
	virtual bool getChannelDownloadDirectory( const RsGxsGroupId& channelId,
	                                          std::string& directory ) = 0;

For each paramether you must specify if it is used as input @param[in] as output @param[out] or both @param[inout]. Paramethers and return value types must be of a type supported by RsTypeSerializer which already support most basic types (bool, std::string…​), RsSerializable and containers of them like std::vector<std::string>. Paramethers passed by value and by reference of those types are both supported, while passing by pointer is not supported. If your paramether or return class/struct type is not supported yet by RsTypeSerializer most convenient approach is to make it derive from RsSerializable and implement serial_process method like I did with RsGxsChannelGroup.

Deriving RsGxsChannelGroup from RsSerializable in rsgxschannels.h
struct RsGxsChannelGroup : RsSerializable
{
	RsGroupMetaData mMeta;
	std::string mDescription;
	RsGxsImage mImage;

	bool mAutoDownload;

	/// @see RsSerializable
	virtual void serial_process( RsGenericSerializer::SerializeJob j,
	                             RsGenericSerializer::SerializeContext& ctx )
	{
		RS_SERIAL_PROCESS(mMeta);
		RS_SERIAL_PROCESS(mDescription);
		RS_SERIAL_PROCESS(mImage);
		RS_SERIAL_PROCESS(mAutoDownload);
	}
};

You can do the same recursively for any member of your struct that is not yet supported by RsTypeSerializer.

Some Retroshare C++ API functions are asyncronous, historically RetroShare didn’t follow a policy on how to expose asyncronous API so differents services and some times even differents method of the same service follow differents asyncronous patterns, thus making automatic generation of JSON API wrappers for those methods impractical. Instead of dealing with all those differents patterns I have chosed to support only one new pattern taking advantage of modern C++11 and restbed features. On the C++11 side lambdas and +std::function+s are used, on the restbed side Server Side Events are used to send asyncronous results.

Lets see an example so it will be much esier to understand.

RsGxsChannels::turtleSearchRequest asyncronous API
	/**
	 * @brief Request remote channels search
	 * @jsonapi{development}
	 * @param[in] matchString string to look for in the search
	 * @param multiCallback function that will be called each time a search
	 * result is received
	 * @param[in] maxWait maximum wait time in seconds for search results
	 * @return false on error, true otherwise
	 */
	virtual bool turtleSearchRequest(
	        const std::string& matchString,
	        const std::function<void (const RsGxsGroupSummary& result)>& multiCallback,
	        std::time_t maxWait = 300 ) = 0;

RsGxsChannels::turtleSearchRequest(…​) is an asyncronous method because it send a channel search request on turtle network and then everytime a result is received from the network multiCallback is called and the result is passed as parameter. To be supported by the automatic JSON API wrappers generator an asyncronous method need a parameter of type std::function<void (…​)> called callback if the callback will be called only once or multiCallback if the callback is expected to be called more then once like in this case. A second mandatory parameter is maxWait of type std::time_t it indicates the maximum amount of time in seconds for which the caller is willing to wait for results, in case the timeout is reached the callback will not be called anymore.

Important

callback and multiCallback parameters documentation must not specify [in], [out], [inout], in Doxygen documentation as this would fool the automatic wrapper generator, and ultimately break the compilation.

RsFiles::turtleSearchRequest asyncronous JSON API usage example
$ cat turtle_search.json
{
    "matchString":"linux"
}
$ curl --data @turtle_search.json http://127.0.0.1:9092/rsFiles/turtleSearchRequest
data: {"retval":true}

data: {"results":[{"size":157631,"hash":"69709b4d01025584a8def5cd78ebbd1a3cf3fd05","name":"kill_bill_linux_1024x768.jpg"},{"size":192560,"hash":"000000000000000000009a93e5be8486c496f46c","name":"coffee_box_linux2.jpg"},{"size":455087,"hash":"9a93e5be8486c496f46c00000000000000000000","name":"Linux.png"},{"size":182004,"hash":"e8845280912ebf3779e400000000000000000000","name":"Linux_2_6.png"}]}

data: {"results":[{"size":668,"hash":"e8845280912ebf3779e400000000000000000000","name":"linux.png"},{"size":70,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.txt.sha1sum"},{"size":3076767744,"hash":"e8845280912ebf3779e400000000000000000000","name":"kali-linux-2016.2-amd64.iso"},{"size":2780872,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.bin"},{"size":917504,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.lzma"},{"size":2278404096,"hash":"e8845280912ebf3779e400000000000000000000","name":"gentoo-linux-livedvd-amd64-multilib-20160704.iso"},{"size":151770333,"hash":"e8845280912ebf3779e400000000000000000000","name":"flashtool-0.9.23.0-linux.tar.7z"},{"size":2847372,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.elf"},{"size":1310720,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux.gz"},{"size":987809,"hash":"e8845280912ebf3779e400000000000000000000","name":"openwrt-ar71xx-generic-vmlinux-lzma.elf"}]}

By default JSON API methods requires client authentication and their wrappers are automatically generated by json-api-generator. In some cases methods need do be accessible without authentication such as rsLoginHelper/getLocations so in the doxygen documentaion they have the custom command @jsonapi{RS_VERSION,unauthenticated}. Other methods such as /rsControl/rsGlobalShutDown need special care so they are marked with the custom doxygen command @jsonapi{RS_VERSION,manualwrapper} and their wrappers are not automatically generated but written manually into JsonApiServer::JsonApiServer(…​).

A bit of history

First writings about this

The previous attempt of exposing a RetroShare JSON API is called libresapi and unfortunatley it requires a bunch of boilerplate code when we want to expose something present in the C++ API in the JSON API.

As an example here you can see the libresapi that exposes part of the retroshare chat C++ API and lot of boilerplate code just to convert C++ objects to JSON

To avoid the C++ to JSON and back conversion boilerplate code I have worked out an extension to our C++ serialization code so it is capable to serialize and deserialize to JSON you can see it in this pull request

So first step toward having a good API is to take advantage of the fact that RS is now capable of converting C++ objects from and to JSON.

The current API is accessible via HTTP and unix socket, there is no authentication in both of them, so anyone having access to the HTTP server or to the unix socket can access the API without extra restrictions. Expecially for the HTTP API this is a big risk because also if the http server listen on 127.0.0.1 every application on the machine (even rogue javascript running on your web browser) can access that and for example on android it is not safe at all (because of that I implemented the unix socket access so at least in android API was reasonably safe) because of this.

A second step to improve the API would be to implement some kind of API authentication mechanism (it would be nice that the mechanism is handled at API level and not at transport level so we can use it for any API trasport not just HTTP for example)

The HTTP server used by libresapi is libmicrohttpd server that is very minimal, it doesn’t provide HTTPS nor modern HTTP goodies, like server notifications, websockets etc. because the lack of support we have a token polling mechanism in libresapi to avoid polling for every thing but it is still ugly, so if we can completely get rid of polling in the API that would be really nice. I have done a crawl to look for a replacement and briefly looked at

taking in account a few metrics like modern HTTP goodies support, license, platform support, external dependencies and documentation it seemed to me that restbed is the more appropriate.

Another source of boilerplate code into libresapi is the mapping between JSON API requests and C++ API methods as an example you can look at this

and this

The abstract logic of this thing is, when libreasapi get a request like /chat/initiate_distant_chat then call ChatHandler::handleInitiateDistantChatConnexion which in turn is just a wrapper of RsMsgs::initiateDistantChatConnexion all this process is basically implemented as boilerplate code and would be unnecessary in a smarter design of the API because almost all the information needed is already present in the C++ API libretroshare/src/retroshare.

So a third step to improve the JSON API would be to remove this source of boilerplate code by automatizing the mapping between C++ and JSON API call.

This may result a little tricky as language parsing or other adevanced things may be required.

Hope this dive is useful for you
Cheers
G10h4ck

Second writings about this

I have been investigating a bit more about:

So a third step to improve the JSON API would be to remove this source of
boilerplate code by automatizing the mapping between C++ and JSON API call
— G10h4ck

After spending some hours investigating this topic the most reasonable approach seems to:

  1. Properly document headers in libretroshare/src/retroshare/ in doxygen syntax specifying wihich params are input and/or output (doxygen sysntax for this is @param[in/out/inout]) this will be the API documentation too.

  2. At compile time use doxygen to generate XML description of the headers and use the XML to generate the JSON api server stub. http://www.stack.nl/~dimitri/doxygen/manual/customize.html#xmlgenerator

  3. Enjoy

You can’t perform that action at this time.