Serval Project, October 2017
The Serval DNA daemon that runs on every node in a Serval Mesh network gives applications access to the network through two main classes of API:
-
the MDP API and MSP API provide "traditional" packet and stream transport, allowing applications to send and receive Serval network packets to and from nearby nodes with latencies of up to several seconds;
-
the various HTTP REST APIs provide applications with access to Serval services:
-
Keyring REST API -- local identity management by querying and modifying the Keyring
-
Rhizome REST API -- store-and-forward (high latency) content distribution by extracting and inserting content in the local Rhizome store
-
MeshMS REST API -- secure one-to-one messaging by reading and writing the local cache of MeshMS messages
-
This document describes the features in common to all the HTTP REST APIs.
The Serval DNA HTTP REST API is an HTTP 1.0 server that only accepts requests on the loopback interface (IPv4 address 127.0.0.1), TCP port 4110. It rejects requests that do not originate on the local host, by replying 403 Forbidden.
The REST API uses plain HTTP without encryption. REST requests and responses are not carried over any physical network link, only local (“logical”) links between processes, so there is no risk of remote eavesdropping. The only potential threat comes from hostile local processes.
Operating system kernels such as Linux (Android, Ubuntu) and Darwin (Apple) prevent normal processes from accessing the traffic on local sockets between other processes. To attack Serval DNA and its clients, a local process on the local host would have to gain super-user privilege (eg, through a privilege escalation vulnerability). A super-user process would have many ways to attack Serval DNA and its clients, much more effective than intercepting their communications, so encrypting client-server communications would offer no protection whatsoever.
Clients of the HTTP REST API must authenticate themselves using Basic Authentication. This narrows the window for opportunistic attacks on the server's HTTP port by malicious applications that scan for open local ports to exploit. Any process wishing to use the REST API must supply valid authentication credentials (name/password), or will receive a 401 Unauthorized response.
Client applications obtain their REST API credentials via a back channel specific to their particular platform. This delegates the exercise of handing out credentials to the application layer, where users can (usually) exercise their own discretion. For example, on Android, a client app sends an Intent to the Serval Mesh app requesting a Serval REST credential, and will receive a reply only if it possesses the right Android Permission. When users install or run the client app, Android informs them that the app requests the "Serval Network" permission, and users may allow or deny it.
As a fall-back mechanism, created primarily to facilitate testing, HTTP REST API credentials can be configured using configuration options of the form:
api.restful.users.USERNAME.password=PASSWORD
PASSWORD is a cleartext secret, so the Serval DNA configuration file must be protected from unauthorised access or modification by other apps. That makes this mechanism unsuitable for general use.
An HTTP REST request is a normal HTTP 1.0 GET or POST:
A GET request consists of an initial "GET" line containing the path and HTTP version, followed by zero or more header lines, followed by a blank line. As usual for HTTP, all lines are terminated by an ASCII CR-LF sequence.
For example:
GET /restful/keyring/identities.json?pin=1234 HTTP/1.0
Authorization: Basic aGFycnk6cG90dGVy
Accept: */*
GET requests only accept parameters as percent encoded [query parameters][] in the path.
A POST request has a similar structure to a GET request except that the first word of the first line is "POST", and there may be a body of content following the blank line that ends the header.
POST requests accept parameters as percent encoded [query parameters][] in the path and also as a request body with a Content-Type of multipart/form-data. These two kinds of parameters are not exclusive; a POST request may contain a mixture of both.
A POST request may also include the following headers as described below:
In a POST request, the Content-Length header gives the exact number of bytes (octets) in the request's body, which must be correct. Serval DNA will not process a request until it receives Content-Length bytes, so if Content-Length is too large, the request will suspend and eventually time out. Serval DNA will ignore any bytes received after it has read Content-Length bytes, so if Content-Length is too small, the request body will be malformed.
Serval DNA treats a missing Content-Length header the same as a Content-Length of zero; it will not attempt to read the request body so any body content will be ignored if sent.
In a POST request, the Content-Type header gives the Internet Media Type of the body.
A missing Content-Type header in a POST request will cause a 400 Bad Request response. An unsupported content type will cause a 415 Unsupported Media Type response.
Serval DNA POST requests that take their parameters in the request body accept the multipart/form-data content type.
The multipart boundary must specify a string consisting of at most 70 ASCII characters that does not occur anywhere within the content of any part, and is used to delimit the parts of the request body. Typically this string is generated using a large random number encoded into printable ASCII, eg:
Content-Type: multipart/form-data;boundary=081d31d4c3d23014
Each form part introduces one named parameter, and consists of its own header section, followed by a blank line (a CR-LF immediately following the CR-LF that terminates the last header), followed by a body that contains the parameter's content, followed by a CR-LF. Every part must have a Content-Disposition header that gives the parameter's name, and may have an optional Content-Type header.
The following example passes two parameters to a request, called first and
second, the first is a single line of plain text terminated by a LF, the
second is an HTML document with no terminating LF:
POST /an/example/request HTTP/1.0
Content-Type: multipart/form-data;boundary=4aceafdc5cc7295d
--4aceafdc5cc7295d
Content-Disposition: form-data; name=first
Content-Type: text/plain; charset=utf-8
Hello, world!
--4aceafdc5cc7295d
Content-Disposition: form-data; name=second; filename="hello_world.html"
Content-Type: text/html; charset=utf-8
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>The second parameter</title>
</head>
<body>
<h1>Hello world!</h1>
</body>
</html>
--4aceafdc5cc7295d--
Serval DNA POST requests do not support the application/x-www-form-urlencoded content type, although some may do so in future.
application/x-www-form-urlencoded is the predecessor web standard to multipart/form-data. It has the benefit of being simpler for requests that take short, mainly textual parameters, but is very inefficient for encoding large binary values and does not provide any means to associate metadata such as content-type, encoding and file name with individual parameters.
Serval DNA POST requests that take plain text parameters use the text/plain content type in the parameter's form part. See the MeshMS send request for an example.
The only supported charset is utf-8; a missing or different charset will cause a 415 Unsupported Media Type response. The correct Content-Type header is:
Content-Type: text/plain; charset=utf-8
Serval DNA POST requests that take SID parameters use the non-standard serval/sid content type within the parameter's form part. See the Rhizome insert request for an example.
At present only the hex format is supported, and must be explicitly specified. A missing or different format will cause a 415 Unsupported Media Type. The correct Content-Type header is:
Content-Type: serval/sid; format=hex
Hex format parameter values may only contain ASCII characters from the set
0123456789ABCDEFabcdef; any other character (such as a trailing newline) will
cause a 400 Bad Request response.
In future other formats may be supported, such as Base-64, 7-bit binary, or 8-bit binary.
HTTP 1.1 Range retrieval is partially supported. In a POST request, the Range header gives the start and end, in byte offsets, of the resource to be returned. The server may respond with exactly the range requested, in which case the response status code will be 206 Partial Content, or it may ignore the Range header and respond with the entire requested resource.
For example, the following header asks that the server omit the first 64 bytes and send only the next 64 bytes (note that ranges are inclusive of their end byte number):
Range: bytes=64-127
The specification allows for more than one start-end range to be supplied, separated by commas, however not all REST API operations support multi ranges. If a multi-range header is used in such a request, then the response may be the entire content or 501 Not Implemented.
In a POST request, a Transfer-Encoding header of "chunked", indicates that the client can generate and transmit the request body without pre-calculating the final length. No other transfer encodings are currently supported.
Each chunk is generated as the size of the chunk in hex, followed by CR-LF, followed by the request bytes, followed by another CR-LF.
The end of the input is indicated with a chunk of zero length.
In a POST request, the presense of an Expect header of "100-Continue" indicates that the server should respond with an intermediate response of "HTTP/1.1 100 Continue" before the client begins to transmit a request body.
If for any reason the server determines that the request body is not needed, or the request is invalid, the server will generate the response immediately without reading the contents of the request body.
An HTTP REST response is a normal HTTP 1.0 response consisting of a header block, a blank line, and an optional body, for example: As usual, all lines are terminated by an ASCII CR-LF sequence. For example:
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 78
{
"http_status_code": 200,
"http_status_message": "OK"
}
The lingua franca of the HTTP REST API is JSON in UTF-8 encoding. All Serval DNA HTTP REST responses have a Content-Type of application/json unless otherwise documented.
Some responses contain non-standard HTTP headers as part of the result they return to the client; for example, Rhizome response headers.
The HTTP REST API response uses the HTTP status code to indicate the outcome of the request as follows:
The operation was successful and no new entity was created. Most requests return this code to indicate success. Requests that create a new entity only return this code if the entity already existed, meaning that the creation was not performed but the request can be considered a success since the desired outcome was achieved: namely, the existence of the entity. (If the entity was created, then these requests return 201 Created instead.)
(Serval APIs are all idempotent with respect to creation: creating the same entity twice yields the same state as creating it once. This is an important property for a purely distributed network that has no central arbiter to enforce sequencing of operations.)
The operation was successful and the entity was created. This code is only returned by requests that create new entities, in the case that the entity did not exist beforehand and has been created successfully.
The operation was successful but the entity was not created. This code is only returned by requests that create new entities, in the case that the request was valid but the entity was not created because other existing entities take precedence. For example, the Rhizome REST API returns this code when inserting a bundle to a full Rhizome store if the new bundle's rank falls below all other bundles, so the new bundle itself would be evicted to make room.
The operation was successful and the response contains part of the requested content. This code is only returned by requests that fetch an entity (the fetched entity forms the body of the response) if the request supplied a Range header that specified less than the entire entity.
The HTTP request was malformed, and should not be repeated without modifications. This could be for several reasons:
- invalid query parameter in the path
- invalid syntax in the request header block
- a POST request parameter is missing, duplicated or out of order
- a POST request was given an unsupported parameter
- a POST request parameter has missing or malformed content
The request did not supply an "Authorization" header with a recognised credential. This response contains a "WWW-Authenticate" header that describes the missing credential:
HTTP/1.0 401 Unauthorized
Content-Type: application/json
Content-Length: 88
WWW-Authenticate: Basic "Serval RESTful API"
{
"http_status_code": 401
"http_status_message": "Unauthorized"
}
The request failed because the server does not accept requests from the originating host.
The request failed because the HTTP request URI does not exist. This could be for several reasons:
- the request specified an incorrect path (typographic mistake)
- the path is unavailable because the API in question is unavailable (eg, the Rhizome REST API) is currently configured as disabled
- the path contains a reference to an entity (eg, SID, Bundle ID) that does not exist
The request failed because the HTTP request method is not supported for the given path. Usually this means that a GET request was attempted on a path that only supports POST, or vice versa.
A POST request did not supply either a Content-Length or Transfer-Encoding header.
The request failed because the HTTP request URI was too long. The server persists the path and a few other pieces of the request in a fixed size request buffer, and this response is triggered if the collective size of these does not leave enough buffer for receiving the remainder of the request.
A POST request failed because of an unsupported content type, which could be for several reasons:
- the request's Content-Type header specified an unsupported media type
- a part of a multipart/form-data request body has:
- a missing
Content-Dispositionheader, or - a
Content-Dispositionheader that is not of typeform-data, or - a missing or unsupported
Content-Typeheader (including a missing or unsupportedcharsetparameter)
- a missing
The Range header specified a range whose start position falls outside the size of the requested entity.
The request failed because the server does not possess and cannot derive the necessary cryptographic secret or credential. For example, updating a Rhizome bundle without providing the bundle secret. This code is not part of the HTTP standard.
A POST request supplied data that was inconsistent or violates semantic constraints, so cannot be processed. For example, the Rhizome insert operation responds with 422 if the manifest filesize and filehash fields do not match the supplied payload.
The request cannot be performed because a necessary resource is busy for reasons outside the control of the requester and server.
This code is returned by Rhizome requests if the Rhizome store database is currently locked by another process. The architecture of Serval DNA is being improved to prevent any process other than the Serval DNA daemon itself from directly accessing the Rhizome database. Once these improvements are done, this code should no longer occur except during unusual testing and development situations.
The request cannot be performed because a necessary resource is temporarily unavailable due to a high volume of concurrent requests.
The original use of this code was for Rhizome operations if the server's manifest table ran out of free manifests, which would only happen if there were many concurrent Rhizome requests holding manifest structures open in server memory.
This code may also be used to indicate temporary exhaustion of other finite resources. For example, if Serval DNA is ever limited to service only a few HTTP requests at a time, then this code will be returned to new requests that would exceed the limit.
The request header block was too long.
Initial implementations of Serval DNA allocated approximately 8 KiB of buffer memory for each request, and the HTTP server read each header line entirely into that buffer before parsing it. If a single header exceeded the size of this buffer, then the 431 response was returned.
The request failed because of an internal error in Serval DNA, not an error in the request itself. This could be for several reasons:
- software defect (bug)
- unavailable system resource (eg, memory, disk space)
- corrupted environment (eg, bad configuration, database inconsistency)
Internal errors of this kind may persist or may resolve if the request is re-tried, but in general they will persist because the cause is not transient. Temporary failures that can be resolved by re-trying the request are generally indicated by other status codes, such as 423 Locked.
The requested operation is valid but not yet implemented. This is used for the following cases:
To support client-side JavaScript applications, Serval DNA has a limited implementation of Cross-Origin Resource Sharing. If a request contains an Origin header with either “null” or a single URI with scheme “http” or “https” or “file”, hostname “localhost” or “127.0.0.1” (or empty in the case of a “file” scheme), and optionally any port number, then the response will contain three Access-Control headers granting permission for other pages on the same site to access resources in the returned response.
For example, given the request:
GET /restful/keyring/identities.json HTTP/1.0
Origin: http://localhost:8080/
...
Serval DNA will respond:
HTTP/1.0 200 OK
Access-Control-Allow-Origin: http://localhost:8080
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Authorization
...
All responses that convey no special content return the following JSON result object:
{
"http_status_code": ...,
"http_status_message": "..."
}
The http_status_code field is an integer equal to the status
code that follows the HTTP/1.0 token in the first
line of the response.
The http_status_message field is usually the same as the reason phrase text
that follows the code in the first line of the HTTP response. This reason
phrase may be a [standard phrase][status code], or it may be more explanatory;
for example, some 404 responses from Rhizome have phrases like, “Bundle not
found”, “Payload not found”, etc.
Some responses augment the JSON result object with extra fields; for example, Rhizome JSON result and Keyring JSON result.
Many HTTP REST responses that return a list of regular objects (eg, GET /restful/rhizome/bundlelist.json) use the following JSON table format:
{
"header":["fieldname1","fieldname2","fieldname3", ... ],
"rows":[
[field1, field2, field3, ... ],
[field1, field2, field3, ... ],
...
]
}
The JSON table format is more compact than the most straightforward JSON representation, an array of JSON objects, which has the overhead of redundantly repeating all field labels in every single object:
[
{
"fieldname1: field1,
"fieldname2: field2,
"fieldname3: field3,
...
},
{
"fieldname1: field1,
"fieldname2: field2,
"fieldname3: field3,
...
},
...
]
A JSON table can easily be transformed into its equivalent array of JSON objects. The test scripts use the following jq(1) expression to perform the transformation:
[
.header as $header |
.rows as $rows |
$rows | keys | .[] as $index |
[ $rows[$index] as $d | $d | keys | .[] as $i | {key:$header[$i], value:$d[$i]} ] |
from_entries |
.["__index"] = $index
]
Copyright 2015-2017 Serval Project Inc.
Available under the Creative Commons Attribution 4.0 International licence.