Network API provides a set of plain text commands for inserting numeric time series, key=value properties, and tagged messages into the Axibase Time Series Database (ATSD) via TCP and UDP network protocols.
You can use
UNIX pipes, and any programming language such as Java that lets you connect to the ATSD server via TCP/UDP protocol.
Extended Data Commands
By default, the ATSD server listens for incoming commands on the following ports:
- 8081 TCP
- 8082 UDP
To encrypt TCP traffic, setup an SSH tunnel or create a VPN connection.
Authentication and authorization are not supported for plain text commands received over TCP and UDP protocols.
Utilize the HTTP command to send plain-text commands over http/https protocols with authentication and authorization enabled.
To send a single command, connect to an ATSD server, and send the command in plain text and terminate the connection.
echo -e "series e:station_1 m:temperature=32.2 m:humidity=81.4 d:2016-05-15T00:10:00Z" | nc atsd_host 8081
printf 'series e:station_2 m:temperature=32.2 m:humidity=81.4 s:1463271035' | nc atsd_host 8081
- UNIX pipe
echo -e "series e:station_3 m:temperature=32.2 m:humidity=81.4" > /dev/tcp/atsd_host/8081
- telnet:one line
telnet atsd_host 8081 << EOF series e:station_4 m:temperature=32.2 m:humidity=81.4 EOF
$ telnet atsd_host 8081 Trying atsd_host... Connected to atsd_host. Escape character is '^]'. series e:station_5 m:temperature=32.2 m:humidity=81.4 ^C Connection closed by foreign host.
Socket s = new Socket("atsd_host", 8081); PrintWriter writer = new PrintWriter(s.getOutputStream(), true); writer.println("series e:station_6 m:temperature=32.2"); s.close();
The above examples insert timestamped temperature and humidity metric samples for station entities.
Separate commands by line feed symbol
0x0A) when sending a batch containing multiple commands over the same connection.
Trailing line feed is not required for the last command in the batch.
-e flag in
echo commands to enable interpretation of backslash escapes.
echo -e "series e:station_1 m:temperature=32.2 m:humidity=81.4 d:2016-05-15T00:10:00Z\nseries e:station_1 m:temperature=32.1 m:humidity=82.4 d:2016-05-15T00:25:00Z" | nc atsd_host 8081
Socket s = new Socket("atsd_host", 8081); PrintWriter writer = new PrintWriter(s.getOutputStream(), true); writer.println("series e:station_6 m:temperature=30.1\nseries e:station_7 m:temperature=28.7"); s.close();
A client application can establish a persistent connection in order to continuously write commands, one command per line, and close the connection.
Trailing line feed is not required for the last command when the connection is closed.
Commands are processed as they're received by the server, without buffering.
To prevent the connection from timing out the client may send
ping command at a regular interval.
Clients can submit different types of commands over the same connection.
$ telnet atsd_host 8081 Trying atsd_host... Connected to atsd_host. Escape character is '^]'. series e:station_1 m:temperature=32.2 property e:station_2 t:location v:city=Cupertino v:state=CA v:country=USA ^C Connection closed by foreign host.
Note that the server will terminate the connection if it receives an unsupported or malformed command.
$ telnet atsd_host 8081 Trying atsd_host... Connected to atsd_host. Escape character is '^]'. unknown_command e:station_1 m:temperature=32.2 Connection closed by foreign host.
If the connection is terminated due to client error, all valid commands sent prior to the first invalid command will be stored.
Due to the fact that channel closing on client error may take some time, the database may also store a few valid commands received after the discarded command.
valid command - stored ... - stored valid command - stored invalid command - discarded -> initiate channel closing valid command - possibly stored if present in buffer ...
The above behavior can be modified by changing the
input.disconnect.on.error setting to
No on the Admin > Server Properties page.
This will cause the database to maintain a client connection even if one of the received commands is malformed or unknown.
The UDP protocol doesn't guarantee delivery but may have a higher throughput compared to TCP due to lower overhead.
In addition, sending commands with UDP datagrams decouples the client application from the server to minimize the risk of blocking I/O time-outs.
echo -e "series e:station_3 m:temperature=32.2 m:humidity=81.4" | nc -u -w1 atsd_host 8082
printf 'series e:station_3 m:temperature=32.2 m:humidity=81.4' | nc -u -w1 atsd_host 8082
Unlike TCP, the last command in a multi-command UDP datagram must be terminated with the line feed character.
echo -e "series e:station_33 m:temperature=32.2\nseries e:station_34 m:temperature=32.1 m:humidity=82.4\n" | nc -u -w1 atsd_host 8082
Multiple commands with the same timestamp and key fields may override each others value.
If such commands are submitted at approximately the same time, there is no guarantee that they will be processed in the order they were received.
- Duplicate example: same key, same current time
echo -e "series e:station_1 m:temperature=32.2\nseries e:station_1 m:temperature=42.1" | nc atsd_host 8081
- Duplicate example: same key, same time
echo -e "series e:station_1 m:temperature=32.2 d:2016-05-15T00:10:00Z\nseries e:station_1 m:temperature=42.1 d:2016-05-15T00:10:00Z" | nc atsd_host 8081
- A command must start with a name such as
seriesfollowed by space-separated fields each identified with a prefix, followed by a (:) colon symbol and field name=value.
- The order of fields is not important.
- Refer to ABNF rules for particular commands for exact rules.
- A field name can contain only printable characters.
- If the field name contains a double-quote (") or equal (=) sign, it must be enclosed in double quotes. For example:
- Any double quote character in the value must be escaped with another double quote.
- A field value can contain printable and non-printable characters including space, line breaks, tab.
- If the field value contains a double-quote (") or equal (=) sign or a non-printable character, it must be enclosed in double quotes. For example:
- Any double quote character in the value must be escaped with another double quote.
Use CSV escaping methods in core libraries where available, for example StringEscapeUtils.escapeCsv in Java.
- Field names are case-insensitive and are converted to lower case when stored in the database.
- Field values are case-sensitive and are stored as submitted, except for entity name, metric name, and property type, which are converted to lower case.
Command Length Limits
The server enforces the following maximum lengths for command lines:
|Command||Maximum Length, bytes|
The client must split a command that is too long into multiple commands.
- New entities, metrics, and tags are created automatically when inserting data.
- The number of unique identifiers is subject to the following default limits:
16777215 in ATSD on HBase 1.x
The timestamp field encodes the time of an observation or an event as determined by the source and can be specified with
|ms||long||UNIX milliseconds since 1970-01-01T00:00:00Z|
|s||int||UNIX seconds since 1970-01-01T00:00:00Z|
|d||string||ISO 8601 date format. Supported formats:
UTC timezone (Z) = yyyy-MM-dd'T'HH:mm:ss[.SSS]'Z', for example 2016-06-09T16:15:04.005Z
Timezone offset = yyyy-MM-dd'T'HH:mm:ss[.SSS]±hh:mm, for example 2016-06-09T12:15:04.005-04:00
Time zone +hh:mm is ahead of UTC and timezone -hh:mm is behind UTC.
- Minimum time that can be stored in the database is 1970-01-01T00:00:00.000Z, or 0 milliseconds from Epoch time.
- Maximum date that can be stored by the database is 2106-02-07T06:59:59.999Z, or 4294969199999 milliseconds from Epoch time.
- If the timestamp field is not specified, time is set to current server time.
- Decimal separator is period (
- No thousands separator.
- No digit grouping.
- Negative numbers use negative sign (
-) at the beginning of the number.
- Not-a-Number is literal
By default, ATSD doesn't return acknowledgements to the client after processing data commands.
debug command at the start of the line to instruct the server to respond with
ok for each processed command.
debugwith valid command
$ echo -e "debug series e:station_1 m:temperature=32.2" | nc atsd_host 8081 ok
debugwith unknown command
$ echo -e "debug my_command e:station_1 m:temperature=32.2" | nc atsd_host 8081 >no response, connection closed
To validate a network received from a client, launch the
netcat utility in server mode, reconfigure the client to send data to the netcat port, and dump incoming data to file:
nc -lk 0.0.0.0 2081 > command-in.log & echo -e "series e:station_1 m:temperature=32.2 m:humidity=81.4 d:2016-05-15T00:10:00Z" | nc localhost 2081 cat command-in.log
TCP Client Examples
Reasons why ATSD server can drop commands:
- Entity, metric, or tag names are not valid.
- Timestamp is negative or earlier than
- Timestamp field
ms:is not numeric or if the
dfield is not in ISO format.
- Metric value could not be parsed as a number using
.as the decimal separator. Scientific notation is supported.
- Multiple data points for the same entity, metric, and tags have the same timestamp in which case commands are considered duplicates and some of them are dropped. This could occur when commands with the same key are sent without a timestamp.
- Data is sent using the UDP protocol and the client UDP send buffer or the server UDP receive buffer overflows.
- Value is below 'Min Value' or above 'Max Value' limit specified for the metric and the 'Invalid Value Action' is set to
- Last command in a multi-line UDP packed doesn't terminate with line feed symbol.
To review dropped commands,
open command*.log files in ATSD.