- Name
- Synopsis
- Description
- Options
- Config File Parameters
- Access Control
- Seedlink Support
- Multi-Protocol Support
- Http Support
- Transfer Logging
- Miniseed Archiving
- Miniseed Scanning
- Author
ringserver [options] [configfile]
ringserver is a TCP-based ring buffer designed for packetized streaming data. The ring buffer is implemented as a FIFO with newly arriving packets pushing older packets out of the buffer. The ring packets are not format specific and may contain any type of data. All communications are performed via TCP interfaces. Data is available to clients using the SeedLink and DataLink protocols and submitted to the server using the DataLink protocol.
The server is configured either with options on the command line or by using a ringserver config file (or both). Only the most common options are available on the command line, all options are tunable via the config file. Detailed descriptions for each option are included in the example config file that accompanies the source code. Many options are dynamic, meaning that they can be changed while the server is running. In this case the server will recognize that the config file has changed and will re-read it's configuration. This is especially useful for updating access control lists, logging verbosity and other logging parameters.
In normal operation packet buffer contents are saved in files when the server is shut down making the server stateful across restarts. By default the packet buffer is managed as a memory-mapped file, it can optionally be maintained completely in system memory and only read and write the buffer contents on startup and shutdown (useful in environments where memory-mapping is not possible).
Client access is controlled using 3 IP address based lists: one list for specific IPs that must match, one list for specific IPs that are to be rejected and one for IPs that are allowed to send data to the server (write access). See Access Control for more details.
Transfer logs can optionally be written to track the transmission and reception of data packets to and from the server. This tracking is stream-based and identifies the number of packet bytes of each unique stream transfered to or from each client connection.
The server supports multiple protocols (SeedLink, DataLink and HTTP) on the same listening port. Support can be limited to certain protocols on a by-port basis. See Multi-protocol support for more information.
The server also has limited support for HTTP requests. When support is enabled, server status, stream lists and other details can be accessed with simple HTTP requests. See HTTP Support for more details.
The slinktool(1) and dalitool(1) programs can be used to query the ringserver for various information via the SeedLink and DataLink interfaces respectively. The dalitool program was developed in parallel with ringserver and the DataLink protocol and is the recommended query tool for ringserver admins.
-V
Print the program version and exit.
-h
Print the program help/usage and exit.
-H
Print an extended help/usage and exit.
-v
Be more verbose. This flag can be used multiple times ("-v -v" or "-vv") for more verbosity.
-I ServerID
Server ID reported to the clients. The value may be a quoted string containing spaces. The default value is "Ring Server".
-M maxclientsperIP
Maximum number of concurrently connected clients per IP address. This limit does not apply to addresses with write permission. There is no default maximum.
-m maxclients
Maximum number of concurrently connected clients. The default maximum is 600.
-Rd ringdir
Base directory for the server to store the packet buffer data files. This option must be specified on the command line or in the config file, there is no default.
-Rs bytes
Size of the packet buffer in bytes, default size is 1 GB. The size of the ring, in combination with the ring packet size, determine how much past data is available in the buffer. If the server is configured to memory-map the packet buffer files the maximum size of the ring is limited to how large a file the host system can support. If the server is configured to maintain the ring in system memory the size is limited to system memory.
-Rm maxID
Maximum ring packet ID used to uniquely identify a packet. Default is 16,777,215 which the maximum supported by the SeedLink protocol, if SeedLink is not being used it can be much higher.
-Rp packetsize
Maximum ring packet data size (not including packet buffer header), default is 512 bytes.
-NOMM
No memory-mapping, maintain the ring buffer in system memory only. On startup the ring buffer files will be read into memory and on shutdown the memory buffers will be written back to the files. This option might be useful in environments where memory-mapping of files is not possbile or is unreliable (e.g. NFS).
-L port
Network port to listen for incoming connections on. By default the server will not listen for connections, setting at least one listen port is needed to communicate with the server. By default, a listening port will accept all supported protocols. Restricting a port to only allow specific protocols can be done using a configuration file, see Multi-protocol Support for more information.
-T logdir
Transfer log file base directory, by default the server does not write transfer logs. If a directory is specified both transmission and reception logs will be written, these two logs can be toggled individually in the server config file.
-Ti hours
Transfer log writing interval in hours, default interval is 24 hours. The interval always starts at day boundaries, for example if the interval is set to 2 hours then the first interval in a given day will cover hours 0 through 2, the next 2 through 4, etc.
-Tp prefix
Transfer log file prefix, by default no prefix is added to the fixed section of the log file name.
-STDERR
Send all diagnostic output to stderr instead of stdout. This is useful in situations where logging output is captured by another program on stderr and to separate ringserver diagnostics from other output on the console.
-MSWRITE format
A special mode of ringserver is to write all miniSEED data records received via DataLink (ring packets ending with the /MSEED suffix) to a user defined directory and file structure. See miniSEED Archiving for more details.
-MSSCAN directory [suboptions]
A special mode of ringserver is to recursively scan a directory for files containing miniSEED formatted data records and insert them into the ring as packets with a /MSEED suffix. Optional suboptions control scanning behavior, the StateFile suboption is highly recommended. See miniSEED Scanning for more details.
-VOLATILE
Create a volatile (non-stateful) ring buffer, in other words do not read packet buffer contents from ring files on startup or write them on shutdown. This is useful in combination with the -MSWRITE option when no stateful buffer is needed.
All of the command line parameters have config file equivalents, below are the config file parameters which have no command line equivalents. Many of the config file parameters are dynamic, if they are changed the server will re-read it's configuration on the fly. See the detailed parameter descriptions in the documented example config file.
AutoRecovery [0|1|2] - Control autorecovery after corruption detection ResolveHostnames [0|1] - Control reverse DNS lookups TimeWindowLimit % - Control limit for time window searches TransferLogTX [0|1] - Control writing of transmission log TransferLogRX [0|1] - Control writing of reception log WriteIP IP[/netmaks] - Add IP address(es) to write permission list LimitIP IP[/netmaks] RegEx - Add IP address(es) to limit list MatchIP IP[/netmaks] - Add IP address(es) to match list RejectIP IP[/netmaks] - Add IP address(es) to reject list
By default all clients are allowed to connect. Specific clients can be rejected using the RejectIP config parameter. If any MatchIP config parameters are specified only addresses that match one of the entries, and are not rejected, are allowed to connect.
By default all clients are allowed access to all streams in the buffer, and clients with write permission are allowed to write any streams. Specific clients can be limited to access or write subsets of streams using the LimitIP config parameter. This parameter takes a regular expression that is used to match stream IDs that the client(s) are allowed access to or to write.
By default all clients are allowed to request the server ID, simple status and list of streams. Specific clients can be allowed to access connection information and more detailed status using the TrustedIP config parameter.
If no client addresses are granted write permission via WriteIP or granted trusted status via TrustedIP then the 'localhost' address (local loopback) are granted those permissions.
Access control is host range (network) based, and specified as an address followed by an optional prefix in CIDR notation. For example: "192.168.0.1/24" specifies the range of addresses from 192.168.0.1 to 192.168.0.254. The address may be a hostname, which will be resolved on startup. The prefix is optional and, if omitted, defaults to specifying only the single address.
The legacy SeedLink protocol only transmits 512-byte miniSEED data records. This server is able to transmit miniSEED records of any length via SeedLink. To ensure compatitiblity with legacy clients, only 512-byte miniSEED records should be inserted into the ring buffer.
This server supports the wildcarding of network and station codes during SeedLink negotiation using the '?' and '*' characters for single or multiple character matches respectively. Not all SeedLink clients support wildcarded network and station codes.
Network listening ports can respond to all supported protocols (SeedLink, DataLink and HTTP). The first command received by the server is used to determine which protocol is being used by the client, all subsequent communication is expected in this protocol.
The protocols allowed by any given listening port can be set to any combination of the supported protocols by adding flags to the Listen parameter of the server configuration file.
Both IPv4 and IPv6 are supported by default (if supported by the system). The server can be limited to a specified network protocol family by adding flags to the Listen parameter of the server configuration file.
The server will respond to HTTP requests for a few fixed resources. If the WebRoot config parameter is set to a directory, the files under that directory will also be served when requesed through the HTTP GET method. Except for the fixed resources, the HTTP server implementation is limited to returning existing files and returning "index.html" files when a directory is requested.
The following fixed resources are supported:
/id - Server identification /streams - List of available streams with time range /streamids - List of available streams, variable levels /status - Server status, limited access* /connections - List of connections, limited access* /seedlink - Initiate WebSocket connection for Seedlink /datalink - Initiate WebSocket connection for DataLink
Access to the status and connections information is limited to clients that have trusted permission.
The streams, streamids and connections endpoints accept a match parameter that is a regular expression pattern used to limit the returned information. For the streams and streamids endpoints the matching is applied to stream IDs. For the connections endpoint the matching is applied to hostname, client IP address and client ID. For example: http://localhost/streams?match=IU_ANMO.
The streamids endpoint accepts a level parameter that limits the returned information to a unique list of stream identifiers at the specified level. Valid values are 1 through 6. Identifier components should be delimited with underscore characters. To illustrate, if a ringserver contains streams in the pattern of "NET_STA_LOC_CHAN/MSEED" a request for level 2 returns a unique list of "NET_STA" values. For example: http://localhost/streamids?level=2.
After a WebSocket connection has been initiated with either the seedlink or datalink end points, the requested protocol is supported exactly as it would be normally with the addition of WebSocket framing. Each server command should be contained in a single WebSocket frame, independent of other commands.
Custom HTTP headers may be included in HTTP responses using the HTTPHeader config file parameter. This can be used, for example, to enable cross-site HTTP requests via Cross-Origin Resource Sharing (CORS).
The -T command line option or the TransferLogTX or TransferLogRX config file parameters turn on logging of data either transmitted or received. The log interval and file name prefix can be changed via the -Ti and -Tp command line options.
Both the transmission (TX) and reception (RX) log files contain entries that following this pattern:
1) A "START CLIENT" line that contains the host name, IP address, protocol, client ID, log time, and connection time.
2) One or more data lines of the following form:
[Stream ID] [bytes] [packets]
3) An "END CLIENT" line including the total bytes or this entry.
Note: the byte counts are the sum of the data payload bytes in each packet and do not include the DataLink or SeedLink protcol headers.
An example "TX" file illustrating a transmission entry:
START CLIENT host.iris.edu [192.168.255.255] (SeedLink|Client) @ 2018-03-30 07:00:05 (connected 2018-03-30 06:59:36) TX IU_SNZO_10_BHZ/MSEED 2560 5 IU_SNZO_00_BHZ/MSEED 2048 4 END CLIENT host.iris.edu [192.168.255.255] total TX bytes: 4608
Using either the -MSWRITE command line option or the MSeedWrite config file parameter the server can be configured to write all miniSEED data records received via DataLink to a user defined directory and file structure. Only ring packets ending with the /MSEED suffix are considered.
The archive format argument is expanded for each packet processed using the following flags:
n : network code, white space removed s : station code, white space removed l : location code, white space removed c : channel code, white space removed q : record quality indicator (D,R,Q,M), single character Y : year, 4 digits y : year, 2 digits zero padded j : day of year, 3 digits zero padded H : hour, 2 digits zero padded M : minute, 2 digits zero padded S : second, 2 digits zero padded F : fractional seconds, 4 digits zero padded D : current year-day time stamp of the form YYYYDDD L : data record length in bytes r : sample rate (Hz) as a rounded integer R : sample rate (Hz) as a float with 6 digit precision h : host name of client submitting data % : the percent (%) character # : the number (#) character
The flags are prefaced with either the % or # modifier. The % modifier indicates a defining flag while the # indicates a non-defining flag. All received packets with the same set of defining flags will be saved to the same file. Non-defining flags will be expanded using the values in the first packet received for the resulting file name.
Time flags are based on the start time of the given packet.
Files are created with (permission) mode 666 and directories are created with mode 777. An operator of ringserver can control the final permissions of the files by adjusting the umask as desired.
Some preset archive layouts are available:
BUD : %n/%s/%s.%n.%l.%c.%Y.%j (BUD layout) CHAN : %n.%s.%l.%c (channel) QCHAN : %n.%s.%l.%c.%q (quality-channel-day) CDAY : %n.%s.%l.%c.%Y:%j:#H:#M:#S (channel-day) SDAY : %n.%s.%Y:%j (station-day) HSDAY : %h/%n.%s.%Y:%j (host-station-day)
The preset archive layouts are used by prefixing a target directory with the preset identifier followed by an '@' character. For example:
BUD@/data/bud/
would write a BUD like structure in the /data/bud/ directory.
Other example:
/archive/%n/%s/%n.%s.%l.%c.%Y.%j
would be expanded to day length files named something like:
/archive/IU/ANMO/IU.ANMO..BHE.2003.055
Using non-defining flags the format string:
/data/%n.%s.%Y.%j.%H:#M:#S.miniseed
would be expanded to:
/data/IU.ANMO.2003.044.14:17:54.miniseed
resulting in hour length files because the minute and second are specified with the non-defining modifier. The minute and second fields are from the first packet in the file.
Using either the -MSSCAN command line option or the MSeedScan config file parameter the server can be configured to recursively scan a directory for files containing miniSEED data records and insert them into the ring. Intended for real-time data re-distribution, files are continuously scanned, newly added records are inserted into the ring.
Sub-options can be used to control the scanning process. The sub-options are specified on the same line as the scan directory as key-value pairs separated by an equals '=' character and may not contain spaces (because they are separated by spaces). Do not use quotes for the values. The available sub-options are:
StateFile : File to save scanning state through restarts Match : Regular expression to match file names Reject : Regular epression to reject file names InitCurrentState : Initialize scanning to current state MaxRecurse : Maximum recursion depth (default is no limit)
Except for special cases the StateFile option should always be specified, otherwise a restart of the server could re-read data records that it has already read.
If the InitCurrentState option is set to 'y' the scanning will only read new data, effectively skipping all the data discovered during the first scan, under the following conditions:
1) No StateFile has been specified 2) StateFile has been specified but does not exist
The InitCurrentState option is useful to avoid reading all existing data when starting a server scanning an existing large dataset. It is also useful to reset the dataflow to current data after a lengthy downtime, simply remove the statefile(s) before starting the server.
To scan a data directory and save the scanning state to a StateFile configure the server with either a config file option or command line, respectively:
MSeedScan /data/miniseed/ StateFile=/opt/ringserver/scan.state
-MSScan "/data/miniseed/ StateFile=/opt/ringserver/scan.state"
To limit the scanning to file names matching a certain pattern use the Match option, e.g. files ending in ".mseed":
MSeedScan /data/miniseed/ StateFile=/data/scan.state Match=.*\\.mseed$
Chad Trabant IRIS Data Management Center
(man page 2020/03/08)