Shard Management Commands (OLAP)

[Table of Contents](https://github.com/dell-oss/Doradus/wiki/OLAP Databases: Table-of-Contents) | Previous | Next
OLAP REST Commands: Shard Management Commands

---

Doradus OLAP provides the following commands to list, merge, and delete shards.

List Shards

The list of all known shards of a given application can be obtained with the following REST command:

GET /{application}/_shards

Each shard that has at least one batch posted are listed, even if the shard has not yet been merged. A typical response in XML:

<result>
	<application name="Email">
		<shards>
			<value>2004-11-16</value>
			<value>2005-08-03</value>
			<value>2005-10-10</value>
			<value>2005-11-01</value>
			<value>2005-12-20</value>
			<value>2006-01-17</value>
			<value>2006-02-01</value>
		</shards>
	</application>
</result>

In JSON:

{"result": {
	"Email": {
		"shards": [
			"2004-11-16",
			"2005-08-03",
			"2005-10-10",
			"2005-11-01",
			"2005-12-20",
			"2006-01-17",
			"2006-02-01"
		]
	}
}}

Get Shard Statistics

Statistics for a specific shard can be retrieved with any of the following REST commands:

GET /{application}/_shards/{shard}
    GET /{application}/_statistics/{shard}
    GET /{application}/_statistics/{shard}?{params}

Statistics will be available for the given {shard} only if at least one batch has been loaded for it and the shard has been merged. The first command shown returns general information for each table and field. A typical response in XML is shown::

<stats memory="0.984 MB" storage="1.654 MB" documents="28455">
	<tables>
		<table documents="370" memory="0.010 MB" name="Domain">
			<numFields>
				<field type="BOOLEAN" min="0" max="1" min_pos="1" bits="1" memory="0.000 MB" name="IsInternal"/>
				</numFields>
			<textFields>
				<field valuesCount="370" doclistSize="370" isSingleValued="true" memory="0.001 MB" name="Name"/>
			</textFields>
			<linkFields>
				<field linkedTableName="Address" inverseLink="Domain" doclistSize="1884" isSingleValued="false" memory="0.009 MB" name="Addresses"/>
			</linkFields>
		</table>
		<table documents="6030" memory="0.272 MB" name="Message">
		...
		</table>
		...
	</tables>
</stats>

In JSON:

{"stats": {
	"memory": "0.984 MB",
	"storage": "1.654 MB",
	"documents": "28455",
	"tables": {
		"Domain": {
			"documents": "370",
			"memory": "0.010 MB",
			"numFields": {
				"IsInternal": {"type": "BOOLEAN", "min": "0", "max": "1", "min_pos": "1", "bits": "1", "memory": "0.000 MB"}
			},
			"textFields": {
				"Name": {"valuesCount": "370", "doclistSize": "370", "isSingleValued": "true", "memory": "0.001 MB"}
			},
			"linkFields": {
				"Addresses": {"linkedTableName": "Address", "inverseLink": "Domain", "doclistSize": "1884", "isSingleValued": "false", "memory":"0.009 MB"}
			}
		},
		"Message": {
			...
		},
		...
	}
}}

The command response provides statistics about data contained in the given shard such as the total number of objects, the number of objects per table, the name value range of scalar fields, and so forth.

The second shard statistics command returns more similar information as the first command plus additional details about the virtual files (segments) stored within the shard. The third statistics command dumps the contents of a specific virtual file. These commands are intended for debugging purposes.

Merge Shard

All batches that have been loaded for a given shard are merged with the following REST command:

POST /{application}/_shards/{shard}[?expire-date=<timestamp>]

where {application} is the owning application's name and {shard} is the shard name. The command requires no input entity. It merges all segments synchronously and returns when the merge is complete.

The Merge Shard command optionally assigns an expiration date to the shard via the expire-date parameter. The given <timestamp> must be in the format of a timestamp (YYYY-MM-DD HH:MM:SS); trailing elements can be omitted. Example:

POST /Email/_shards/2014-01-01?expire-date=2015-01-01

If the expire-date is not given, the shard will have no expiration date, even if it was previously assigned one. If the command is successful, a 200 OK response is returned, and all updates will be visible to queries.

Note that only one process can submit a merge request for a given shard at a time. If a merge request is submitted for a shard that is already undergoing a merge request, the second request immediately receives an error.

Delete Shard

A shard can be explicitly deleted with the following command:

DELETE /{application}/_shards/{shard}

All of the shard's data is removed, even if it was previously assigned an expire-date that has not yet passed. If the request is successful, a 200 OK response is returned.