Skip to content

SolarUser Datum Export API

Jump to bottom Edit New page
Matt Magoffin edited this page Mar 27, 2024 · 13 revisions

The SolarUser Datum Export API provides methods to manage data export tasks. All paths are relative to a /solaruser prefix. All dates and times are represented in the Gregorian calendar system. All requests must provide a valid user authentication token. See SolarNet API authentication for information on how to use authentication tokens.

Endpoints

Verb Endpoint Description
GET /user/export/adhoc List previously submitted ad hoc export tasks.
POST /user/export/adhoc Submit an ad hoc export task request.
POST /user/export/adhocRef Submit an ad hoc export task request that references existing output and destination configurations.
GET /user/export/services/compression List the available export compression types.
GET /user/export/services/destination List the available export destination types.
GET /user/export/services/output List the available export output format types.

Localized responses

Many endpoints return localized messages. The locale of the response is determined by the Accept-Language HTTP header passed on the request. If not provided, the default locale of the SolarNetwork service will be used.

Localized setting specifiers

Some responses include a localizedInfoMessages response object. This objects contains a pair of localized messages for the setting specifier key values returned in the same response. The localized message pairs are in the form X.key (a title) and X.desc (a longer description) where X is a setting specifier key. For example, a response might look like:

{
  "id": "net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService",
  "locale": "en-NZ",
  "settingSpecifiers": [
	{
	  "key": "includeHeader",
	  "defaultValue": true,
	  "trueValue": true,
	  "falseValue": false,
	  "transient": false,
	  "type": "net.solarnetwork.settings.ToggleSettingSpecifier"
	}
  ],
  "localizedDescription": "Export data in comma separated values (spreadsheet) format.",
  "localizedInfoMessages": {
	"includeHeader.key": "Include Header",
	"includeHeader.desc": "Toggle the inclusion of a CSV header row."
  },
  "localizedName": "CSV"
}

A single includeHeader setting is defined for this service. The localizedInfoMessages object thus defines the title for that setting via includeHeader.key (Include Header) and a description via includeHeader.desc (Toggle the inclusion of a CSV header row.).

List ad hoc export tasks

List the available ad hoc export tasks previously submitted via a POST to this same URL.

GET /solaruser/api/v1/sec/user/export/adhoc
states An optional comma-delimited list of task state key values. If not provided all states are returned.
success An optional true or false to only include tasks with a matching success value. If not provided all success values are returned.

For example, to find all pending tasks you'd call this API like:

/solaruser/api/v1/sec/user/export/adhoc?states=q,p,e

To find all failed tasks you'd call this API like:

/solaruser/api/v1/sec/user/export/adhoc?states=c&success=false

To find all completed tasks (both successful and failed) you'd call this API like:

/solaruser/api/v1/sec/user/export/adhoc?states=c

List ad hoc export tasks response

The response contains a list of ac hoc export tasks. Tasks have the following properties:

Property Type Description
id string A unique identifier assigned by SolarNetwork.
userId number The ID of the user that owns the task.
scheduleTypeKey string Will always be the value a for ad hoc.
created string The date the task was submitted, in YYYY-MM-dd HH:mm:ss.SSS'Z' format, in the UTC time zone.
config object The export configuration object.
task object An export task execution object.

An example response looks like:

{
  "success": true,
  "data": [
    {
      "id": "0968234c-0c6e-4a14-8e7b-9148f7e41080",
      "userId": 7,
      "scheduleTypeKey": "a",
      "created": "2019-01-31 21:36:33.750Z",
      "config": {
        "name": "Ad hoc: 2019-01-31T21:36:33+00:00",
        "dataConfiguration": {
          "datumFilter": {
            "nodeIds": [
              11
            ],
            "sourceIds": [
              "Main"
            ],
            "aggregation": "None",
            "aggregationKey": "0",
            "mostRecent": false,
            "startDate": "2014-03-01 08:00:00.000Z",
            "endDate": "2014-03-31 11:00:00.000Z",
            "offset": 0,
            "location": {},
            "withoutTotalResultsCount": false
          }
        },
        "outputConfiguration": {
          "serviceIdentifier": "net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService",
          "compressionTypeKey": "x",
          "serviceProperties": {
            "includeHeader": true
          }
        },
        "destinationConfiguration": {
          "serviceIdentifier": "net.solarnetwork.central.datum.export.dest.s3.S3DatumExportDestinationService",
          "serviceProperties": {
            "path": "s3-us-west-2.amazonaws.com/my-bucket/data-exports",
            "accessKey": "mRWO6qzn57AFfIq0Qo60zyD3",
            "secretKey": "{SSHA-256}CXZ3Rz5botMSZi4AOyKgxoJr8OoCw==",
            "filenameTemplate": "adhoc-data-export-{date}.{ext}"
          }
        },
        "schedule": "Adhoc",
        "hourDelayOffset": 0
      },
      "task": {
        "id": "0968234c-0c6e-4a14-8e7b-9148f7e41080",
        "created": "2019-01-31 21:36:33.750Z",
        "modified": "2019-01-31 21:41:11.925Z",
        "exportDate": "2019-01-31 21:36:33.723Z",
        "config": {
          ...
        },
        "statusKey": "c",
        "completionDate": "2019-01-31 21:41:11.923Z",
        "success": true
      }
    }
  ]
}

Submit ad hoc export task

Submit an ad hoc export task configuration for asynchronous execution. Once submitted, the export task will be executed at some point in the future. Use the list endpoint to check on the status of each task.

POST /solaruser/api/v1/sec/user/export/adhoc

The request must be submitted as application/json as a JSON task configuration object with the following properties:

Property Type Description
name string A name for the export task.
dataConfiguration object Data configuration object that defines which data to export.
outputConfiguration object Output format configuration object that defines the format to encode the data as.
destinationConfiguration object Destination configuration object that defines where to export the data to.

For example, the following task will export node 123, source /Meter/1 raw data for all of 2011 (UTC) as CSV data with a header row, compressed using xz, to S3 at /my-bucket/data-exports:

{
  "name": "Ad hoc: 2019-01-31T21:36:33+00:00",
  "dataConfiguration": {
    "datumFilter": {
      "nodeIds": [
        123
      ],
      "sourceIds": [
        "/Meter/1"
      ],
      "startDate": 1293840000000,
      "endDate": 1325376000000
    }
  },
  "outputConfiguration": {
    "compressionTypeKey": "x",
    "serviceIdentifier": "net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService",
    "serviceProperties": {
      "includeHeader": true
    }
  },
  "destinationConfiguration": {
    "serviceIdentifier": "net.solarnetwork.central.datum.export.dest.s3.S3DatumExportDestinationService",
    "serviceProperties": {
      "path": "s3-us-west-2.amazonaws.com/my-bucket/data-exports",
      "filenameTemplate": "adhoc-data-export-{date}.{ext}",
      "accessKey": "mRWO6qzn57AFfIq0Qo60zyD3",
      "secretKey": "alkjdsfoaseifuasldfasdfas"
    }
  }
}

Ad hoc data configuration

The dataConfiguration request property defines which data to include in the export. It is essentially a search filter: any datum that matches the filter will be included in the export. It supports the following properties:

Property Type Description
datumFilter object The filter criteria object.

The datumFilter object supports the following properties:

Property Type Description
startDate number An inclusive starting date to use in the query, as milliseconds since the Unix epoch or an ISO 8601 instant string like 2019-01-01T00:00:00Z.
endDate number An exclusive ending date to use in the query, as milliseconds since the Unix epoch or an ISO 8601 instant string like 2019-01-01T00:00:00Z.
aggregation string An optional aggregation type. If not provided, then non-aggregated data will be returned.
nodeId number The ID of a single node.
nodeIds array An array of node IDs.
sourceId string The ID of a single source to restrict the results to. A wildcard pattern can be used to restrict the results to only source IDs that match the pattern.
sourceIds array An array of source IDs to restrict the results to. Wildcard patterns can be used in any value.
combiningType string An optional combining type to combine raw results into virtual ones. Specifying a type here requires some combintation of nodeIdMaps and sourceIdMaps parameters also be provided. See Virtual IDs for more information.
nodeIdMaps array Return combined node IDs as virtual node IDs using the specified combiningType. The format of each value is VID:IDLIST where VID is a virtual node ID (integer) and IDLIST is a comma-delimited list of real node IDs. See Virtual IDs for more information.
sourceIdMaps array Return combined source IDs as virtual source IDs using the specified combiningType. The format of each value is VID:IDLIST where VID is a virtual source ID and IDLIST is a comma-delimited list of real source IDs. See Virtual IDs for more information.
readingType string Export reading-style data by specifying the type of reading to perform. See Datum reading types for possible values. If used with aggregation then only Hour, Day, and Month aggregation values and only the Difference reading type is supported. See the SolarNet aggregation guide for more info.

An example data configuration object that includes non-aggregate data for node ID 123, source ID /Meter/1 for the year 2011 (UTC) looks like this:

{
    "datumFilter": {
      "nodeIds": [
        123
      ],
      "sourceIds": [
        "/Meter/1"
      ],
      "startDate": 1293840000000,
      "endDate": 1325376000000
    }
}

Ad hoc output format configuration

The outputConfiguration request property defines the format to encode the data as, and if the data should be compressed. The supported output formats are available via the List output formats endpoint. The supported compression types are available via the List compression types endpoint.

Property Type Description
compressionTypeKey string The id of any supported compression type.
serviceIdentifier string The id of any supported output format.
serviceProperties object An optional object of setting keys and associated values to apply on the selected output format.

The supported serviceProperties are specific to each output format. Consult the List output formats documentation for more information.

An example data configuration object that exports the data in CSV form with a header row and compresses the result with XZ compression looks like this:

{
  "serviceIdentifier": "net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService",
  "compressionTypeKey": "x",
  "serviceProperties": {
    "includeHeader": true
  }
}

Ad hoc destination configuration

The destinationConfiguration request property defines where to export the data to. The supported destinations are available via the List destinations endpoint.

Property Type Description
serviceIdentifier string The id of any supported destination.
serviceProperties object An optional object of setting keys and associated values to apply on the selected destination.

The supported serviceProperties are specific to each destination. Consult the List destinations documentation for more information.

An example data configuration object that exports the data to S3 at /my-bucket/data-exports looks like this:

{
  "serviceIdentifier": "net.solarnetwork.central.datum.export.dest.s3.S3DatumExportDestinationService",
  "serviceProperties": {
    "path": "s3-us-west-2.amazonaws.com/my-bucket/data-exports",
    "accessKey": "mRWO6qzn57AFfIq0Qo60zyD3",
    "secretKey": "{SSHA-256}CXZ3Rz5botMSZi4AOyKgxoJr8OoCw==",
    "filenameTemplate": "adhoc-data-export-{date}.{ext}"
  }
}

Submit ad hoc export tasks response

The response will be a task object. See the list response for more details.

List compression types

This method will list the compression types supported by the SolarNetwork datum export service.

GET /solaruser/api/v1/sec/user/export/services/compression

List compression types response

The response contains an array of compression service definitions.

Property Type Description
id string The service unique identifier.
settingSpecifiers array The configurable setting specifiers for the service.
locale string The locale of the localized messages in the response.
localizedName string A localized name for the service.
localizedDescription string A localized description of the service.
localizedInfoMessages object Localized messages for the associated setting specifiers.

An example response looks like:

{
  "success": true,
  "data": [
    {
      "id": "n",
      "locale": "en-NZ",
      "localizedDescription": "No compression.",
      "localizedName": "None"
    },
    {
      "id": "g",
      "locale": "en-NZ",
      "localizedDescription": "GZIP compression.",
      "localizedName": "GZIP"
    },
    {
      "id": "x",
      "locale": "en-NZ",
      "localizedDescription": "XZ compression.",
      "localizedName": "XZ"
    }
  ]
}

Submit ad hoc reference export task

Submit an ad hoc reference export task configuration for asynchronous execution. The "reference" part of this API means the ad hoc export refers to existing output and destination configurations, so they do not need to be configured in the submitted request (just referenced by ID). Once submitted, the export task will be executed at some point in the future. Use the list endpoint to check on the status of each task.

POST /solaruser/api/v1/sec/user/export/adhocRef

The request must be submitted as application/json as a JSON task configuration object with the following properties:

Property Type Description
name string A name for the export task.
dataConfiguration object Data configuration object that defines which data to export.
outputConfigurationId number The ID of an existing Output format configuration object that defines the format to encode the data as.
destinationConfigurationId number The ID of an existing Destination configuration object that defines where to export the data to.

For example, the following task will export node 123, source /Meter/1 raw data for all of 2011 (UTC) using the output configuration with ID 123 and the destination configuration with ID 234:

{
  "name": "Ad hoc: 2019-01-31T21:36:33+00:00",
  "outputConfigurationId": 123,
  "destinationConfigurationId": 234,
  "dataConfiguration": {
    "datumFilter": {
      "nodeIds": [
        123
      ],
      "sourceIds": [
        "/Meter/1"
      ],
      "startDate": 1293840000000,
      "endDate": 1325376000000
    }
  }
}

List destinations

This method will list the destinations supported by the SolarNetwork datum export service.

GET /solaruser/api/v1/sec/user/export/services/destination

SolarNetwork supports the following destinations:

Destination Description
FTP Upload the export result to a FTP server.
HTTP Upload the export result to a URL using HTTP.
S3 Upload the export result to any AWS S3 compatible server.

FTP destination

ID net.solarnetwork.central.datum.export.dest.ftp.FtpDatumExportDestinationService

The FTP destination uploads the export result to an FTP server. TLS is supported (ftps, see below for more information. It supports the following settings:

Setting Type Description
url string The absolute URL to use, including the path and file name, such as ftps://example.com/export-{date}.{ext}. Use ftps:// for TLS encrypted connections, ftp:// otherwise. The default port is 21. Include the port in the URL if anything else is required, for example ftps://example.com:2121. Template parameters are supported in the form {key}, which are replaced at runtime by the value of a parameter key, or an empty string if no such parameter exists. The supported parameters are listed below.
username string An optional username to use for authorization. The password setting must also be configured.
password string An optional password to use for authorization. The username setting must also be configured.
implicitTls boolean Enable to establish a TLS connection immediately on ftps:// URLs (for example when using port 990). When disabled, a non-TLS connection is made first, and then TLS is negotiated with the FTP server.
dataTls boolean Enable to use TLS for data connections on ftps:// URLs. Otherwise only control connections will be encrypted.

FTP destination TLS support

Encrypted TLS communication is supported (and recommended), and enabled by using a ftps:// style URL. If the FTP server requires "implicit TLS" (often on port 990) set implicitTls to true. FTP uses two connections: one for control messages and one for file data. Only if dataTls is true will the file data connection use TLS. The dataTls setting defaults to true, and we highly recommend you only upload to an FTP server that supports this mode.

HTTP destination

ID net.solarnetwork.central.datum.export.dest.http.HttpDatumExportDestinationService

The HTTP destination uploads the export result to a URL using HTTP. The upload can be included directly as the HTTP request body content, or as a multipart/form-data "file upload" style request. It supports the following settings:

Setting Type Description
method string The HTTP verb to use, such as POST or PUT. Defaults to POST.
url string The absolute URL to use, such as https://example.com/upload. Template parameters are supported in the form {key}, which are replaced at runtime by the value of a parameter key, or an empty string if no such parameter exists. The supported parameters are listed below.
username string An optional username to use with HTTP Basic authorization. The password setting must also be configured.
password string An optional password to use with HTTP Basic authorization. The username setting must also be configured.
multipartFilenameTemplate string An optional filename to use with a multipart/form-data "file upload" style request. If not provided the export data will be sent directly as the HTTP body content. Template parameters are supported.
headers object An optional object whose properties will be included as HTTP headers in the request. Template parameters are supported in property values (not property names).
headersValue string An optional comma-delimited list of colon-delimited key-value pairs to include as HTTP headers in the request. Template parameters are supported in header values (not names).

HTTP destination template parameters

The headers, headersValue, multipartFilenameTemplate, and urlsettings support template parameters, so runtime export attributes can be encoded in the HTTP request. For example a url setting of https://example.com/upload?id={id} would include the export ID in the URL as a query parameter, or a headers settings of {"x-id": "{id}"} would include the export ID as a custom HTTP header.

The supported template parameters are:

Parameter Description
date The current date, in 2019-12-31 form.
ext An appropriate filename extension, for example csv or json.
id The export job ID.
jobName The export job configuration name. The value will be normalized such that sequences of non-alphanumeric characters are replaced by a single _ character.
name The export job configuration name.
now The export process time as a millisecond Unix epoch.

HTTP destination multipart

If the multipartFilenameTemplate setting is configured, the HTTP request will be of type multipart/form-data, commonly referred to as a "file upload" style request. For example, imagine configuring this setting as export-{date}-{now}.{ext}. The HTTP request for this export would look something like this:

Content-Type: multipart/form-data; charset=ISO-8859-1; boundary=ZJXPQJRnPeHRyGnVXOSaoqbNzintPxstN_VKmNzd

--ZJXPQJRnPeHRyGnVXOSaoqbNzintPxstN_VKmNzd
Content-Disposition: form-data; name="export1"; filename="export-2024-03-20-1710959588073.csv.xz"
Content-Type: application/x-xz

...
--ZJXPQJRnPeHRyGnVXOSaoqbNzintPxstN_VKmNzd--

Note the filename in the Content-Disposition multipart header, which comes from the multipartFilenameTemplate setting. The name value will be export followed by a 1-based index, so export1 for the first export resource.

HTTP destination Basic authorization

If both username and password are configured, the HTTP request will include a Basic authorization header.

HTTP destination extra headers

The headers and headersValue settings allow you to include arbitrary HTTP headers in the request. The configured values may contain template parameters as well. For example a headers setting could be defined as:

{
  "x-id": "{id}",
  "x-something-else": "abc123"
}

That same value encoded for the headersValue setting would be a string like this:

x-id: {id}, x-something-else: abc123

S3 destination

ID net.solarnetwork.central.datum.export.dest.s3.S3DatumExportDestinationService

The S3 destination uploads the export result to AWS S3 (or any compatible service). It supports the following settings:

Setting Type Description
path string The full path to the S3 endpoint, bucket, and folder to use. For example s3-ap-southeast-2.amazonaws.com/my-bucket/folder.
filenameTemplate string A template filename to use. This template is allowed to contain parameters in the form {key}, which are replaced at runtime by the value of a parameter key, or an empty string if no such parameter exists. The supported parameters are listed below.
accessKey string The AWS access key to use for uploading the exported data.
secretKey string The AWS secret key to use for uploading the exported data.

S3 destination filename template parameters

The filenameTemplate setting supports the following template parameters:

Parameter Description
date The current date, in 2019-12-31 form.
ext An appropriate filename extension, for example csv or json.
id The export job ID.
jobName The export job configuration name. The value will be normalized such that sequences of non-alphanumeric characters are replaced by a single _ character.
name The export job configuration name.
now The export process time as a millisecond Unix epoch.

List destinations response

The response contains an array of destination service definitions.

Property Type Description
id string The service unique identifier.
settingSpecifiers array The configurable setting specifiers for the service.
locale string The locale of the localized messages in the response.
localizedName string A localized name for the service.
localizedDescription string A localized description of the service.
localizedInfoMessages object Localized messages for the associated setting specifiers.

An example response looks like:

{
  "success": true,
  "data": [
    {
      "id": "net.solarnetwork.central.datum.export.dest.s3.S3DatumExportDestinationService",
      "locale": "en-NZ",
      "settingSpecifiers": [
        {
          "key": "path",
          "defaultValue": "",
          "secureTextEntry": false,
          "transient": false,
          "type": "net.solarnetwork.settings.TextFieldSettingSpecifier"
        },
        {
          "key": "filenameTemplate",
          "defaultValue": "data-export-{date}.{ext}",
          "secureTextEntry": false,
          "transient": false,
          "type": "net.solarnetwork.settings.TextFieldSettingSpecifier"
        },
        {
          "key": "accessKey",
          "defaultValue": "",
          "secureTextEntry": false,
          "transient": false,
          "type": "net.solarnetwork.settings.TextFieldSettingSpecifier"
        },
        {
          "key": "secretKey",
          "defaultValue": "",
          "secureTextEntry": true,
          "transient": false,
          "type": "net.solarnetwork.settings.TextFieldSettingSpecifier"
        }
      ],
      "localizedDescription": "Save exported data to AWS S3.",
      "localizedInfoMessages": {
        "path.key": "S3 Path",
        "path.desc": "The full path to the S3 endpoint, bucket, and folder to use. For example <code>s3-ap-southeast-2.amazonaws.com/my-bucket/folder</code>.",
        "filenameTemplate.key": "Filename",
        "filenameTemplate.desc": "A template filename to use. This template is allowed to contain parameters in the form <code>{key}</code>, which are replaced at runtime by the value of a parameter <code>key</code>, or an empty string if no such parameter exists. The supported parameters are <code>{date}</code> and <code>{ext}</code> (file extension).",
        "accessKey.key": "Access Key",
        "accessKey.desc": "The AWS access key to use for uploading the exported data.",
        "secretKey.key": "Secret Key",
        "secretKey.desc": "The AWS secret key to use for uploading the exported data."
      },
      "localizedName": "S3"
    }
  ]
}

List output formats

This method will list the output formats supported by the SolarNetwork datum export service. Output formats determine what the exported data will be encoded as.

GET /solaruser/api/v1/sec/user/export/services/output

SolarNetwork supports the following output formats:

Destination Description
CSV Format the data as comma separated values (spreadsheet).
JSON Format the data as JSON.

CSV output format

ID net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService

The CSV output format encodes each datum as a CSV row, with datum properties as columns. It supports the following settings:

Setting Type Description
includeHeader boolean Toggle the inclusion of a CSV header row. Defaults to true.

Example output looks like this:

created,nodeId,sourceId,localDate,localTime,watts,wattHours,tags
2014-03-01T08:02:30.561Z,11,Main,2014-03-01,21:02:30.561,820,44555,"foo,reset"
2014-03-01T08:03:30.545Z,11,Main,2014-03-01,21:03:30.545,835,44567,
2014-03-01T08:04:30.545Z,11,Main,2014-03-01,21:04:30.545,856,44578,

JSON output format

ID net.solarnetwork.central.datum.export.standard.JsonDatumExportOutputFormatService

The JSON output format encodes the export as a JSON array of datum JSON objects, with data properties as object keys. It does not support any settings. Example output looks like this:

[
  {"created":"2014-03-01 08:02:30.561Z","nodeId":11,"sourceId":"Main","localDate":"2014-03-01","localTime":"21:02","watts":820,"wattHours":44555,"tags":["foo","reset"]},
  {"created":"2014-03-01 08:03:30.545Z","nodeId":11,"sourceId":"Main","localDate":"2014-03-01","localTime":"21:03","watts":835,"wattHours":44567},
  {"created":"2014-03-01 08:04:30.545Z","nodeId":11,"sourceId":"Main","localDate":"2014-03-01","localTime":"21:04","watts":856,"wattHours":44578}
]

List output formats response

The response contains an array of output format service definitions.

Property Type Description
id string The service unique identifier.
settingSpecifiers array The configurable setting specifiers for the service.
locale string The locale of the localized messages in the response.
localizedName string A localized name for the service.
localizedDescription string A localized description of the service.
localizedInfoMessages object Localized messages for the associated setting specifiers.

An example response looks like:

{
  "success": true,
  "data": [
    {
      "id": "net.solarnetwork.central.datum.export.standard.CsvDatumExportOutputFormatService",
      "locale": "en-NZ",
      "settingSpecifiers": [
        {
          "key": "includeHeader",
          "defaultValue": true,
          "trueValue": true,
          "falseValue": false,
          "transient": false,
          "type": "net.solarnetwork.settings.ToggleSettingSpecifier"
        }
      ],
      "localizedDescription": "Export data in comma separated values (spreadsheet) format.",
      "localizedInfoMessages": {
        "includeHeader.key": "Include Header",
        "includeHeader.desc": "Toggle the inclusion of a CSV header row."
      },
      "localizedName": "CSV"
    },
    {
      "id": "net.solarnetwork.central.datum.export.standard.JsonDatumExportOutputFormatService",
      "locale": "en-NZ",
      "settingSpecifiers": [],
      "localizedDescription": "Export data in JSON format.",
      "localizedInfoMessages": {},
      "localizedName": "JSON"
    }
  ]
}
Add a custom footer
Clone this wiki locally