The ioboxd
service provides export functionality from an ERMrest catalog via a REST API.
The service is generally configured to be co-located on the same server with the ERMrest catalog that it is providing data services for.
This API endpoint is used to perform one or more queries to an ERMrest catalog and create corresponding individual
result files which can then be retrieved via GET
. URLs for result files are returned in the response body as Content-Type:text/uri-list
.
Executes one more ERMrest queries and writes the results to individual files.
/iobox/export/file
POST
None
The input data is composed of a JSON object with the following form:
Variable | Type | Inclusion | Description |
---|---|---|---|
catalog |
catalog |
required | A catalog object. See below. |
Variable | Type | Inclusion | Description |
---|---|---|---|
host |
string | required | The hostname (and port) of the ERMrest service to query. |
catalog_id |
string | optional | The catalog identifier, e.g. 1 . |
username |
string | optional | If username is not specified, authentication is assumed to come from the webauthn authentication context stored in the caller's cookie. |
password |
string | optional | The password field is only used when username is specifed. |
queries |
array[query ] |
required | An array of query objects. See below. |
Parent Object | Variable | Type | Inclusion | Description | Interpolatable |
---|---|---|---|---|---|
query | processor | string | required | This is a string value used to select from one of the built-in query output processor formats. Valid values are env , csv , json , json-stream , download , or fetch . |
No |
query | processor_type | string | optional | A fully qualified Python class name declaring an external processor class instance to use. If this parameter is present, it OVERRIDES the default value mapped to the specified processor . This class MUST be derived from the base class deriva.transfer.download.processors.BaseDownloadProcessor . For example, "processor_type": "deriva.transfer.download.processors.CSVDownloadProcessor" . |
No |
query | processor_params | object | required | This is an extensible JSON Object that contains processor implementation-specific parameters. | No |
processor_params | query_path | string | required | This is string representing the actual ERMRest query path to be used in the HTTP(S) GET request. It SHOULD already be percent-encoded per RFC 3986 if it contains any characters outside of the unreserved set. |
Yes |
processor_params | output_path | string | required | This is a POSIX-compliant path fragment indicating the target location of the retrieved data relative to the specified base download directory. | Yes |
Code: 200
Content:
http://localhost:8080/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/genotypes.csv
http://localhost:8080/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/phenotypes.csv
- 404: NOT FOUND
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
var exportParameters =
{
"catalog":
{
"host": "http://localhost:8080",
"catalog_id": "1",
"username": "devuser",
"password": "devpass",
"query_processors": [
{
"processor": "csv",
"processor_params": {
"query_path": "/entity/A:=dev:subject/A1:=snp_v/snp_id=rs6265",
"output_path": "genotypes",
}
},
{
"processor": "csv",
"processor_params": {
"query_path": "/entity/A:=dev:subject/A1:=snp_v/snp_id=rs6265/$A/B:=dev:subject_phenotypes_v",
"output_path": "phenotypes",
}
}
]
}
};
$.ajax({
url: "/iobox/export/file",
dataType: "json",
type : "POST",
data: exportParameters,
success : function(r) {
console.log(r);
}
});
Retrieves a file previously created by a POST
.
/iobox/export/file/<id>/<filename>
GET
Required:
id=[string]
Optional:
filename=[string]
- This argument is required when the uri-list
returned from POST
contains more than one entry.
If it is not specified and there is more than one file result, a 400 Bad Request
is returned.
None
Code: 200
Content: The file content.
- 404: NOT FOUND
- 403: FORBIDDEN
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
$.ajax({
url: "/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/genotypes.csv",
type : "GET",
success : function(r) {
console.log(r);
}
});
Retrieves the log file generated by an invocation of POST
.
/iobox/export/file/<id>/log
GET
Required:
id=[string]
None
Code: 200
Content: The file content.
- 404: NOT FOUND
- 403: FORBIDDEN
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
$.ajax({
url: "/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/log",
type : "GET",
success : function(r) {
console.log(r);
}
});
This API endpoint is used to perform one or more queries to an ERMrest catalog and create a
BDBag archive file which can then be retrieved via GET
. The URL for the result bag is returned in the response body as Content-Type:text/uri-list
.
Executes one more ERMrest queries and writes the results to a BDBag.
/iobox/export/bdbag
POST
None
The input data is composed of a JSON object with the following form:
Variable | Type | Inclusion | Description |
---|---|---|---|
bag |
bag |
required | A bag object. See below. |
catalog |
catalog |
required | A catalog object. See below. |
Variable | Type | Inclusion | Description |
---|---|---|---|
bag_name |
string, enum ["zip" ,"tgz" ,"bz2" ,"tar" ] |
required | The base file name of the bag. An appropriate extension will be added to the base depending on the archive type selected. |
bag_archiver |
string | required | The archive format used to serialize the result bag. |
bag_metadata |
object | optional | A simple 'dictionary' object consisting of key-value pairs. The only supported primitive type for value pairs is string . The metdata object will be written directly to the bag's bag-info.txt file. |
Variable | Type | Inclusion | Description |
---|---|---|---|
host |
string | required | The hostname (and port) of the ERMrest service to query. |
catalog_id |
string | optional | The catalog identifier, e.g. 1 . |
username |
string | optional | If username is not specified, authentication is assumed to come from the webauthn authentication context stored in the caller's cookie. |
password |
string | optional | The password field is only used when username is specifed. |
queries |
array[query ] |
required | An array of query objects. See below. |
Parent Object | Variable | Type | Inclusion | Description | Interpolatable |
---|---|---|---|---|---|
query | processor | string | required | This is a string value used to select from one of the built-in query output processor formats. Valid values are env , csv , json , json-stream , download , or fetch . |
No |
query | processor_type | string | optional | A fully qualified Python class name declaring an external processor class instance to use. If this parameter is present, it OVERRIDES the default value mapped to the specified processor . This class MUST be derived from the base class deriva.transfer.download.processors.BaseDownloadProcessor . For example, "processor_type": "deriva.transfer.download.processors.CSVDownloadProcessor" . |
No |
query | processor_params | object | required | This is an extensible JSON Object that contains processor implementation-specific parameters. | No |
processor_params | query_path | string | required | This is string representing the actual ERMRest query path to be used in the HTTP(S) GET request. It SHOULD already be percent-encoded per RFC 3986 if it contains any characters outside of the unreserved set. |
Yes |
processor_params | output_path | string | required | This is a POSIX-compliant path fragment indicating the target location of the retrieved data relative to the specified base download directory. | Yes |
Code: 200
Content:
http://localhost:8080/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/sample-bag.zip
- 404: NOT FOUND
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
var exportParameters =
{
"bag":
{
"bag_name": "sample-bag",
"bag_archiver":"zip",
"bag_metadata":
{
"Source-Organization": "USC Information Sciences Institute, Informatics Systems Research Division",
"Contact-Name": "Mike D'Arcy",
"External-Description": "A bag containing a sample PheWas cohort for downstream analysis.",
"Internal-Sender-Identifier": "USC-ISI-IRSD"
}
},
"catalog":
{
"host": "https://localhost:8080",
"path": "/ermrest/catalog/1",
"username": "",
"password": "",
"queries":
[
{
"processor": "csv",
"query_path": "/entity/A:=pnc:subject/A1:=snp_v/snp_id=rs6265/$A/B:=pnc:metrics_v",
"output_path": "metrics"
},
{
"processor": "csv",
"processor_params": {
"query_path": "/entity/A:=pnc:subject/A1:=snp_v/snp_id=rs6265",
"output_path": "genotypes"
}
},
{
"processor": "csv",
"processor_params": {
"query_path": "/entity/A:=pnc:subject/A1:=snp_v/snp_id=rs6265/$A/B:=pnc:subject_phenotypes_v",
"output_path": "phenotypes"
}
},
{
"processor": "fetch",
"processor_params": {
"query_path": "/attribute/A:=pnc:subject/A1:=snp_v/snp_id=rs6265/$A/B:=pnc:image_files/filename::ciregexp::0mm.mgh/url:=B:uri,length:=B:bytes,filename:=B:filepath,sha256:=B:sha256sum",
"output_path": "images"
}
}
]
}
};
$.ajax({
url: "/iobox/export/file",
dataType: "json",
type : "POST",
data: exportParameters,
success : function(r) {
console.log(r);
}
});
Retrieves a bag previously created by a POST
.
/iobox/export/bdbag/<id>
GET
Required:
id=[string]
None
Code: 200
Content: The file content.
- 404: NOT FOUND
- 403: FORBIDDEN
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
$.ajax({
url: "/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc",
type : "GET",
success : function(r) {
console.log(r);
}
});
Retrieves the log file generated by an invocation of POST
.
/iobox/export/bdbag/<id>/log
GET
Required:
id=[string]
None
Code: 200
Content: The file content.
- 404: NOT FOUND
- 403: FORBIDDEN
- 401: UNAUTHORIZED
- 400: BAD REQUEST
- 500: INTERNAL SERVER ERROR
$.ajax({
url: "/iobox/export/file/9ad15e5b-9c2c-4faf-8829-05fa8252c8bc/log",
type : "GET",
success : function(r) {
console.log(r);
}
});