Permalink
Fetching contributors…
Cannot retrieve contributors at this time
4482 lines (3634 sloc) 164 KB
FORMAT: 1A
HOST: https://connection.keboola.com
# Storage API
Storage API provides an interface for sending data to Keboola Connection (KBC). The following are the most
important features of the API:
- Importing CSV data into Table Storage
- Exporting CSV data from Table Storage
- Loading arbitrary files into File Storage
- Managing tables including keys and indexes
- Managing buckets including sharing
- Managing component configurations
The CSV files interchanged with Storage must adhere to the
[RFC4180 Specification](http://tools.ietf.org/html/rfc4180) and must use the `UTF-8` encoding.
The files can be sent uncompressed or gzipped. You can find more information about working
with these files in [our manual](https://help.keboola.com/storage/tables/csv-files/).
For a list of available clients and their features, see the
[Developers Documentation](https://developers.keboola.com/integrate/storage/).
In it you will also find more complex examples of
[working with the API](https://developers.keboola.com/integrate/storage/api/).
This documentation assumes that you are already familiar with the
[Keboola Connection Storage component](https://help.keboola.com/storage/).
## HTTP Response Codes
The response from Storage API will have an
[HTTP status code](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) that will help
you determine if the request was successful. In case of an error, the HTTP status code
will help you determine the cause of the error. The status codes used by Storage API are
listed in the below tables.
### Success responses
<table>
<tr>
<td>200 <code>OK</code></td>
<td>The request was successful.</td>
</tr>
<tr>
<td>201 <code>Created</code></td>
<td>The request was successful and a new resource was created.</td>
</tr>
<tr>
<td>202 <code>Accepted</code></td>
<td>Used for <a href='#introduction/synchronous-and-asynchronous-calls'>Asynchronous tasks</a>. A job resource is returned. The actual result must be obtained in another API call.</td>
</tr>
<tr>
<td>204 <code>No Content</code></td>
<td>The request was successful but there is nothing to return. Usually used as a response of DELETE requests.</td>
</tr>
</table>
### Error responses
<table>
<tr>
<td>400 <code>Bad Request</code></td>
<td>The request was invalid. Usually caused by invalid input data (missing arguments, invalid arguments values, etc.). The cause of the error is described in the response.</td>
</tr>
<tr>
<td>401 <code>Unauthorized</code></td>
<td>Authentication failed.</td>
</tr>
<tr>
<td>403 <code>Forbidden</code></td>
<td>You don't have access to the resource.</td>
</tr>
<tr>
<td>404 <code>Not Found</code></td>
<td>You're asking for something that doesn't exist.</td>
</tr>
<tr>
<td>500 <code>Internal Server Error</code></td>
<td>Something went wrong. We are sorry, it is our fault and we will make our best to fix it! Feel free to <a href='mailto:suppor@keboola.com'>send us a ticket.</a></td>
</tr>
<tr>
<td>503 <code>Temporary Unavailable</code></td>
<td>This response is typically returned when the system is under maintenance.
The maintenance reason and expected maintenance end time are also returned in the response.
</td>
</tr>
</table>
## Authentication
Authentication is done via Storage API **Token**. There are multiple options
for [obtaining a Storage API Token](https://help.keboola.com/storage/tokens/).
The token must be sent as a value of the `X-StorageApi-Token` HTTP header with each API call.
Most API calls work within a *KBC Project*. The project is always derived from the token used for the API call.
So the token includes both authentication and authorization to a single project.
## Synchronous and Asynchronous Calls
Calls that represent potentially long-running actions are performed asynchronously.
These are for example: loading table data, snapshotting, exporting table data, table structure modifications, etc.
All asynchronous calls return the HTTP Response code `202` and a [Job Resource](#reference/jobs).
To obtain the actual result of the call, you have to monitor the job status by
[polling the Job Resource URL](https://developers.keboola.com/integrate/storage/api/import-export/#python-example).
## Partial Response
Some API calls that list objects are returning related resources by default. For example,
the [table list](#reference/tables/create-or-list-tables/tables-in-bucket) also returns
the table bucket properties and table attributes. You can use the `include` parameter
to return only the required resources in the response.
# Group Maintenance
## Maintenance Response Example [/v2/storage/]
### Example [GET]
In case the system, or a project is under maintenance, you will receive a response similar to the below one.
Maintenance typically applies to a single project, for example due to migration. Be sure to use the `Retry-After` header
when accessing the API programatically.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 503
+ Headers
Content-type: application/json
Retry-After: 559
+ Body
{
"status":"maintenance",
"estimatedEndTime":"2013-05-25T10:30:44+02:00",
"reason":"Thesystemisdownformaintenance",
}
# Group Tokens and Permissions
When joining a project, each project administrator is assigned a [*master* token](https://help.keboola.com/storage/tokens/),
which enables them to create other tokens with limited privileges for buckets and components.
## Tokens Collection [/v2/storage/tokens]
### List all tokens [GET]
Lists all tokens in the project.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": "38",
"token": "9eabdd51eaab4040f952e0e6ae6eff03",
"description": "",
"created": "2012-11-13T16:49:28+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"uri": "https://connection.keboola.com/v2/storage/tokens/38",
"isMasterToken": true,
"canManageBuckets": true,
"canManageTokens": false,
"bucketPermissions": {
"in.c-main": "manage",
"out.c-main": "manage",
"in.c-ga": "manage"
}
},
{
"id": "40",
"token": "68ab920b621f159f72dcce4ae39a9a0d",
"description": "",
"created": "2012-11-13T16:49:28+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"uri": "https://connection-devel.keboola.com/v2/storage/tokens/40",
"isMasterToken": false,
"canManageBuckets": false,
"canManageTokens": false,
"bucketPermissions": {
"in.c-main": "read"
},
"componentAccess": [
"someComponentName"
]
}
]
### Create token [POST]
Creates a new token in the project. Note that it is not possible to create a new master token.
+ Attributes
+ description (required) - Token description
+ bucketPermissions[{id_bucket}] (optional) - Permissions for a bucket. Available permissions: `read`, `write`.
+ componentAccess[] (optional) - Grants access for component configurations. Allowed
values are [valid component IDs](#reference/miscellaneous/api-index/get).
+ canManageBuckets (optional, boolean) - Allows full access to all buckets including newly created buckets.
Overrides `bucketPermissions` settings.
+ Default: false
+ canReadAllFileUploads (optional, boolean) - Allows access to all file uploads. When set to false,
only files uploaded by the current token are accessible.
+ Default: false
+ expiresIn (optional, number) - Number of seconds until the token expires. If not provided, the token never expires.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
description=my-new-token&bucketPermissions[in.c-main]=read&componentAccess[]=keboola.ex-db-mysql&expiresIn=60
+ Response 201 (application/json)
+ Body
{
"id": "484",
"token": "75-12d4bfa4995694a1c956e7bb1eea0df5cc53acdb",
"created": "2017-11-13T16:48:28+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"description": "my-new-token",
"uri": "https://connection.keboola.com/v2/storage/tokens/484",
"isMasterToken": false,
"canManageBuckets": false,
"canManageTokens": false,
"bucketPermissions": {
"in.c-main": "read"
},
"componentAccess": [
"keboola.ex-db-mysql"
],
"expires": "2012-11-13T16:49:28+0100",
"isExpired": false,
"isDisabled": false,
"dailyCapacity": 5
}
## Token [/v2/storage/tokens/{id_token}]
### Token detail [GET]
Returns all properties of a token. You can obtain the Token Id, for example, using
the [Verify call](#reference/tokens-and-permissions/token/token-verification).
+ Parameters
+ id_token (required, number) - Token Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
{
"id": "44",
"token": "bdc4ce6a0ea85a2f4313b5b0b1800df3",
"description": "",
"created": "2017-11-13T16:48:28+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"uri": "https://connection-devel.keboola.com/v2/storage/tokens/44",
"isMasterToken": false,
"bucketPermissions": [],
"owner": {
"name": "Buckets 5",
"id": "58"
}
}
### Update Token [PUT]
Updates Token properties. Beware that all current bucket permissions are replaced. If you want to add new
bucket permissions, don't forget to specify the previous permissions.
+ Attributes
+ description (required) - Token description
+ bucketPermissions[{id_bucket}] (optional) - Permissions for the bucket. Available permissions: `read`, `write`.
+ componentAccess[] (optional) - Grants access for component configurations. Allowed
values are [valid component IDs](#reference/miscellaneous/api-index/get).
+ canReadAllFileUploads (optional, boolean) - Allows access to all file uploads. When set to false,
only files uploaded by the current token are accessible.
+ Default: false
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
description=New Description&bucketPermissions[in.c-main]=write
+ Response 200 (application/json)
+ Body
{
"id": "67319",
"token": "578-67319-460300c259ec3fcba4b95d1d5c14a8419268bc2b",
"created": "2017-02-13T10:27:50+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"description": "new-description",
"uri": "https://connection.keboola.com/v2/storage/tokens/67319",
"isMasterToken": false,
"canManageBuckets": false,
"canManageTokens": false,
"canReadAllFileUploads": false,
"expires": null,
"isExpired": false,
"isDisabled": false,
"dailyCapacity": 5,
"creatorToken": {
"id": 27978,
"description": "ondrej.popelka@keboola.com"
},
"componentAccess": [],
"bucketPermissions": {
"in.c-main": "write"
}
}
## Token Verification [/v2/storage/tokens/verify]
### Token verification [GET]
Checks the token privileges and returns information about the project to which the token belongs (`owner`)
and the associated administrator (`admin`). This call can be executed by all tokens.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
{
"id": "44",
"token": "your_token",
"description": "",
"created": "2017-02-13T10:27:50+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"uri": "https://connection-devel.keboola.com/v2/storage/tokens/44",
"isMasterToken": false,
"bucketPermissions": [],
"componentPermissions": [],
"owner": {
"name": "Buckets 5",
"id": "58",
"limits": {
"components.jobsParallelism": {
"name": "components.jobsParallelism",
"value": 10
},
"goodData.dataSizeBytes": {
"name": "goodData.dataSizeBytes",
"value": 1000000000
},
"goodData.demoTokenEnabled": {
"name": "goodData.demoTokenEnabled",
"value": 1
},
"goodData.prodTokenEnabled": {
"name": "goodData.prodTokenEnabled",
"value": 1
},
"goodData.usersCount": {
"name": "goodData.usersCount",
"value": 30
},
"kbc.adminsCount": {
"name": "kbc.adminsCount",
"value": 10
},
"kbc.extractorsCount": {
"name": "kbc.extractorsCount",
"value": 0
},
"kbc.writersCount": {
"name": "kbc.writersCount",
"value": 0
},
"orchestrations.count": {
"name": "orchestrations.count",
"value": 10
},
"storage.dataSizeBytes": {
"name": "storage.dataSizeBytes",
"value": 50000000000
},
"storage.jobsParallelism": {
"name": "storage.jobsParallelism",
"value": 10
}
},
"metrics": {
"goodData.dataSizeBytes": {
"name": "goodData.dataSizeBytes",
"value": 3589276
},
"goodData.rowsCount": {
"name": "goodData.rowsCount",
"value": 23898
},
"goodData.usersCount": {
"name": "goodData.usersCount",
"value": 0
},
"storage.dataSizeBytes": {
"name": "storage.dataSizeBytes",
"value": 3019882496
},
"storage.rowsCount": {
"name": "storage.rowsCount",
"value": 3316230
}
}
}
}
## Share Token [/v2/storage/tokens/{token_id}/share]
### Share Token [POST]
Use this call whenever you want to securely deliver a token to someone. The link to the token retrieval page will be sent to the
provided email address. The link will expire in 2 hours.
+ Parameters
+ token_id (required, number) - Token Id
+ Attributes
+ recipientEmail (required) - Recipient's email address
+ message (required) - Message for the token recipient
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
recipientEmail=martin@keboola.com&message=Hi
+ Response 204
## Token Refresh [/v2/storage/tokens/{id_token}/refresh]
### Refresh Token [POST]
Use this method to generate a new token value; the old token value will become immediately invalid.
This method can be executed by all tokens.
+ Parameters
+ id_token - A valid token id. You can get it from [Verify call](#reference/tokens-and-permissions/token/token-verification).
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
{
"id": "48",
"token": "new_token",
"description": "Master Token",
"created": "2017-02-13T10:27:50+0100",
"refreshed": "2017-11-13T16:48:28+0100",
"uri": "https://connection-devel.keboola.com/v2/storage/tokens/id_token",
"isMasterToken": false,
"bucketPermissions": {
"in.c-main": "manage",
"out.c-main": "manage",
"in.c-twitter": "manage"
}
}
# Group Miscellaneous
## API Index [/v2/storage]
### Component List [GET]
Use this API call to obtain definitions of all components and services available in KBC. For more information about
the KBC Component architecture, see the [Developers documentation](https://developers.keboola.com/overview/).
+ Response 200 (application/json)
+ Body
{
"api": "storage",
"documentation": "http://docs.keboola.apiary.io/",
"version": "v2",
"components": [
{
"id": "orchestrator-2",
"uri": "https://orchestrator-2.keboola.com/"
},
{
"id": "transformation",
"uri": "https://transformation.keboola.com/"
}
],
"services": [
{
"id": "docker-runner",
"url": "https:\/\/docker-runner.keboola.com"
},
{
"id": "import",
"url": "https:\/\/import.keboola.com"
},
{
"id": "syrup",
"url": "https:\/\/syrup.keboola.com"
}
],
"urlTemplates": {
"orchestrationJob": "\/admin\/projects\/&&projectId&&\/orchestrations\/&&orchestrationId&&\/jobs\/&&jobId&&"
}
}
# Group Buckets
[Buckets](https://help.keboola.com/storage/buckets/) are containers for one or more data tables.
Access to buckets can be limited by access tokens. Each bucket has a *backend* in which all tables are created:
- Snowlake (default)
- Redshift
## Create or List Buckets [/v2/storage/buckets]
### List all buckets [GET /v2/storage/buckets?include={include}]
All buckets with their attributes are returned.
+ Parameters
+ include (optional) - Comma-separated list of resources to include for each table. Available resources: 'attributes, metadata, linkedBuckets'.
+ Default: 'attributes'
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-main",
"id": "in.c-main",
"name": "c-main",
"stage": "in",
"description": "Main user storage",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-main/tables",
"backend": "snowflake"
"linkedBy": [
{
"id": "in.c-linked-bucket",
"created": "2017-02-13T12:01:05+0100",
"project": {
"id": 321,
"name": "Some Project"
}
}, {
"id": "in.c-another-linked-bucket",
"created": "2017-02-13T12:01:05+0100",
"project": {
"id": 322,
"name": "Some Other Project"
}
}
]
},
{
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-organizationData",
"id": "in.c-organizationData",
"name": "c-organizationData",
"stage": "in",
"description": "Source bucket description",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-organizationData/tables",
"backend": "snowflake",
"isReadonly": true,
"sourceBucket": {
"id": "in.c-main",
"name": "c-main",
"description": "Organization shared data"
"project": {
"id": 123,
"name": "Project name"
}
}
}
]
### Create Bucket [POST]
Using this API call, you can either create a new bucket in the project, or you can link an
existing bucket from another project (see below).
+ Attributes
+ name (required) - New bucket name; only alphanumeric characters and underscores are allowed.
+ stage (optional, enum[string]) - Assigns the bucket to one of the stages.
+ Members
+ in
+ out
+ Default: in
+ description (optional) - Bucket description.
+ backend (optional, enum[string]) - Bucket backend type; the default value is determined by the project settings.
+ Members
+ snowflake
+ redshift
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
name=my-new-bucket&stage=in&description=Some Description
+ Response 201 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-my-new-bucket",
"id": "in.c-my-new-bucket",
"name": "c-my-new-bucket",
"stage": "in",
"description": "Some Description",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-my-new-bucket/tables",
"created": "2017-02-13T12:01:05+0100",
"lastChangeDate": null,
"isReadOnly": false,
"dataSizeBytes": 0,
"rowsCount": 0,
"isMaintenance": false,
"backend": "snowflake",
"sharing": null,
"attributes": []
}
### Link Shared Bucket [POST]
Use this API call to create a new bucket which contains contents of a
[shared bucket](https://help.keboola.com/storage/buckets/sharing/) in a source project.
Linking a bucket from another project is only possible if it has been
[enabled](#reference/buckets/bucket-sharing/share-bucket) in the source project.
+ Attributes
+ name (required) - New bucket name; only alphanumeric characters and underscores are allowed.
+ sourceProjectId (required, number) - Id of the source project from which the bucket is being shared
+ sourceBucketId (required) - Id of the bucket being shared
+ stage (optional, enum[string]) - Assigns the bucket to one of the stages.
+ Members
+ in
+ out
+ Default: in
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
name=organizationData&stage=in&sourceProjectId=123&sourceBucketId=in.c-shared-bucket
+ Response 201 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-organizationData",
"id": "in.c-organizationData",
"name": "c-organizationData",
"stage": "in",
"description": "Source bucket description",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-organizationData/tables",
"backend": "snowflake",
"isReadonly": true,
"sourceBucket": {
"id": "in.c-shared-bucket",
"name": "c-shared-bucket",
"description": "Organization shared data"
"project": {
"id": 123,
"name": "Project name"
}
}
}
## Manage Bucket [/v2/storage/buckets/{bucket_id}]
### Bucket Detail [GET]
Obtains information about a bucket.
+ Parameters
+ bucket_id (required) - Bucket Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-ga",
"id": "in.c-ga",
"name": "c-ga",
"stage": "in",
"description": "Google Analytics",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-ga/tables",
"backend": "snowflake",
"linkedBy": [
{
"id": "in.c-linked-bucket",
"created": "2017-02-13T12:01:05+0100",
"project": {
"id": 321,
"name": "Some Project"
}
}, {
"id": "in.c-another-linked-bucket",
"created": "2017-02-13T12:01:05+0100",
"project": {
"id": 322,
"name": "Some Other Project"
}
}
]
}
### Drop Bucket [DELETE /v2/storage/buckets/{bucket_id}/?force={force}]
Deletes a bucket from the project. In the default mode, only empty buckets can be deleted.
Use the optional `force` parameter to delete all bucket content too.
+ Parameters
+ bucket_id (required) - Bucket Id
+ force (optional, boolean) - Drops all tables and aliases. Tables must not have any dependencies (aliases, etc.).
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Bucket Sharing [/v2/storage/buckets/{bucket_id}/share]
### Share Bucket [POST]
[Enables sharing](https://help.keboola.com/storage/buckets/sharing/) of a bucket. The bucket will be shared
to the entire organization to which the project belongs. It may then be shared to any project of that organization.
This operations is available only for organization administrators.
+ Parameters
+ bucket_id (required) - Bucket Id
+ Attributes
+ sharing (optional, enum[string]) - Sharing type
+ Members
+ organization - Bucket is shared to organization. Any organization member can link bucket to project in organization.
+ `organization-project` - Bucket is shared to organization. Any organization or organization's project member can link bucket to project.
+ Default: organization
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"sharing": "organization"
}
### Change Bucket Sharing [PUT]
Change [sharing type](https://help.keboola.com/storage/buckets/sharing/#sharing-type) of a bucket.
This operations is available only for organization administrators.
+ Parameters
+ bucket_id (required) - Bucket Id
+ Attributes
+ sharing (required, enum[string]) - Sharing type
+ Members
+ `organization` - Bucket is shared to organization. Any organization member can link bucket to project in organization.
+ `organization-project` - Bucket is shared to organization. Any organization or organization's project member can link bucket to project.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"sharing": "organization"
}
### Stop Bucket Sharing [DELETE]
Disables sharing of a bucket. A bucket must not be linked to other projects.
To unshare an already linked bucket, you have to delete the links first - use the
[Drop Bucket](#reference/buckets/manage-bucket/drop-bucket) in the linking project.
This operations is available only for organization administrators.
+ Parameters
+ bucket_id (required) - Bucket Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## List Shared Buckets [/v2/storage/shared-buckets]
Lists buckets which may be shared to the project.
### Shared buckets list [GET]
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": "in.c-main",
"name": "c-main",
"description": "Organization shared data"
"project": {
"id": 123,
"name": "Project name"
}
}
]
# Group Tables
## Create or List Tables [/v2/storage/buckets/{bucket_id}/tables]
### Create new table from CSV file [POST]
Creates a new table in a bucket. The CSV file must follow the [RFC 4180 Specification](https://tools.ietf.org/html/rfc4180).
More information about the requirements can be found in the [Manual](https://help.keboola.com/storage/tables/csv-files/).
Column names are extracted from the CSV file header; spaces and any other special characters in column names are
replaced with an underscore. This API call should be used only for tables which can be safely transferred in a single HTTP request; for large
tables use the [asynchronous call](#reference/tables/create-table-asynchronously/create-new-table-from-csv-file-asynchronously).
+ Parameters
+ bucket_id (required) - Bucket Id
+ Attributes
+ name (required) - New table name
+ data (required) - CSV file containing data for import. The data have to be encoded in **UTF-8**. The file can be raw or gzipped. Gzipped files must have the extension `.gz`.
+ delimiter (optional) - Field delimiter used in the CSV file. The default value is ','. Use '\t' or type the char for the tabulator.
+ enclosure (optional) - Field enclosure used in the CSV file. The default value is '"'. (Note: you can specify either `enclosure` or `escapedBy` parameter, not both.)
+ escapedBy (optional) - Escape character used in the CSV file. The default value is an empty value - no escape character is used. (Note: you can specify either `enclosure` or `escapedBy` parameter, not both.)
+ primaryKey (optional) - Primary key of the table. If the primary key is set, updates can be done on the table. See the CSV incremental import. The primary key can be also composed by multiple columns; the columns must be separated by a comma.
+ Request (multipart/form-data)
+ Headers
X-StorageApi-Token: your_token
+ Body
------WebKitFormBoundaryqPuY88H6m4oIp6zq
Content-Disposition: form-data; name="name"
my_table_name
------WebKitFormBoundaryqPuY88H6m4oIp6zq
Content-Disposition: form-data; name="data"; filename="my_table_name.csv"
Content-Type: application/octet-stream
file_content
------WebKitFormBoundaryqPuY88H6m4oIp6zq
Content-Disposition: form-data; name="delimiter"
,
------WebKitFormBoundaryqPuY88H6m4oIp6zq
Content-Disposition: form-data; name="enclosure"
"
------WebKitFormBoundaryqPuY88H6m4oIp6zq--
+ Response 201 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.languages",
"id": "in.c-main.languages",
"name": "languages",
"gdName": "languages",
"created": "2012-07-23 09:51:45",
"lastImportDate": null,
"columns": [
"id",
"name_id",
"nm_id"
],
"indexedColumns": [
"id"
],
"bucket": {
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-main",
"id": "in.c-main",
"name": "c-main",
"stage": "in",
"description": "Main user storage",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-main/tables"
}
}
### Tables in bucket [GET /v2/storage/buckets/{bucket_id}/tables?include={include}]
Lists tables in a given bucket.
+ Parameters
+ bucket_id (required) - Bucket Id
+ include (optional, enum[string]) - Comma-separated list of resources to include for each table
+ Members
+ attributes
+ columns
+ Default: attributes
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.tweets",
"id": "in.c-main.tweets",
"name": "tweets",
"gdName": "",
"created": "2012-07-13 09:58:47",
"lastImportDate": "2012-07-13 10:00:42",
"backend": "snowflake"
}
]
## Create Table Asynchronously [/v2/storage/buckets/{bucket_id}/tables-async]
### Create new table from CSV file asynchronously [POST]
The method of creating a table asynchronously should be used for larger imports (everything with 100 lines and more).
The create request is created and added to a queue, the job resource url for status polling is returned.
The actual data must be provided in one of the following attributes:
- `dataFileId`,
- `snapshotId`,
- `dataWorkspaceId` and `dataTableName`
- `sourceTableId`
+ Parameters
+ bucket_id (required) - The bucket Id
+ Attributes
+ name (required) - New table name
+ dataFileId (optional) - Id of the file stored in [File Uploads](#reference/files)
+ snapshotId (optional) - Id of a table snapshot - a table will be created from the snapshot.
+ dataWorkspaceId (optional) - Load from the table [workspace](#reference/workspaces). Use with the **dataTableName** attribute.
+ dataTableName (optional) - Load from a table in [workspace](#reference/workspaces).
+ sourceTableId (optional) - Create time travel snapshot of this table.
+ timestamp (optional) - Datetime string to be used for the time travel snapshot. Use with the **sourceTableId** attribute. The preferred format is `YYYY-MM-DD hh:mm:ss Z`, other formats may also work, but time zone is required.
+ delimiter (optional) - Field delimiter used in the CSV file. The default value is ','. Use '\t' or type the tab char for the tabulator.
+ enclosure (optional) - Field enclosure used in the CSV file. The default value is '"'. (Note: you can specify either the `enclosure` or `escapedBy` parameter, not both.)
+ escapedBy (optional) - Escape character used in the CSV file. The default value is an empty value - no escape character is used. (Note: you can specify either the `enclosure` or `escapedBy` parameter, not both.)
+ primaryKey (optional) - Primary key of a table. If the primary key is set, updates can be done on the table. See CSV incremental import. The primary key can be also composed by multiple columns; the columns must be separated by a comma.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
name=table_name&dataFileId=234delimiter=%2C&enclosure=%22
+ Response 202 (application/json)
{
"id": 11,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/11",
"operationName": "tableCreate",
"operationParams": {
"params": {
"enclosure": "\"",
"delimiter": ",",
"escapedBy": "",
},
"source": {
"fileId": "234",
"type": "file"
}
},
"createdTime": "2013-05-31T16:11:05+0200",
"crea
"startTime": null,
"endTime": null
}
## Load Data [/v2/storage/tables/{table_id}/import]
### Load data from csv file to prepared table [POST]
**This method of import is DEPRECATED**, please use [asynchronous import](#reference/tables/load-data-asynchronously).
Each CSV table should contain a header with column names; all columns present in the table must be also present in the CSV file, new columns not present in the table are created. One of the `data` and `dataString` attributes must be provided.
The CSV file can be raw or gzipped. Gzipped files must have the extension .gz.
+ Attributes
+ data (optional) - Multipart CSV file upload
+ dataString (optional) - CSV as a string in the request body
+ incremental (optional) - If incremental is set to 0, the target table is truncated before each import. Default is 0.
+ delimiter (optional) - Field delimiter used in the CSV file. The default value is ','. Use '\t' or type the tab char for the tabulator.
+ enclosure (optional) - Field enclosure used in the CSV file. The default value is '"'.
+ columns[] (optional) - List of columns present in the CSV file; the first line of the file will not be treated as a header!
+ withoutHeaders (optional, boolean) - The CSV file doesn't contain headers, columns are matched by their order. If this option is used, columns option is ignored.
+ Request (multipart/form-data; boundary=----WebKitFormBoundaryU2xN082HVaIRptvd)
+ Headers
X-StorageApi-Token: your_token
+ Body
------WebKitFormBoundaryU2xN082HVaIRptvd
Content-Disposition: form-data; name="incremental"
0
------WebKitFormBoundaryU2xN082HVaIRptvd
Content-Disposition: form-data; name="data"; filename="tw_accounts.csv"
Content-Type: application/octet-stream
file_content
------WebKitFormBoundaryU2xN082HVaIRptvd
Content-Disposition: form-data; name="delimiter"
,
------WebKitFormBoundaryU2xN082HVaIRptvd
Content-Disposition: form-data; name="enclosure"
"
------WebKitFormBoundaryU2xN082HVaIRptvd
+ Response 200 (application/json)
{
"status": "ok",
"warnings": []
}
## Load data asynchronously [/v2/storage/tables/{table_id}/import-async]
### Import data [POST]
The method of asynchronous importing should be used for larger imports (everything with 100 lines and more).
The import request is created and added to a queue; the job resource url for status polling is returned.
Asynchronous imports requested on the same table will be serialized.
The actual data must be provided in one of the following attributes:
- **dataFileId**,
- **snapshotId**,
- **dataWorkspaceId** and **dataTableName**
An imported file can be sliced into multiple chunks; the conditions described in [File Uploads](#reference/files) must be satisfied.
The CSV file can be raw or gzipped. Gzipped files must have the extension `.gz`. A typical asynchronous import negotiation consists of these steps:
1. Upload a CSV file using [File Upload](#reference/files).
2. Submit an [asynchronous import request](#reference/tables/load-data-asynchronously/imports-data) with `dataFileId` set to the File Id returned in the previous step.
3. Poll job resource returned in the previous step until the `status` is `success` or `error`.
Further information can be found in the [Developers Documentation](https://developers.keboola.com/integrate/storage/api/import-export/).
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ dataFileId (optional) - Id of the file stored in [File Uploads](#reference/files); all gzipped and sliced files are supported.
+ snapshotId (optional) - Id of a table snapshot - a table will be created from the snapshot.
+ dataWorkspaceId (optional) - Load from the table [workspace](#reference/workspaces). Use with the **dataTableName** attribute.
+ dataTableName (optional) - Load from the table in the [workspace](#reference/workspaces).
+ incremental (optional, number) - If incremental is set to 0, the target table is truncated before each import.
Default: 0
+ delimiter (optional) - Field delimiter used in the CSV file. The default value is ','. Use '\t' or type the tab char for tabulator.
+ enclosure (optional) - Field enclosure used in the CSV file. The default value is '"'. (Note: you can specify either `enclosure` or `escapedBy` parameter, not both.)
+ escapedBy (optional) - Escape character used in the CSV file. The default value is an empty value - no escape character is used. (Note: you can specify either `enclosure` or `escapedBy` parameter, not both.)
+ columns[] (optional) - List of columns present in the CSV file; the first line of the file will not be treated as a header!
+ withoutHeaders (optional) - The CSV file doesn't contain headers, columns are matched by their order. If this option is used, columns option is ignored.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
dataFileId=12345&incremental=0&delimiter=%2C&enclosure=%22
+ Response 202 (application/json)
{
"id": 22070846,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/22070846",
"tableId": "in.c-main.test",
"operationName": "tableImport",
"operationParams": {
"params": {
"incremental": false,
"partial": false,
"transaction": null,
"source": [],
"withoutHeaders": false,
"columns": [],
"delimiter": ",",
"enclosure": "\""
},
"source": {
"fileId": "12345",
"type": "file"
},
"queue": "main_fast"
},
"createdTime": "2017-02-13T12:51:48+0100",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
## Data Preview [/v2/storage/tables/{table_id}/data-preview]
### Data preview [GET]
Returns up to 1000 rows from table in CSV (RFC) format. Rows can be filtered by various filters described below.
For full table download please read more in [Asynchronous Unload documentation](#reference/tables/unload-data-asynchronously)
#### HTTP Compression
To enable compression of API response traffic, please include the following HTTP header with the API request: `Accept-encoding: gzip`.
#### Error Handling
The moment the streaming connection is established, the HTTP 200 response code is returned and data are streamed row by row.
When an error occurs and data cannot be streamed any more, a special row is returned to stream `KBC_INTERNAL_ERROR`.
When you receive this message, please run the export again; these errors are very rare.
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ limit (optional, number) - Limit the number of returned rows. Maximum allowed value is `1000`
+ Default: 100
+ changedSince (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
+ changedUntil (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
Examples: `-2 days`, `1360138863`, `2013-02-12T15:19:21+00:00`
+ columns (optional) - Comma separated list of columns to export; by default all columns are exported.
+ whereColumn (optional) - Column for exporting only matching rows; see more in the rows filtering section.
+ whereValues[] (optional) - Values for exporting only matching rows; see more in the rows filtering section.
+ whereOperator (optional, enum[string]) - Operator for value comparison.
+ Members
+ eq - Equals - can be used with multiple values.
+ ne - Not equals - can be used with multiple values.
+ Default: eq
+ Request
+ Headers
X-StorageApi-Token: your_token
Accept-encoding: gzip
+ Response 200 (text/csv)
+ Headers
Content-Encoding: gzip
+ Body
"id","name"
"26","czech"
"1","english"
"11","finnish"
"24","french"
## Unload Data [/v2/storage/tables/{table_id}/export]
# Export table into CSV [GET]
This method is **Deprecated**.
For table data unload please use [Asynchronous version](#reference/tables/unload-data-asynchronously) of data unload.
If you want to just fetch sample of table data please use [Table Data Preview](#reference/tables/data-preview).
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ limit (optional, number) - Limit the number of returned rows. Maximum allowed value is `1000`
+ Default: 100
+ changedSince (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
+ changedUntil (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
Examples: `-2 days`, `1360138863`, `2013-02-12T15:19:21+00:00`
+ columns (optional) - Comma separated list of columns to export; by default all columns are exported.
+ whereColumn (optional) - Column for exporting only matching rows; see more in the rows filtering section.
+ whereValues[] (optional) - Values for exporting only matching rows; see more in the rows filtering section.
+ whereOperator (optional, enum[string]) - Operator for value comparison.
+ Members
+ eq - Equals - can be used with multiple values.
+ ne - Not equals - can be used with multiple values.
+ Default: eq
+ Request
+ Headers
X-StorageApi-Token: your_token
Accept-encoding: gzip
+ Response 200 (text/csv)
+ Headers
Content-Encoding: gzip
+ Body
"id","name"
"26","czech"
"1","english"
"11","finnish"
"24","french"
## Unload Data Asynchronously [/v2/storage/tables/{table_id}/export-async]
### Asynchronous Export [POST]
Exports data into File Storage. The Id of the created file is returned.
This task is asynchronous; Id of the created file is stored in the `results` of the created job when the job is finished.
#### Rows filtering
Exported rows can be filtered by a single filter. Examples of filtering by `userId` column:
- `whereColumn=UserId&whereValues[]=23423` - Filters by single values of one column.
- `whereColumn=UserId&whereValues[]=23423&whereValues[]=123` - Filters by multiple values of one column.
- `whereColumn=UserId&whereValues[]=23423&whereOperator=ne` - Uses a different comparison operator.
The comparison operator essentially behaves like the [SQL IN](http://www.w3schools.com/sql/sql_in.asp) (and NOT IN) operator. That is, the
`eq` value matches if any `whereValues` match, and the `new` value matches if none `whereValues` match.
#### Caching
Asynchronous exports support internal caching. When the export file for a certain table
export parameters was created and there were no changes in the table data since, the
previous cached file will be returned.
#### Job results
The response contains a link to job results. Once the job is finished, the job result
will show information about the created file and caching:
```
{
"file": {
"id": 234
},
"cacheHit": false
}
```
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ limit (optional, number) - Limit the number of returned rows.
+ days (optional, number) - **DEPRECATED will be removed in the future.** It returns rows created or updated in the last X days.
+ format (optional, enum[string]) - File format
+ Members
+ rfc - According to [RFC](http://www.ietf.org/rfc/rfc4180.txt).
+ escaped - Delimited by a comma, enclosed by a double quote, special characters (e.g. new line, tab) escaped by a backslash. Each row in the CSV file is equal to one row in Storage.
+ raw - Delimited by tab, without an enclosure, special characters (e.g. new line, tab) escaped by a backslash. Each row in the CSV file is equal to one row in Storage.
+ Default: rfc
+ changedSince (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
+ changedUntil (optional) - Filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php).
Examples: `-2 days`, `1360138863`, `2013-02-12T15:19:21+00:00`
+ columns (optional) - Comma separated list of columns to export; by default all columns are exported.
+ whereColumn (optional) - Column for exporting only matching rows; see more in the rows filtering section.
+ whereValues[] (optional) - Values for exporting only matching rows; see more in the rows filtering section.
+ whereOperator (optional, enum[string]) - Operator for value comparison.
+ Members
+ eq - Equals - can be used with multiple values.
+ ne - Not equals - can be used with multiple values.
+ Default: eq
+ gzip (optional, boolean) - The response will be gzipped if set to true.
+ Request
+ Headers
X-StorageApi-Token: your_token
Accept-encoding: gzip
+ Response 202 (application/json)
+ Body
{
"id": 22070936,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/22070936",
"tableId": "in.c-main.test",
"operationName": "tableExport",
"operationParams": {
"export": {
"tableName": null,
"columns": [],
"limit": null,
"changedSince": null,
"changedUntil": null,
"gzipOutput": false,
"format": "rfc",
"whereFilters": []
},
"queue": "main_fast"
},
"createdTime": "2017-02-13T12:56:57+0100",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
## Manage Tables [/v2/storage/tables/{table_id}]
### Table detail [GET]
Obtains information about a table.
+ Parameters
+ table_id (required) - Table Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-application-testing.cashier-data",
"id": "in.c-application-testing.cashier-data",
"name": "cashier-data",
"transactional": false,
"primaryKey": [],
"indexedColumns": [],
"created": "2015-11-03T10:58:28+0100",
"lastImportDate": "2015-11-03T10:58:31+0100",
"lastChangeDate": "2015-11-03T10:58:32+0100",
"rowsCount": 199,
"dataSizeBytes": 25165824,
"isAlias": false,
"columns": [
"time_spent_in_shop",
"number_of_items",
"customer_shoe_size",
"customer_age",
"age_segment",
"no_segment",
"customer_segment",
"visit_id"
],
"columnMetadata": [],
"attributes": [],
"metadata": [],
"bucket": {
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-application-testing",
"id": "in.c-application-testing",
"name": "c-application-testing",
"stage": "in",
"description": "",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-application-testing/tables",
"created": "2015-11-03T10:54:55+0100",
"lastChangeDate": "2016-11-16T17:48:36+0100",
"isReadOnly": false,
"dataSizeBytes": 130023424,
"rowsCount": 10628,
"isMaintenance": false,
"backend": "redshift",
"sharing": null,
"metadata": [],
}
}
### Drop table [DELETE /v2/storage/tables/{table_id}/?force={force}]
Deletes a table from Storage. In default mode, only table having any aliases can be deleted.
Use the optional `force` parameter to delete its aliases too.
+ Parameters
+ table_id (required) - Table Id
+ force (optional, boolean) - Drops the table and its aliases.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Table Optimize [/v2/storage/tables/{table_id}/optimize]
### Optimize table [POST]
This is a utility command implemented only for projects with the Redshift backend.
Redshift tables with a lot of small-increment loads bloat in size. An optimize command is automatically scheduled to fix
this issue. This API call can be used to trigger immediate optimization of a table.
+ Parameters
+ table_id (required) - Table Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
{
"id": 245,
"status": "waiting",
"url": "/v2/storage/jobs/245",
"tableId": "in.c-API-tests.MyLanguages_test",
"operationName": "tableOptimize",
"operationParams": {
"queue": "main_fast"
},
"createdTime": "2016-10-17T10:31:52+0200",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "31",
"description": "dev@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
## List Tables [/v2/storage/tables?include={include}]
### List All Tables [GET]
Lists all tables accessible by the token. By default, all tables are returned with their
attributes and information about the containing bucket.
+ Parameters
+ include (optional) - Comma separated list of resources to include for each table.
Possible values: `attributes`, `buckets` and `columns`.
Default: `attributes,buckets`
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.data",
"id": "in.c-main.data",
"name": "data",
"transactional": false,
"primaryKey": [],
"indexedColumns": [],
"created": "2016-06-23T20:41:07+0200",
"lastImportDate": "2016-07-07T11:25:32+0200",
"lastChangeDate": "2016-07-07T11:25:34+0200",
"rowsCount": 18,
"dataSizeBytes": 12288,
"isAlias": false,
"attributes": [],
"bucket": {
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-main",
"id": "in.c-main",
"name": "c-main",
"stage": "in",
"description": "Main project storage",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-main/tables",
"created": "2015-02-05T10:57:08+0100",
"lastChangeDate": "2017-02-12T10:36:15+0100",
"isReadOnly": false,
"dataSizeBytes": 685750272,
"rowsCount": 2438881,
"isMaintenance": false,
"backend": "snowflake",
"sharing": null
}
},
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.data_items",
"id": "in.c-main.data_items",
"name": "data_items",
"transactional": false,
"primaryKey": [],
"indexedColumns": [],
"created": "2016-06-23T20:41:38+0200",
"lastImportDate": "2016-07-07T11:26:03+0200",
"lastChangeDate": "2016-07-07T11:26:05+0200",
"rowsCount": 48,
"dataSizeBytes": 12288,
"isAlias": false,
"attributes": [],
"bucket": {
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-main",
"id": "in.c-main",
"name": "c-main",
"stage": "in",
"description": "Main project storage",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-main/tables",
"created": "2015-02-05T10:57:08+0100",
"lastChangeDate": "2017-02-12T10:36:15+0100",
"isReadOnly": false,
"dataSizeBytes": 685750272,
"rowsCount": 2438881,
"isMaintenance": false,
"backend": "snowflake",
"sharing": null
}
}
]
## Create Table Column [/v2/storage/buckets/{table_id}/columns/]
### Add Column to Table [POST]
Adds a new column to an existing table. This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
+ Parameters
+ table_id (required) - Table Id
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
name=new-column
+ Response 202 (application/json)
+ Body
{
"id":1189,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1189",
"tableId":"in.c-main.another",
"operationName":"tableColumnAdd",
"operationParams":{
"name":"new-column"
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
## Manage Table Columns [/v2/storage/tables/{table_id}/columns/{column_name}]
### Delete Column from Table [DELETE /v2/storage/tables/{table_id}/columns/{column_name}/?force={force}]
Deletes an existing column from an existing table. This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
Use the optional `force` parameter to delete the column from table aliases too.
+ Parameters
+ table_id (required) - Table Id
+ column_name (required) - Column Name
+ force (optional, boolean) - Delete the column from table and table aliases. All aliases must have `aliasColumnsAutosync` enabled.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"id":1189,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1189",
"tableId":"in.c-main.another",
"operationName":"tableColumnDelete",
"operationParams":{
"name":"new-column",
"force": false
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
## Primary Keys [/v2/storage/tables/{table_id}/primary-key/]
### Create Primary Key [POST]
If a primary key is set, updates can be done on the table. A primary key can be composed of multiple columns.
This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ columns[] (required) - Array of columns used as the primary key
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
columns[]=id
+ Response 202 (application/json)
+ Body
{
"id":1189,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1189",
"tableId":"in.c-main.another",
"operationName":"tablePrimaryKeyAdd",
"operationParams":{
"columns":["id"]
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
### Remove Primary Key [DELETE]
Removes the primary key from a table. This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
+ Parameters
+ table_id (required) - Table Id
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"id":1189,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1189",
"tableId":"in.c-main.another",
"operationName":"tablePrimaryKeyDelete",
"operationParams":{
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
## Manage Table Rows [/v2/storage/tables/{table_id}/rows]
### Delete Table Rows [DELETE]
Deletes all rows matching the specified filter. The number of deleted rows is returned.
This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
+ Parameters
+ table_id (required) - Table Id
+ Attributes
+ whereColumn (optional) - Column for exporting only matching rows, see more in the [rows filtering section](#reference/tables/unload-data-asynchronously/asynchronous-export).
+ whereValues[] (optional) - Values for exporting only matching rows, see more in the [rows filtering section](#reference/tables/unload-data-asynchronously/asynchronous-export).
+ whereOperator (optional, enum[string]) - Operator for value comparison.
+ Members
+ eq - Equals - can be used with multiple values.
+ ne - Not equals - can be used with multiple values.
+ Default: eq
+ changedSince - (optional) filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php). Examples: `-2 days`, `1360138863`, `2013-02-12T15:19:21+00:00`
+ changedUntil - (optional) filtering by import date - timestamp of import is stored within each row. Both until and since values can be a unix timestamp or any date accepted by [strtotime](http://www.php.net/manual/en/function.strtotime.php). Examples: `-2 days`, `1360138863`, `2013-02-12T15:19:21+00:00`
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"id":1190,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1190",
"tableId":"in.c-main.another",
"operationName":"tableRowsDelete",
"operationParams":{
"whereColumn":"id",
"whereValues": [12344],
"whereOperator": "eq"
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
# Group Table Aliases
[Aliases](https://help.keboola.com/storage/tables/) behave like database Views. Alias tables are read-only.
Aliases cannot be chained and can be applied only between buckets with the same backend.
Rows present in an alias table can be filtered by a simple condition similar to
[table export filtering](#reference/tables/unload-data-asynchronously/asynchronous-export).
The filter parameter `aliasFilter` must be set during an alias table creation or later to enable filtering:
* **column** - Filtered by column name
* **operator** - Filter operator: `eq`, `ne`
* **values[]** - One or more filter values
#### Alias table columns
By default, the alias columns are automatically synced with the source table. You can disable this behaviour
by setting up `aliasColumnsAutosync` to `false` and setting up list of columns by `aliasColumns` parameter.
Later you can modify the alias columns by the table columns methods.
When turning `aliasColumnsAutoSync` back on, the columns will be immediately synced with the source table.
## Create Table Alias [/v2/storage/buckets/{bucket_id}/table-aliases]
### Create new alias table [POST]
+ Parameters
+ bucket_id (required) - Id of a bucket in which the alias table will be created.
+ Attributes
+ name (required) - Name of the new table alias
+ sourceTable (required) - Id of the source table
+ aliasFilter[column] (optional) - Name of the column for filtering
+ aliasFilter[operator] (optional, enum[string]) - Comparison operator
+ Members
+ eq - Equal to
+ ne - Not equal to
+ Default: eq
+ aliasFilter[values][] (optional) - Values for the filter
+ aliasColumnsAutosync (optional, boolean) - Disables column synchronization with the source table.
Default: false
+ aliasColumns[] (optional) - List of alias columns; this disables column synchronization with the source table.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
sourceTable=in.c-main.data&name=test-alias2&aliasFilter%5Bcolumn%5D=campaign_id&aliasFilter%5Bvalues%5D%5B%5D=68abf5473f&aliasFilter%5Bvalues%5D%5B%5D=03e9b81530&aliasFilter%5Boperator%5D=eq&aliasColumns%5B%5D=campaign_id&aliasColumns%5B%5D=email
+ Response 201 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-Test2.test-alias2",
"id": "in.c-Test2.test-alias2",
"name": "test-alias2",
"transactional": false,
"primaryKey": [],
"indexedColumns": [],
"created": "2017-02-13T13:53:31+0100",
"lastImportDate": "2016-07-07T11:25:32+0200",
"lastChangeDate": "2016-07-07T11:25:34+0200",
"rowsCount": null,
"dataSizeBytes": null,
"isAlias": true,
"sourceTable": {
"id": "in.c-main.data",
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.data",
"project": {
"id": 578,
"name": "Odinuv Sandbox"
}
},
"aliasColumnsAutoSync": false,
"aliasFilter": {
"column": "campaign_id",
"operator": "eq",
"values": [
"68abf5473f",
"03e9b81530"
]
},
"columns": [
"campaign_id",
"email"
],
"bucket": {
"uri": "https://connection.keboola.com/v2/storage/buckets/in.c-Test2",
"id": "in.c-Test2",
"name": "c-Test2",
"stage": "in",
"description": "",
"tables": "https://connection.keboola.com/v2/storage/buckets/in.c-Test2/tables",
"created": "2017-02-13T11:41:55+0100",
"lastChangeDate": "2017-02-13T13:50:55+0100",
"isReadOnly": false,
"dataSizeBytes": 0,
"rowsCount": 0,
"isMaintenance": false,
"backend": "snowflake",
"sharing": null
}
}
## Manage Alias Filters [/v2/storage/tables/{table_id}/alias-filter]
### Update Alias Filter [POST]
Using this API call, you can update an existing filter. You can also set a new filter, but the
alias must be created with the `aliasFilter` parameter.
+ Parameters
+ table_id (required) - Id of the alias table
+ Attributes
+ column (optional) - Column name
+ values[] (optional) - Column filter values
+ operator (optional, enum[string]) - Comparison operator
+ Members
+ eq - Equal to
+ ne - Not equal to
+ Default: eq
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
column=campaign_id&values%5B%5D=68abf5473f
+ Response 200 (application/json)
+ Body
{
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-Test2.test-alias2",
"id": "in.c-Test2.test-alias2",
"name": "test-alias2",
"transactional": false,
"primaryKey": [],
"indexedColumns": [],
"created": "2017-02-13T13:53:31+0100",
"lastImportDate": "2016-07-07T11:25:32+0200",
"lastChangeDate": "2016-07-07T11:25:34+0200",
"rowsCount": null,
"dataSizeBytes": null,
"isAlias": true,
"sourceTable": {
"id": "in.c-main.data",
"uri": "https://connection.keboola.com/v2/storage/tables/in.c-main.data",
"project": {
"id": 578,
"name": "Odinuv Sandbox"
}
},
"aliasColumnsAutoSync": false,
"aliasFilter": {
"column": "campaign_id",
"operator": "eq",
"values": [
"68abf5473f"
]
}
}
### Remove Alias Filter [DELETE]
Deletes a filter from an alias. The alias table will contain all rows from the source table.
+ Parameters
+ table_id (required) - Id of the alias table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Alias Column Synchronization [/v2/storage/tables/{table_id}/alias-columns-auto-sync]
### Enable Column Synchronization [POST]
Enables automatic column synchronization between the alias and source table. All columns from the source table
will be immediately synchronized into the alias table.
+ Parameters
+ table_id (required) - Id of the alias table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
### Disable Column Synchronization [DELETE]
Disables automatic column synchronization between the alias and source table. Columns from the source table
will not be synchronized with the alias table any more. When you add new columns into the source table, they
will not propagate to the alias.
+ Parameters
+ table_id (required) - Id of the alias table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
# Group Table Snapshotting
Table data and metadata (attributes, column settings) can be persisted by creating a snapshot.
When a snapshot is created you can use it to restore the table back to the snapshotted state.
During the restoration process, the data can be applied to the original table or they can be restored
into a new table.
## Create or List Snapshots [/v2/storage/tables/{table_id}/snapshots]
### Create Table Snapshot [POST]
Creates a snapshot of the current state of the table. This request is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
A table snapshot will contain:
- Table data,
- Table properties (primary key, indexed columns, etc.),
- Table attributes.
Snapshots of an alias table contain only alias settings, actual data are not stored with the snapshot.
+ Parameters
+ table_id (required) - Id of a table
+ Attributes
+ description (optional) - Snapshot description
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
description=My first snapshot
+ Response 202 (application/json)
+ Body
{
"id": 22073730,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/22073730",
"tableId": "in.c-main.test",
"operationName": "tableSnapshotCreate",
"operationParams": {
"description": null,
"queue": "main_fast"
},
"createdTime": "2017-02-13T14:44:46+0100",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
### List Table Snapshots [GET]
Retrieves a list of all snapshots of a table.
+ Parameters
+ table_id (required) - Id of a table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200
+ Headers
Content-type: application/json
+ Body
[
{
"id": "82629",
"description": "Bucket backend migration (mysql -> snowflake)",
"createdTime": "2016-05-26T14:26:56+0200",
"type": "table",
"creatorToken": {
"id": 0,
"description": "Internal - Bucket backend migration"
},
"uri": "https://connection.keboola.com/v2/storage/snapshots/82629",
"dataFileId": "188803533"
}
]
## Manage Snapshots [/v2/storage/snapshots/{snapshot_id}]
### Snapshot Detail [GET]
Retrieves information about a single table snapshot.
+ Parameters
+ snapshot_id - Id of the snapshot; use the
[List Snapshots Call](#reference/table-snapshotting/create-or-list-snapshots/list-table-snapshots) to find snapshot Id.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
{
"id": "20",
"description": "",
"createdTime": "2013-08-12T12:33:04+0200",
"type": "table",
"creatorToken": {
"id": 221,
"description": "Master Token"
},
"dataFileId": "161757",
"table": {
"id": "in.c-dalsi.alias",
"primaryKey": [],
"indexedColumns": [],
"attributes": [
{
"name": "afd",
"value": "sdfa",
"protected": false
},
{
"name": "pokus",
"value": "neco",
"protected": false
},
{
"name": "test",
"value": "adsf",
"protected": false
},
{
"name": "sadf",
"value": "",
"protected": false
},
{
"name": "asd",
"value": "",
"protected": false
}
],
"columns": [
"id",
"name"
]
}
}
### Delete Table Snapshot [DELETE]
Removes a table snapshot. The current table data is not modified. This request
is [asynchronous](#introduction/synchronous-and-asynchronous-calls).
+ Parameters
+ snapshot_id - Id of the snapshot. Use the
[List Snapshots Call](#reference/table-snapshotting/create-or-list-snapshots/list-table-snapshots) to find snapshot Id.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 202 (application/json)
+ Body
{
"id":1189,
"status":"waiting",
"url":"https:\/\/connection.keboola.com\/v2\/storage\/jobs\/1189",
"tableId":"in.c-main.another",
"operationName":"tableSnapshotDelete",
"operationParams":{
"snapshotId":"80954"
},
"createdTime":"2013-07-08T10:01:03+0200",
"startTime":null,
"endTime":null,
"runId":null,
"results":null,
"creatorToken":{
"id":"221",
"name":"Master Token"
}
}
# Group Workspaces
Workspaces provide a playground for your data. You can load data from your storage, explore or modify it
and then, optionally, load it back to storage tables. A workspace is represented by access credentials to a storage
backend with permissions to write to the workspace specific database schema.
## Workspaces Collection [/v2/storage/workspaces]
### Create new workspace [POST]
Creates a new workspace and return its credentials.
+ Attributes
+ backend: snowflake (optional, enum[string]) - Workspace backend. When omitted, the default backend is used.
+ Members
+ redshift
+ snowflake
the system. Supported only for Snowflake workspaces.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
backend=snowflake
+ Response 201 (application/json)
+ Attributes (WorkspaceCreated)
### List workspaces [GET]
Returns all workspaces for the project.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Attributes (array[Workspace])
## Manage Workspace [/v2/storage/workspaces/{workspace_id}]
### Workspace detail [GET]
Retrieves information about a given workspace. Note that the password to the workspace can be retrieved only when the workspace is created.
+ Parameters
+ workspace_id (number)
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Attributes (Workspace)
### Delete workspace [DELETE /v2/storage/workspaces/{workspace_id}/?async={async}]
Deletes a workspace. This also irreversibly removes the workspace content.
+ Parameters
+ workspace_id (required, number)
+ async (optional, boolean) - Workspace will be deleted [asynchronously](#introduction/synchronous-and-asynchronous-calls). A job will be created and enqueued.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Load Data [/v2/storage/workspaces/{workspace_id}/load]
+ Parameters
+ workspace_id (required, number)
### Load Data [POST]
Loads tables from Storage into a Workspace.
+ Attributes
+ input[] (required, array) - Mappings of source tables with destinations to be loaded into the workspace
+ Items
+ (object)
+ Properties
+ source (required) - Full table identifier of the source table (e.g.: in.c-bucket.mytable)
+ destination (required) - Destination table name
+ rows (optional, number) - Limits the number of returned rows
+ days (optional, number) - `DEPRECATED will be removed in the future.` Return rows created or updated in the last X days.
+ seconds (optional, number) - Returns rows created or updated in the last X seconds.
+ columns[] (optional, array) - Array of column definition; by default all columns are exported.
+ Items
+ (object)
+ Properties
+ source (required, string) - Column name
+ destination (optional, string) - Destination column name. If not set, source column name will be used.
+ type (required, string) - Data type, eg. VARCHAR
+ length (optional, string) - Data type size, where applicable, eg. 255 for VARCHAR or 10,2 for NUMBER
+ nullable (optional, boolean) - Allow NULL values in destination table
Default: true
+ convertEmptyValuesToNull (optional, boolean) - Empty values replaced by NULL
Default: false
+ compression (optional, enum[string]) - For Redshift only
+ Members
+ RAW
+ BYTEDICT
+ DELTA
+ DELTA32K
+ LZO
+ MOSTLY8
+ MOSTLY16
+ MOSTLY32
+ RUNLENGTH
+ TEXT255
+ TEXT32K
+ ZSTD
+ whereColumn (optional) - Column for [filtering](#reference/tables/unload-data-asynchronously/asynchronous-export)
+ whereValues[] (optional) - Values for filtering
+ whereOperator (optional, enum[string]) - Comparison operator
+ Members
+ eq - Equal to
+ ne - Not equal to
+ Default: eq
+ sortKey[] (optional, array) - Redshift only - Column(s) to be used as a sort key
+ distStyle (optional, enum[string]) - Redshift only - Distribution style (even, all, or key)
+ Members
+ even
+ all
+ key
+ distKey (optional) - Redshift only - Column to use for the key distribution style
+ incremental (optional, boolean) - Rows will be appended to an existing table
Default: false
+ preserve (optional, boolean) - Keep existing tables in the workspace, otherwise the workspace will be purged before loading
Default: false
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
input%5B0%5D%5Bsource%5D=in.c-application-testing.cashier-data&input%5B0%5D%5Bdestination%5D=my-table
+ Response 201 (application/json)
+ Body
{
"id": 22077337,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/22077337",
"tableId": null,
"operationName": "workspaceLoad",
"operationParams": {
"workspaceId": "78423",
"preserve": false,
"input": [
{
"source": "in.c-application-testing.cashier-data",
"destination": "my-table"
}
],
"queue": "main_fast"
},
"createdTime": "2017-02-13T16:41:18+0100",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
## Load Data using CLONE [/v2/storage/workspaces/{workspace_id}/load-clone]
+ Parameters
+ workspace_id (required, number)
### Load Data using CLONE [POST]
Load tables from Storage into a Workspace using zero-copy cloning functionality in Snowflake. This load type does not
support any filtering or processing parameters. Only tables stored in Snowflake are supported.
The cloned table will contain all columns of the source table (all of them as `VARCHAR(1048576)`)
and a `_timestamp` system column.
This column contains a unix timestamp value (`TIMESTAMP_NTZ(9)`) when the row was added to the table or
when any of the row values were changed during an incremental import.
+ Attributes
+ input[] (required, array) - Mappings of source tables with destinations to be loaded into the workspace
+ Items
+ (object)
+ Properties
+ source (required) - Full table identifier of the source table (e.g.: in.c-bucket.mytable)
+ destination (required) - Destination table name
+ preserve (optional, boolean) - Keep existing tables in the workspace, otherwise the workspace will be purged before loading
Default: false
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
input%5B0%5D%5Bsource%5D=in.c-application-testing.cashier-data&input%5B0%5D%5Bdestination%5D=my-table
+ Response 201 (application/json)
+ Body
{
"id": 22077337,
"status": "waiting",
"url": "https://connection.keboola.com/v2/storage/jobs/22077337",
"tableId": null,
"operationName": "workspaceLoad",
"operationParams": {
"workspaceId": "78423",
"preserve": false,
"input": [
{
"source": "in.c-application-testing.cashier-data",
"destination": "my-table"
}
],
"queue": "main_fast"
},
"createdTime": "2017-02-13T16:41:18+0100",
"startTime": null,
"endTime": null,
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,
"inBytes": 0,
"inBytesUncompressed": 0,
"outCompressed": false,
"outBytes": 0,
"outBytesUncompressed": 0
}
}
## Password reset [/v2/storage/workspaces/{workspace_id}/password]
### Password reset [POST]
Generates new password for the given workspace
+ Parameters
+ workspace_id (number)
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 201 (application/json)
+ Body
{
"password": "2Fd4e1c67a2d28fced849ee1bb76e7391b93eb12",
}
## Data Structures
### Workspace
+ id: 234 (number)
+ name: `boring_wozniak` (string) - working schema name
+ component: `wr-db` (string) - name of the [registered component](#reference/miscellaneous/api-index/component-list) that created the workspace
+ configurationId: `aws-1` (string) - id of a [configuration](#reference/component-configurations/component-list) used to create workspace
+ created: `2016-05-17T11:11:20+0200` (string)
+ connection
+ backend: `snowflake` (string)
+ host: `keboola.snowflakecomputing.com` (string)
+ database: `keboola_123` (string)
+ schema: `boring_wozniak` (string)
+ warehouse: `SAPI_PROD` (string)
+ user: `xzy` (string)
+ creatorToken
+ id: 234 (number)
+ description: `martin@keboola.com` (string)
+ creatorUser
+ id: 234 (number)
+ name: `Martin` (string)
### WorkspaceCreated (Workspace)
+ connection
+ backend: snowflake (string)
+ host: keboola.snowflakecomputing.com (string)
+ database: keboola_123 (string)
+ schema: boring_wozniak (string)
+ warehouse: SAPI_PROD (string)
+ user: xzy (string)
+ password: abc (string) - Password is not stored and is returned only after workspace creation
# Group Events
Virtually every interaction of the client (an API call) creates an Event. Apart from that, external events may
also be added via an API. Events are available for 6 months from their creation.
You can access all events created on an account.
Events can be filtered by various filters:
- `sinceId`, `maxId` - Events newer or older than the passed id
- `component` - Component name
- `configurationId` - Configuration id
- `runId` - Run id
- `q` - Advanced search using a query string. [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax)
The events pagination uses two parameters:
- `limit` - Number of returned events. The default value: 100
- `offset` - Pagination offset
## Events [/v2/storage/events]
### Create Event [POST]
Creates a new external event. The maximum allowed size of an event is 200KB. If this is exceeded,
the HTTP 400 response code is returned.
+ Attributes
+ message (required) - Event message
+ component (required) - Name of a [registered component](#reference/miscellaneous/api-index/component-list)
+ description (optional) - Event description
+ type (optional, enum[string]) - Type of the event
+ Members
+ info
+ success
+ warn
+ error
+ Default: info
+ configurationId (optional) - Id of a configuration executed by the component
+ params (optional, string) - Serialized JSON object with custom data
+ results (optional, string) - Serialized JSON object with custom data
+ duration (optional, number) - Duration of the associated task in seconds
+ runId (optional) - Id of the associated component job
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
component=keboola.test-component&message=This+is+a+word+of+warning&type=warn
+ Response 201 (application/json)
+ Body
{
"id": "13008826"
}
### Events list [GET /v2/storage/events?sinceId={sinceId}&maxId={maxId}&component={component}&configurationId={configurationId}&runId={runId}&q={q}&limit={limit}&offset={offset}]
Lists all events in the project.
+ Parameters
+ sinceId (optional) - Shows only events after the specified Id.
+ maxId (optional) - Shows only events before the passed Id.
+ component (optional) - Shows only events emitted by the specified component.
+ configurationId (optional) - Show only events related to given configuration Id.
+ runId (optional) - Shows only events with the specified run Id.
+ q (optional) - Searches events using advanced [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax).
+ limit (optional, number) - Pagination limit. Maximum allowed value is `10000`
+ Default: 100
+ offset (optional, number) - Pagination offset
+ Default: 0
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Attributes (array[Event])
## List Bucket Events [/v2/storage/buckets/{bucket_id}/events?sinceId={sinceId}&maxId={maxId}&component={component}&configurationId={configurationId}&runId={runId}&q={q}&limit={limit}&offset={offset}]
### Bucket events list [GET]
Lists all events associated with a particular bucket.
+ Parameters
+ sinceId (optional) - Shows only events after the specified Id.
+ maxId (optional) - Shows only events before the passed Id.
+ component (optional) - Shows only events emitted by the specified component.
+ configurationId (optional) - Shows only events related to given configuration Id.
+ runId (optional) - Shows only events with the specified run Id.
+ q (optional) - Searches events using advanced [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax).
+ limit (optional, number) - Pagination limit
+ Default: 100
+ offset (optional, number) - Pagination offset
+ Default: 0
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Attributes (array[Event])
## List Table Events [/v2/storage/tables/{table_id}/events?sinceId={sinceId}&maxId={maxId}&component={component}&configurationId={configurationId}&runId={runId}&q={q}&limit={limit}&offset={offset}]
### Table events list [GET]
Lists all events associated with a particular table.
+ Parameters
+ sinceId (optional) - Shows only events after the specified Id.
+ maxId (optional) - Shows only events before the passed Id.
+ component (optional) - Shows only events emitted by the specified component.
+ configurationId (optional) - Shows only events related to the given configuration Id.
+ runId (optional) - Shows only events with the specified run Id.
+ q (optional) - Searches events using advanced [Query string syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax).
+ limit (optional, number) - Pagination limit
+ Default: 100
+ offset (optional, number) - Pagination offset
+ Default: 0
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Attributes (array[Event])
## Data Structures
### Event
+ id: 498839405 - Event Id
+ event: storage.workspaceLoaded - Event identifier such as `storage.tableImportDone` or `storage.tableDetail`
+ component: storage - Name of the component that logged the event. Events can be logged by the `storage` component or any other
[registered component](#reference/miscellaneous/api-index/component-list).
+ message: `Table in.c-application-testing.cashier-data was loaded to the workspace workspace_78423 as my-table.` - Human readable description of the event
+ description: `` - Additional description of the event, such as a detailed error description
+ type: `info` (enum[string]) - Type of an event
+ Members
+ info
+ success
+ warn
+ error
+ runId: null - Id of an associated job.
+ created: `2017-02-13T16:42:00+0100` - Datetime when the event was created.
+ configurationId: null - Id of a [configuration](#reference/component-configurations/component-list) used by an external component.
+ objectId: `in.c-application-testing.cashier-data` - Id of the target object of the event - e.g. *id* of table for `storage.tableImportDone event`.
+ objectName: `cashier-data` - Name of the target object - e.g. a *table name* for a `storage.tableImportDone` event.
+ objectType: table - Type of the target object - e.g. *table* for the `storage.tableImportDone` event.
+ context (object) - Details about the origin of the API call that triggered the event
+ remoteAddr
+ httpReferer
+ httpUserAgent
+ apiVersion
+ userAgent
+ async (boolean)
+ params (object) - Arbitrary parameters associated with the event - e.g. CSV file settings for table import
+ results (object) - Arbitrary results of the event
+ performance (object) - Arbitrary performance metrics associated with the event
+ token (object) - Creator of the event
+ id: 1234 - Id of the token
+ name: support@keboola.com - Token owner
+ uri: 498839405 - Event URI.
+ attachments (object) - Arbitrary list of attached files - e.g. backups for table import events
# Group Attributes
Attributes are key-value pairs and let you assign custom metadata to resources.
You can associate custom attributes with buckets or tables. Bucket attributes methods are described
below, you can work with table attributes in the same manner.
## Create or Manage Bucket Attributes [/v2/storage/buckets/{bucket_id}/attributes/{attribute_key}]
### Set Bucket Attribute [POST]
Updates or creates an attribute. If the attribute exists, it will be updated. If it does not
exist, a new one will be created. All attributes associated with a bucket are returned in the response.
+ Parameters
+ bucket_id (required) - Id of the bucket
+ attribute_key (required) - Name of the attribute
+ Attributes
+ value (required) - Attribute value.
+ protected (optional, boolean) - Attribute will be stored encrypted.
Default: false
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
value=someValue&protected=false
+ Response 201 (application/json)
+ Body
[
{
"name": "other-attribute",
"value": "something",
"protected": false
},
{
"name": "testAttribute",
"value": "someValue",
"protected": false
}
]
### Delete Bucket Attribute [DELETE]
Removes an attribute from a bucket.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Replace or List Bucket Attributes [/v2/storage/buckets/{bucket_id}/attributes]
### Replace Attributes [POST]
Sets all attributes by one call.
+ Parameters
+ bucket_id (required) - Id of the bucket
+ Attributes
+ `attributes` (array)
+ (object)
+ name - attribute name
+ value - attribute value
+ protected (boolean) - True if the attribute will be stored encrypted.
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
attributes[][value]=something&attributes[][name]=first
+ Response 201 (application/json)
+ Body
[
{
"name": "first",
"value": "something",
"protected": false
}
]
### Bucket Attributes list [GET]
Shows all attributes assigned to a bucket.
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"name": "other-attribute",
"value": "something",
"protected": false
},
{
"name": "test",
"value": "2012-06-05",
"protected": false
}
]
# Group Metadata
You can associate custom metadata with buckets, tables or columns.
Metadata are timestamped key-value pairs separated by the provider, where the provider is the originating component.
Metadata allows you to assign custom information to your resources. Bucket attributes methods are described
below. Work with table and column metadata in the same manner; simply use a table or column instead of a bucket in the url.
## Bucket Metadata [/v2/storage/buckets/{bucket_id}/metadata]
### Create or Update [POST]
Sets multiple metadata with one call. If the given key and provider combination already exist
for the bucket, the data will be updated with the new value and timestamp. All metadata associated
with the bucket are returned in the response.
+ Parameters
+ bucket_id (required) - Id of the bucket
+ Attributes
+ provider (required) - Id of the creating [component](#reference/miscellaneous/api-index/component-list)
+ metadata (required) - Array of metadata objects
+ Metadata (object)
+ key (required)
+ value (required)
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
provider=test-component&metadata%5B0%5D%5Bkey%5D=some-key&metadata%5B0%5D%5Bvalue%5D=Some+value
+ Response 201 (application/json)
+ Body
[
{
"id": "19509",
"key": "some-key",
"value": "Some value",
"provider": "test-component",
"timestamp": "2017-02-13 23:36:10"
}
]
### List [GET]
Metadata can be listed for your bucket column.
+ Parameters
+ bucket_id (required) - Id of the bucket
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": "123"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "attribute_key",
"value": "attribute_value"
},
{
"id": "124"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "another_attribute_key",
"value": "attribute_value"
}
]
### Delete [DELETE]
Deletes all Metadata for the specified bucket.
+ Parameters
+ bucket_id (required) - Id of the bucket
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Table Metadata [/v2/storage/tables/{table_id}/metadata]
### Create or Update [POST]
Sets multiple metadata with one call. If the given key and provider combination already exist
for the table, the data will be updated with the new value and timestamp. All metadata associated
with the table are returned in the response.
+ Parameters
+ table_id (required) - Id of the table
+ Attributes
+ provider (required) - Id of the creating [component](#reference/miscellaneous/api-index/component-list).
+ metadata (required) - Array of metadata objects
+ Metadata (object)
+ key (required)
+ value (required)
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
provider=test-component&metadata%5B0%5D%5Bkey%5D=some-key&metadata%5B0%5D%5Bvalue%5D=Some+value
+ Response 201 (application/json)
+ Body
[
{
"id": "19509",
"key": "some-key",
"value": "Some value",
"provider": "test-component",
"timestamp": "2017-02-13 23:36:10"
}
]
### List [GET]
Shows all metadata associated with the specified table.
+ Parameters
+ table_id (required) - Id of the table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": "123"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "attribute_key",
"value": "attribute_value"
},
{
"id": "124"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "another_attribute_key",
"value": "attribute_value"
}
]
### Delete [DELETE]
Deletes all Metadata for the specified table.
+ Parameters
+ table_id (required) - Id of the table
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
## Column Metadata [/v2/storage/columns/{column_id}/metadata]
### Create or Update [POST]
Sets multiple metadata with one call. If the given key and provider combination already exist
for the column, the data will be updated with the new value and timestamp. All metadata associated
with the column are returned in the response.
+ Parameters
+ column_id (required) - Id of the table
+ Attributes
+ provider (required) - Id of the creating [component](#reference/miscellaneous/api-index/component-list)
+ metadata (required) - Array of metadata objects
+ Metadata (object)
+ key (required)
+ value (required)
+ Request (application/x-www-form-urlencoded)
+ Headers
X-StorageApi-Token: your_token
+ Body
provider=test-component&metadata%5B0%5D%5Bkey%5D=some-key&metadata%5B0%5D%5Bvalue%5D=Some+value
+ Response 201 (application/json)
+ Body
[
{
"id": "19509",
"key": "some-key",
"value": "Some value",
"provider": "test-component",
"timestamp": "2017-02-13 23:36:10"
}
]
### List [GET]
Shows all metadata associated with the specified column.
+ Parameters
+ column_id (required) - Id of the column
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": "123"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "attribute_key",
"value": "attribute_value"
},
{
"id": "124"
"provider": "keboola.some-komponent",
"timestamp": "2016-09-19 13:02:11",
"key": "another_attribute_key",
"value": "attribute_value"
}
]
### Delete [DELETE]
Deletes all Metadata for the specified column.
+ Parameters
+ column_id (required) - Id of the column
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 204
# Group Jobs
Jobs are objects that manage asynchronous tasks, these are all potentially long-running actions such as
loading table data, snapshotting, table structure modifications. Jobs are created by actions on target
resources, e.g., an [Asynchronous import request](#reference/tables/load-data-asynchronously/import-data) creates
a table import job. For each new job, the Id and URL of the job is returned.
When working with jobs, follow these steps:
1. **Start the job** - Job resource is created and a URL for polling is returned.
2. **Poll job status** - Poll the job URL in a loop until the job `status` is set to `success` or `error`.
3. **Check the results** - When the job is finished, the `results` field will contain the actual result of the original request.
Available Statuses:
- `waiting` - The job is in the queue and is waiting for execution.
- `processing` - The job is being processed by a worker.
- `success` - The job is done with a success.
- `error` - The job is done with an error.
## List Jobs [/v2/storage/jobs]
### Jobs list [GET]
+ Request
+ Headers
X-StorageApi-Token: your_token
+ Response 200 (application/json)
+ Body
[
{
"id": 22077337,
"status": "success",
"url": "https://connection.keboola.com/v2/storage/jobs/22077337",
"tableId": null,
"operationName": "workspaceLoad",
"operationParams": {
"workspaceId": "78423",
"preserve": false,
"input": [
{
"source": "in.c-application-testing.cashier-data",
"destination": "my-table"
}
],
"queue": "main_fast"
},
"createdTime": "2017-02-13T16:41:18+0100",
"startTime": "2017-02-13T16:41:18+0100",
"endTime": "2017-02-13T16:42:00+0100",
"runId": null,
"results": null,
"creatorToken": {
"id": "27978",
"description": "ondrej.popelka@keboola.com"
},
"metrics": {
"inCompressed": false,