Permalink
Switch branches/tags
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
359 lines (277 sloc) 12.5 KB
FORMAT: 1A
HOST: https://syrup.keboola.com/docker/
# Docker Runner
Docker Runner is a Keboola service that runs Docker containers as workers in Keboola Connection jobs. See the [Developer documentation](https://developers.keboola.com/extend/docker-runner/) for overview of the service and integration options. This API description assumes using the US region KBC stack, if you are using other stacks, you need to use the [correct URL](https://developers.keboola.com/overview/api/#regions-and-endpoints).
# Group Encrypt
Encryption docs moved to [new location](https://keboolaencryption.docs.apiary.io/#).
## Encryption Migration [/migrate?componentId={componentId}&projectId={projectId}&configId={configId}]
Used to migrate legacy ciphers (KBC::XXXEncrypted==) to new format (KBC::XXXSecure::).
If any of the `projectId` or `configId` is specified, a suitable encryptor is selected (`KBC::ComponentSecure::`,
`KBC::ProjectSecure::` or `KBC::ConfigSecure::`), so that the encrypted
value can be decrypted only in an environment matching the parameters.
+ Parameters
+ componentId(required, string) ... Keboola Connection Component Id
+ projectId(optional, string) ... Keboola Connection Project Id
+ configId(optional, string) ... Keboola Connection Configuration Id (when used, `projectId` must also be specified)
### Convert Ciphers [POST]
+ Request Encrypt plain text (text/plain)
+ Body
KBC::Encrypted==ENCODEDSTRING==
+ Response 200 (text/plain)
KBC::ProjectSecure::ENCODEDSTRING==
+ Request Encrypt JSON structure (application/json)
+ Body
{
"config": 1,
"#test": "KBC::Encrypted==ENCODEDSTRING=="
}
+ Response 202 (application/json)
{
"config": 1,
"#test": "KBC::ProjectSecure::ENCODEDSTRING=="
}
# Group Run
Run a configuration of a component. The configuration is specified either in `config` or `configData` property of the request body.
The `config` property contains the Id of an existing configuration. When you want to run a inline defined configuration, use the
`configData` property to specify the entire configuration.
One of `config` or `configData` must always be specified.
If both `config` and `configData` are specified, then `config` is only used in conjunction with `default_bucket`
option to resolve the bucket name for output mapping.
When the `config` property is specified, a configuration with multiple rows can also use the `row` property to specify a single row to be executed.
This setting will also override the `isDisabled` property of the configuration row.
That means that it will execute a disabled configuration row.
## Create a job [/{component}/run]
Create an execution job on a specified component.
+ Parameters
+ component (string) ... KBC component Id
### Run job [POST]
+ Request Stored configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"config": "next-order-1",
"row": "my-row"
}
+ Request Custom configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"configData": {
"storage": {
"input": {
"files": [
{
"query": "name: testfile.png"
}
],
"tables": [
{
"source": "in.c-redshift.table1",
"destination": "table1.csv"
},
{
"source": "in.c-redshift.some-table",
"destination": "table2.csv"
}
]
},
"output": {
"files": [
{
"source": "file.csv",
"tags": [
"processed-file",
"csv"
]
}
],
"tables": [
{
"source": "sliced.csv",
"destination": "out.c-main.data"
}
]
}
},
"parameters": {
"foo": "bar",
"script": null
}
"runtime": {
"baz": "bar"
}
}
}
+ Response 202 (application/json)
{
"id": "67121676",
"url": "https://syrup.keboola.com/queue/job/67121676",
"status": "waiting"
}
## Create a job with image [/{component}/run/tag/{tag}]
Create an execution job on a specified component while using a specific image tag.
+ Parameters
+ component (string) - KBC component Id
+ tag (string) - Docker image tag to be used
### Run job [POST]
+ Request Stored configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"config": "next-order-1",
"row": "my-row"
}
+ Request Custom configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"configData": {
"storage": {
"input": {
"files": [
{
"query": "name: testfile.png"
}
]
},
"output": {
"tables": [
{
"source": "sliced.csv",
"destination": "out.c-main.data"
}
]
}
},
"parameters": {
"foo": "bar",
"script": null
}
"runtime": {
"baz": "bar"
}
}
}
+ Response 202 (application/json)
{
"id": "67121676",
"url": "https://syrup.keboola.com/queue/job/67121676",
"status": "waiting"
}
# Group Debug
Debug API call is handy when developing or troubleshooting components.
## Debug Component [/{componentId}/debug]
This API call accepts the same body attributes as the [Run call](#reference/run/create-a-job/run-job),
see [Examples](https://documenter.getpostman.com/view/3086797/kbc-samples/77h845D#4c9c7c9f-6cd6-58e7-27e3-aef62538e0ba). It
creates a job that prepares the `data` folder including the serialized configuration files. Then it compresses
the `data` folder and uploads it to your project's Files in Storage. This way you will get a snapshot of what
the `data` folder looked like before the component started. If processors are used, a snapshot of the data folder
is created before each processor. After the entire component finishes, another snapshot is made.
For example, if you run component `a` with processor `b` and `c` in the `after` section, you will receive:
- `stage_0` file with contents of the data folder before component `a` was run
- `stage_1` file with contents of the data folder before processor `b` was run
- `stage_2` file with contents of the data folder before processor `c` was run
- `stage_output` file with contents of the data folder before output mapping was about to be performed (after `c` finished).
If configuration rows are used, then the above is repeated for each configuration row. If the job finishes with
and error, only the stages before the error are uploaded.
This API call does not upload any tables or files to Storage. I.e. when the component finishes, its output is
discarded and the output mapping to storage is **not performed**. This makes this API call generally very safe to call, because
it cannot break the KBC project in any way. However keep in mind, that if the component has any outside side effects, these
will get executed. This applies typically to writers which will write the data into the external system even with this debug API call.
Note that the snapshot archive will contain all files in the `data` folder including any temporary files produced be the component.
The snapshot will not contain the output `state.json` file. This is because the snapshot is made before a component is run
where the out state of the previous component is not available any more. Also note that all encrypted values are removed
from the configuration file and there is no way to retrieve them. It is also advisable to run this command with limited input
mapping so that you don't end up with gigabyte size archives.
+ Parameters
+ componentId (string) ... KBC component Id
### Create a debug job [POST]
+ Request Stored configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"config": "next-order-1",
"row": "my-row"
}
+ Request Custom configuration (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"configData": {
"storage": {
"input": {
"files": [
{
"query": "name: testfile.png"
}
],
"tables": [
{
"source": "in.c-redshift.table1",
"destination": "table1.csv"
},
{
"source": "in.c-redshift.some-table",
"destination": "table2.csv"
}
]
},
"output": {
"files": [
{
"source": "file.csv",
"tags": [
"processed-file",
"csv"
]
}
],
"tables": [
{
"source": "sliced.csv",
"destination": "out.c-main.data"
}
]
}
},
"parameters": {
"foo": "bar",
"script": null
}
}
}
+ Response 202 (application/json)
{
"id": "67121676",
"url": "https://syrup.keboola.com/queue/job/67121676",
"status": "waiting"
}
# Group Actions
Runs a Docker image synchronously, waits for the result and returns as response. This part of the API has different base URL! On the US stack it is https://docker-runner.keboola.com/.
If you are using other stacks, you need to use the [correct URL](https://developers.keboola.com/overview/api/#regions-and-endpoints).
## Run custom component action [/{image}/action/{action}]
+ Parameters
+ image (string) ... KBC component Id
+ action (string) ... Action name
### Process action [POST]
The body attribute `configData` contains component configuration.
You need to provide the whole configuration, linking a stored configuration is not supported.
+ Request Process action (application/json)
+ Headers
X-StorageApi-Token: 123456
+ Body
{
"configData": {
"parameters": {}
}
}
+ Response 200 (application/json)
{
"response": 1
}