Skip to content

Latest commit

 

History

History
152 lines (101 loc) · 4.95 KB

import.md

File metadata and controls

152 lines (101 loc) · 4.95 KB
Raw

Import command

Import command can be used to import data from any supported database source into appbase.io/ES cluster. It goes like -

abc import --src_uri {URI} --src_type {DBType} --tail [URI|Appname]

To view the complete list of input parameters supported, use -

abc import --help

At the time of writing, the list of parameters supported looks like -

--config=                                    Path to external config file, if specified, only that is used
--log.level="info"                           Only log messages with the given severity or above. Valid levels: [debug, info, error]
--replication_slot=standby_replication_slot  [postgres] replication slot to use
--src_filter=.*                              Namespace filter for source, accepts a regex
--src_type=postgres                          type of source database
--src_uri=http://user:pass@host:port/db      url of source database
--src_username="username"                    username for source connection
--src_password="password"                    password for source connection
--src_realm="realm"                          realm for source connection
--sac_path=./ServiceAccountCredentials.json  Path to the service account credentials file obtained after creating a firebase app.
--tail=false                                 allow tail feature
--test=false                                 if set to true, only pipeline is created and sync is not started. Useful for checking your configuration
--transform_file=                             URI of transform file to use
--typename=mytype                            [csv] typeName to use

Note that you only need to set the parameters that are required for the source database type. For example, you don't set replication_slot when taking CSV as the source.

Note - --src_filter switch can be used as a namespace filter for source and accepts a regex. For e.g., if you want to filter one or more tables from a collection, you can specify them as --src_filter="table1|table2|table3"

Note - Help for transform_file is available here.

Examples

In the examples below, the destination URI has been provided as:

https://USERID:PASS@scalr.api.appbase.io/APPNAME

Instead of using your user-id and password, you could also use your admin API key.

https://admin-API-key@scalr.api.appbase.io/APPNAME

You can find your admin API key inside your app page at appbase.io under Security -> API Credentials.

CSV

abc import --src_type=csv --typename=csvTypeName --src_uri="file.csv" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

ElasticSearch

abc import --src_type=elasticsearch --src_uri="http://USER:PASS@HOST:PORT/INDEX" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

We can also use an Appbase app as source.

abc import --src_type=elasticsearch --src_uri="https://USER:PASS@scalr.api.appbase.io/APPNAME2" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

We can even use the app's name as URI once we are logged in.

abc import --src_type=elasticsearch --src_uri=APPNAME2 APPNAME

Cloud Firestore

abc import --src_type=firestore --sac_path="/home/johnappleseed/ServiceAccountKey.json" --src_filter="users" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

Kafka

abc import --src_type=kafka --src_uri="kafka://USER:PASS@HOST:PORT/TOPIC1,TOPIC2" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

MongoDB

abc import --src_type=mongodb -t --src_uri="mongodb://USER:PASS@HOST:PORT/DB" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

MSSQL

abc import --src_type=mssql --src_uri="sqlserver://USER:PASSWORD@SERVER:PORT?database=DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

For more source URL patterns, see go-mssqldb's GitHub page.

MySQL

abc import --src_type=mysql --src_uri="USER:PASS@tcp(HOST:PORT)/DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

For more source URL patterns, see go-sql-driver/mysql's GitHub page.

Postgres

abc import --src_type=postgres -t --replication_slot="standby_replication_slot" --src_uri="postgresql://USER:PASS@HOST:PORT/DBNAME" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

Redis

abc import --src_type=redis --src_uri="redis://USER:PASS@HOST:PORT/DBNUMBER" "https://USER:PASS@scalr.api.appbase.io/APPNAME"

Using a config file

abc import --config=test.env

File extension doesn't matter. The file test.env should be an INI/ENV like file with key value pair containing the values of attributes required for importing. Example of a test.env file is --

src_type=csv
src_uri=/full/path/to/file.csv
typename=csvTypeName

dest_type=elasticsearch
dest_uri=https://USER:PASS@scalr.api.appbase.io/APPNAME

Note that the key names are same as what we have in import parameters.