Skip to content

Commit

Permalink
rewrite data migration section.
Browse files Browse the repository at this point in the history
  • Loading branch information
c0c0n3 committed Oct 16, 2019
1 parent bc0695b commit c43aa58
Show file tree
Hide file tree
Showing 2 changed files with 73 additions and 39 deletions.
75 changes: 70 additions & 5 deletions docs/manuals/admin/dataMigration.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,74 @@
# Data-Migration-Tool
# Data Migration

Data-Migration-Tool is designed to automatically migrate data stored in [STH-Comet](https://github.com/telefonicaid/fiware-sth-comet) to [QuantumLeap](https://github.com/smartsdk/ngsi-timeseries-api) database. After data migration, data can be accessed by using QuantumLeap's [API](https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb/0.2).
A few tools are available to assist with migrating data to QuantumLeap.

This tool is developed in [Java](https://en.wikipedia.org/wiki/Java_(software_platform)) using [Eclipse](https://www.eclipse.org/). A python script is used to convert data in [MongoDB](https://github.com/mongodb/mongo) to be compatible for [CrateDB](https://github.com/crate/crate).

Tool is available [here](https://github.com/Data-Migration-Tool/STH-to-QuantumLeap).
## Migrating STH Comet data

User guide for the tool is available [here](https://github.com/Data-Migration-Tool/STH-to-QuantumLeap/blob/master/docs/manuals/README.md).
[Data-Migration-Tool][dmt] is a program designed to automatically
migrate data stored in [STH-Comet][comet] to a QuantumLeap [CrateDB][crate]
database. After migration, the data can be accessed through QuantumLeap's
[REST API][ql-api].

[Data-Migration-Tool][dmt] is developed in [Java][java] using the
[Eclipse IDE][eclipse]. A Python script transforms data in [MongoDB][mongo]
into the format expected by the QuantumLeap [CrateDB][crate] back end.

The tool can be downloaded [here][dmt] and the accompanying user guide
is also [available online][dmt-man].


## Migrating from QuantumLeap Crate to Timescale

QuantumLeap provides a self-contained Python script to help with
migrating tables from a QuantumLeap CrateDB database to a QuantumLeap
Timescale database. The script is located in the `timescale-container`
directory and is called `crate-exporter.py`.
It exports rows in a given Crate table and generates, on `stdout`,
all the SQL statements needed to import that data into Timescale.
These include creating a corresponding schema, table and hypertable
in PostgreSQL as needed. Note that the script generates DDL statements
that, when executed, will result in the exact same table structures
the QuantumLeap Timescale back end would have generated on seeing
NGSI entities corresponding to the rows stored in the Crate table.

Here's an example usage

$ python crate-exporter.py --schema mtyoutenant --table etdevice \
> mtyoutenant.etdevice-import.sql

where we export all the rows in the Crate table `mtyoutenant.etdevice`.
The generated file contains all the SQL statements to recreate the
table and insert the data in Timescale. You may want to put this file
in the `quantumleap-db-setup` script's init directory so that data
are migrated automatically for you when you bootstrap the QuantumLeap
DB on Timescale as explained in the [Timescale section][ts-admin].

By default the script exports all the rows in the Crate table, but
you can also use the `--query` argument to specify a query to select
only a subset of interest as shown below:

$ python crate-exporter.py --schema mtyoutenant --table etdevice --query \
"SELECT * FROM mtyoutenant.etdevice where time_index > '2019-04-15';"




[comet]: https://github.com/telefonicaid/fiware-sth-comet
"FiWare STH Comet Home"
[crate]: https://crate.io
"CrateDB Home"
[dmt]: https://github.com/Data-Migration-Tool/STH-to-QuantumLeap
"Data-Migration-Tool Home"
[dmt-man]: https://github.com/Data-Migration-Tool/STH-to-QuantumLeap/blob/master/docs/manuals/README.md
"Data-Migration-Tool Manual"
[eclipse]: https://www.eclipse.org/
"Eclipse Home"
[java]: https://en.wikipedia.org/wiki/Java_(software_platform)
"Wikipedia - Java"
[mongo]: https://github.com/mongodb/mongo
"MongoDB Home"
[ql-api]: https://app.swaggerhub.com/apis/smartsdk/ngsi-tsdb/0.2
"QuantumLeap REST API"
[ts-admin]: ./timescale.md
"QuantumLeap Timescale"
37 changes: 3 additions & 34 deletions docs/manuals/admin/timescale.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ possible to dynamically select, at runtime, which storage back end to
use (Crate or Timescale) depending on the tenant who owns the entity
being persisted. Also, QuantumLeap ships with tools to automate the
Timescale back end setup and generate Crate-to-Timescale migration
scripts.
scripts---details in the [Data Migration section][admin.dm].


## Operation overview
Expand Down Expand Up @@ -144,41 +144,10 @@ Crate. Any tenant other than `t1`, `t2`, or `t3` gets the default
Crate back end.


## Migrating from Crate

QuantumLeap provides a self-contained Python script to help with
migrating tables from Crate to Timescale. The script is located
in the `timescale-container` directory and is called `crate-exporter.py`.
It exports rows in a given Crate table and generates, on `stdout`,
all the SQL statements needed to import that data into Timescale.
These include creating a corresponding schema, table and hypertable
in PostgreSQL as needed. Note that the script generates DDL statements
that, when executed, will result in the exact same table structures
the QuantumLeap Timescale back end would have generated on seeing
NGSI entities corresponding to the rows stored in the Crate table.

Here's an example usage

$ python crate-exporter.py --schema mtyoutenant --table etdevice \
> mtyoutenant.etdevice-import.sql

where we export all the rows in the Crate table `mtyoutenant.etdevice`.
The generated file contains all the SQL statements to recreate the
table and insert the data in Timescale. You may want to put this file
in the `quantumleap-db-setup` script's init directory so that data
are migrated automatically for you when you bootstrap the QuantumLeap
DB on Timescale as explained earlier.

By default the script exports all the rows in the Crate table, but
you can also use the `--query` argument to specify a query to select
only a subset of interest as shown below:

$ python crate-exporter.py --schema mtyoutenant --table etdevice --query \
"SELECT * FROM mtyoutenant.etdevice where time_index > '2019-04-15';"




[admin.dm]: ./dataMigration.md
"QuantumLeap Data Migration"
[postgres]: https://www.postgresql.org
"PostgreSQL Home"
[postgis]: https://postgis.net/
Expand Down

0 comments on commit c43aa58

Please sign in to comment.