Skip to content
A declarative data-migration package
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
docs
py.requirements
src/data_migrator
tests
.bandit
.bumpversion.cfg
.coveragerc
.editorconfig
.gitattributes
.gitignore
AUTHORS
CHANGELOG.md
CONTRIBUTING.md
LICENSE
MANIFEST.in
Makefile
NOTICE
README.rst
pylintrc
setup.cfg
setup.py

README.rst

Data-migrator (version 0.6.3.dev2) is a simple data-migration package for python lovers.

https://circleci.com/gh/schubergphilis/data-migrator.svg?style=shield
Updates Documentation Status https://api.codacy.com/project/badge/Grade/bf6030e9e7e248979607802880336611 https://api.codacy.com/project/badge/Coverage/bf6030e9e7e248979607802880336611

Data-migrator is a declarative DSL for table driven data transformations, set up as an open and extensive system. Use this to create data transformations for changing databases as a result of changing code, initial loads to datalakes (it contains a Kinesis provider)and more.

Example

Data-migrator assumes data is extracted and loaded with client access.

$ mysql source_db -E 'select id,a,b from table' -B  | python my_filter.py | mysql target_db

It than offers a wide range of primitives with default settings to build complex transformations fast, readable and extendable

from data_migrator import models, transform

class Result(models.Model):
  id   = models.IntField(pos=0) # keep id
  uuid = models.UUIDField()     # generate new uuid4 field
  a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())
  b    = models.StringField(pos=2, name='my_b')

if __name__ == "__main__":
  transform.Transformer(models=[Result]).process()

Installation

Execute the following command to install data-migrator with pip:

pip install data-migrator

See the Installation Instructions in Documentation for more instructions on installing, upgrading, and uninstalling data-migrator.

The project is maintained at GitHub.

Support and contribute

Questions, comments, bug reports and especially tested patches may be submitted directly to the issue tracker.

Everyone interacting with this codebase, issue trackers, chat rooms, and mailing lists is expected to follow the Code of Conduct.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.