Skip to content
Data Transformation/Migration Tool
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bsonutil
db
mock
model
.gitignore
LICENSE
README.rst
anser.go
anser_test.go
config.go
config_test.go
dependency_manager.go
dependency_manager_test.go
dependency_network.go
dependency_network_test.go
environment.go
environment_test.go
evergreen.yaml
example_test.go
generator.go MAKE-802: update to use newer amboy interfaces Jun 4, 2019
generator_manual.go
generator_manual_test.go
generator_simple.go
generator_simple_test.go
generator_stream.go
generator_stream_test.go
makefile
migration.go
migration_helper.go MAKE-496: Fix typos (#13) Oct 31, 2018
migration_helper_mock_test.go
migration_helper_test.go
migration_job_manual.go MAKE-380: update amboy and job interface (#10) Mar 30, 2018
migration_job_manual_test.go
migration_job_simple.go
migration_job_simple_test.go
migration_job_stream.go
migration_job_stream_test.go

README.rst

anser -- Database Migration Toolkit

Summary

Anser is a toolkit for managing evolving data sets for applications. It focuses on on-line data transformations and providing higher-level tools to support data modeling and access.

For the Evergreen project, anser allows us to treat these routine data migrations, data back fills, and retroactively changing the schema of legacy data as part of application code rather than one-off shell scripts.

Overview

In general, anser migrations have a two-phase approach. First a generator runs with some configuration and an input query to collect input documents and creation migration jobs. Then, the output of these generators, are executed in parallel

You can define generators either directly in your own code, or you can use the configuration-file based approach for a more flexible approach.

Concepts

There are three major types of migrations:

  • simple: these migrations perform their transformations using MongoDB's update syntax. Use these migrations for very basic migrations, particularly when you want to throttle the rate of migrations and avoid the use of larger difficult-to-index multi-updates.
  • manual: these migrations call a user-defined function on a bson.RawDoc representation of the document to migrate. Use these migrations for more complex transformations or those migrations that you want to write in application code.
  • stream: these migrations are similar to manual migrations; however, they pass a database session and an iterator to all documents impacted by the migration. These jobs offer ultimate flexibility.

Internally these jobs execute using amboy infrastructure and make it possible to express dependencies between migrations. Additionally the MovingAverageRateLimitedWorkers and SimpleRateLimitingWorkers were developed to support anser migrations, as well as the adaptive ordering local queue which respects dependency-driven ordering.

Considerations

While it's possible to do any kind of migration with anser, we have found the following properties to be useful to keep in mind when building migrations:

  • Write your migration implementations so that they are idempotent so that it's possible to run them multiple times with the same effect.
  • Ensure that generator queries are supported by indexes, otherwise the generator processes will force collection scans.
  • Rate-Limiting, provided by configuring the underlying amboy infrastructure, focuses on limiting the number of migration (or generator) jobs executed, rather than limiting the jobs based on their impact.
  • Use batch limits. Generators have limits to control the number of jobs that they will produce. This is particularly useful for tests, but may have adverse effects on job dependency, particularly if logical migrations are split across more than one generator function.

Installation

Anser uses grip for logging and amboy for task management. Because anser does not vendor these dependencies, you should also vendor them.

Resources

Please consult the godoc for most usage. Most of the API is in the top level package; however, please do also consider the model and bsonutil package.

Additionally you can use the interfaces db package as a wrapper for mgo to access MongoDB which allows you to use mocks as needed for testing without depending on a running database instance.

Project

Please file feature requests and bug reports in the MAKE project of the MongoDB Jira instance. This is also the place to file related amboy and grip requests.

Future anser development will focus on supporting additional migration workflows, supporting additional MongoDB and BSON utilities, and providing tools to support easier data-life-cycle management.

You can’t perform that action at this time.