Skip to content

mattmccutchen/MigratingTable

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code
This branch is 7 commits ahead, 2 commits behind johnterickson:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
doc
 
 
 
 
 
 
 
 
 
 

MigratingTable

MigratingTable is a wrapper library for Azure Table storage that allows data to be migrated from one underlying table to another transparently to the application. See these slides for information about the motivation and design.

The initial focus in developing MigratingTable was to put it through systematic concurrency testing using P# to gain confidence in its correctness; we contributed a description of this case study of P# to the FAST 2016 paper "Uncovering Bugs in Distributed Storage Systems during Testing (Not in Production!)". Consequently, this open source release includes the P# test harness, but not some of the adapters necessary to actually use MigratingTable on real Azure tables. MigratingTable also accepts an optional parameter to enable various "bugs" to demonstrate the ability of the test to find them. As development continues, it may make sense to remove the "bug" code paths to make the code easier to read.

MigratingTable was initially developed by Matt McCutchen (personal email: matt@mattmccutchen.net) while an intern at Microsoft in summer 2015 with manager John Erickson John.Erickson@microsoft.com. We decided to make it available to the public even though we are out of time to prepare complete public documentation. If you're interested, by all means please contact us so we can prioritize improving the documentation or reviewing existing Microsoft internal documentation for public release.

Building and running the P# test

This procedure assumes you have Visual Studio 2015 installed on Windows.

  1. Run git submodule update --init to check out the fork of P# required by the test (if not already done).
  2. Open and build PSharp\PSharp.sln with Visual Studio.
  3. Open and build Migration.sln with Visual Studio. This should automatically download the required NuGet packages.
  4. Run:
PSharp\Binaries\Debug\PSharpCompiler.exe /s:Migration.sln /test:Migration

Add any desired P# flags, e.g., /i:1000 for number of iterations or /schseed:314159 to set the starting random seed for hopefully reproducible behavior. To enable one of the "bugs", add /testvar:mtablebug=N where N is the integer value of the MTableOptionalBug enum constant.

Using in production (if you dare)

Starting with a live application using the "old" table, to migrate to the "new" table:

  1. Write an adapter that wraps a CloudTable and implements IChainTable2 and an implementation of IConfigurationService. (Microsoft has both but they could use polishing.)
  2. Push MasterMigratingTable.INITIAL_CONFIGURATION to the configuration service.
  3. Update your application processes to use a MigratingTable instance backed by the same old and new tables and configuration service. This can be done one at a time; MigratingTable in its initial state is compatible with direct access to the old table.
  4. Have a "migrator" process construct a MasterMigratingTable and call MigrateAsync.
  5. Update your application processes to access the new table directly. This can also be done one at a time.

Known bugs in MigratingTable

None.

Known bugs in the test harness

  • A missing annotation in the MigratingTable code (an easy mistake to make, which I actually made once) will cause all machines to become blocked, and P# considers that a success. We need to enable liveness checking to catch this.

About

Azure Table Wrapper Library for live migration between backing tables

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • C# 99.9%
  • Batchfile 0.1%