Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
cascading-dbmigrate makes it easy to run Cascading flows on sql tables with a primary key of an int or a long. We use it at BackType to migrate data from our databases to HDFS. Usage: To read data from a database in a Cascading flow, use DBMigrateTap. DBMigrateTap's constructor has the following signature: DBMigrateTap(int numChunks, String dbDriver, String dbUrl, String username, String pwd, String tableName, String pkColumn, String columnNames) numChunks: The number of splits to create of the database. This will correspond to the number of mappers created to read the database. dbDriver: For example, "com.mysql.jdbc.Driver" dbUrl: The connection url to connect to your database username: Username to connect to your database password: Password to connect to your database tableName: The table to read during the flow pkColumn: The name of the primary key column of the table columnNames: The names of the columns you want to read into the flow The tap will emit tuples containing one field for each column read, the field names being the column names. Building: To build cascading-dbmigrate, follow these instructions: 1. Set HADOOP_HOME environment variable to the root directory of your hadoop distribution 2. Set CASCADING_HOME environment variable to the root directory of your cascading distribution 3. ant jar This will produce a single jar called cascading_dbmigrate.jar in the build/ directory. Thanks to Chris Wensel for his help in developing this project.