Skip to content
This repository has been archived by the owner on Dec 22, 2020. It is now read-only.

Adding MySQL and Mongo slave functionality #37

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

smchugh
Copy link

@smchugh smchugh commented Feb 12, 2014

Here is a patch to add in MySQL functionality along with a few other things we needed.

Changes:

  • Copy to MySQL instead of Postgres, which includes adapter-specific error handling
  • Column options (null, index, default)
  • Reading from a Mongo replica slave

Thanks for taking a look.

Steve McHugh added 3 commits February 6, 2014 18:01
  - Fix issues with primary key length
  - Added checks for adapter type (postgres, mysql2, ...) when code paths diverge
  - Added mysql-specific and generic non-postgres initial load scripts
  - Added mysql-specific and (untested) generic non-postgres check for duplicate key error. Used to check if the oplog has changed since the last run
  - Added the ability to specify column as an index, null or not null, and a default value
  - Condensed initial input copy to only use the INSERT syntax for all non-postgres adapters
  - Improved readme documentation
 - Added the ability for the mongo instance being read from to be a replica slave
@nelhage
Copy link
Contributor

nelhage commented Feb 12, 2014

Thanks for the patchset! I'm super busy right now, but I'll try to look at this in more detail by early next week at the latest.

@smchugh
Copy link
Author

smchugh commented Feb 13, 2014

Thanks for taking a look at this. The Travis CI error seems to be from the fact that it's looking for 0.3.1 rather than 0.3.2. I'll change the version file to use 0.3.1 when I go in and remove the slaveOk commit

@nelhage
Copy link
Contributor

nelhage commented Feb 21, 2014

I'm fine taking this PR if you fix the things I've mentioned. I'd ideally like to see the code factored out so we have backend-specific adapters that implement a common interface, but I'm happy to merge this for now without that.

@goutamdan
Copy link

During the initial import, the query for bulk copying always fails because of syntax errors caused by wrong quoting. The field names have double quotes and the values have no quotes at all. So this always falls back to the update and insert queries. This makes initial import extremely slow. Any plans to fix this?

@smchugh
Copy link
Author

smchugh commented Jul 11, 2014

@goutamdan We ended up not using this tool, so the projected ended up dropped. I'd ideally like to pick it back up on my own time, but haven't had the chance. If you wanted to pick this up, I'd be more than glad to accept a PR into this branch.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Steve McHugh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants