Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding sync functionality to copy only new/updated records #35

Merged
merged 4 commits into from Mar 25, 2014

Conversation

Projects
None yet
2 participants
@hammady
Copy link
Contributor

hammady commented Mar 6, 2014

With the new 'sync' command, mongify can be ran repeatedly as it only copies new records or updates the updated ones. The only prerequisite is to have an 'updated_at' column that should be updated each time a record is updated. For Ruby ActiveRecord, this is the default behaviour and no extra coding is required.
For non ActiveRecord database manipulation, such field should be created in the schema and updated on each update with the current system time.
Sync works by creating a special table in the Sql database to store the last updated_at timestamp for each synced table. Each time it is invoked, it only selects rows having a bigger updated_at. It keeps the pre_mongified_id in the mongoDB because it is always needed for consecutive sync runs.

hammady added some commits Mar 6, 2014

Adds a new mongify command: 'sync' that will insert/update new or upd…
…ated records only, based on a mandatory updated_at column. The command can be safely ran repeatedly as it only syncs changes.
@anlek

This comment has been minimized.

Copy link

anlek commented on 744ade5 Mar 6, 2014

This is great, however before I can accept it, it needs to have passing tests.

- added full tests for the sync process
- added support for postgres
- redirecting rspec deprecation warnings to a file
@hammady

This comment has been minimized.

Copy link
Contributor Author

hammady commented Mar 13, 2014

Here we go. Please have a look on the new specs, I am relatively new to rspec.
Thank you.

@anlek anlek merged commit d765f69 into anlek:master Mar 25, 2014

@anlek

This comment has been minimized.

Copy link
Owner

anlek commented Mar 25, 2014

Hey Hammady, I've merged your code and added a few more notes into the README. I also added your name to the CHANGE LOG.
I'm looking at making Mongify work on Ruby 2.x before releasing version 1.1 to the public.

@hammady

This comment has been minimized.

Copy link
Contributor Author

hammady commented Mar 25, 2014

I can see that 1.1 is already out!

Thanks
Hossam Hammady
http://twitter.com/hammady

On Tue, Mar 25, 2014 at 4:57 AM, Andrew Kalek notifications@github.comwrote:

Hey Hammady, I've merged your code and added a few more notes into the
README. I also added your name to the CHANGE LOG.
I'm looking at making Mongify work on Ruby 2.x before releasing version
1.1 to the public.


Reply to this email directly or view it on GitHubhttps://github.com//pull/35#issuecomment-38523295
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.