Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract email preferences into contact_rollups_raw #33557

Merged
merged 5 commits into from Mar 12, 2020

Conversation

bencodeorg
Copy link
Contributor

Basic logic to extract email preferences into contact_rollups_raw by executing bin/cron/build_contact_rollups_v2.

Testing story

Tested locally and on a newly created ad hoc connected to a production clone. email_preferences has close to a million rows in production -- some interesting findings:

  • using truncate in raw SQL (in this PR) instead of delete_all in ActiveRecord (not in this PR) takes .3 seconds instead of 14 seconds.
  • changing find_in_batches batch size from 1000 to 10000 has minimal impact on execution time (actually was 7 seconds slower, 337 vs 330)
  • using a raw SQL query for inserts (not in this PR) instead of batched active record queries (in this PR) significantly reduces run time (17 seconds vs. 5.5 minutes) -- here is SQL query that was used:
  insert into contact_rollups_raw(email, sources, data, data_updated_at, created_at, updated_at)
  select email, 'dashboard.email_preferences', json_object('opt_in', opt_in), updated_at, now(), now() 
  from email_preferences

General finding is raw SQL dramatically outperforming ActiveRecord.

Reviewer Checklist:

  • Tests provide adequate coverage
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

@hacodeorg
Copy link
Contributor

hacodeorg commented Mar 11, 2020

You can add contact-rollups label to this PR. It will be easier to search for all related PRs using this link

dashboard/app/models/contact_rollups_raw.rb Show resolved Hide resolved
dashboard/test/models/contact_rollups_raw_test.rb Outdated Show resolved Hide resolved
test 'bunch of email prefences works' do
3.times {|i| create :email_preference, email: "contact_#{i}@rollups.com"}
ContactRollupsRaw.extract_email_preferences
assert 3, ContactRollupsRaw.count
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also want to do ContactRollupsRaw.delete_all to assure the table is empty before inserting to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that an assumption of our test framework (cleanup/teardown of database before/after every test)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I don't know what is our current test cleaning strategy and how it behaves in local dev machine, in Drone and in test machine. Would be very helpful for me and the team if you could do some research and share what you find.

Copy link
Contributor

@hacodeorg hacodeorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks, Ben!

@bencodeorg bencodeorg merged commit cd6d744 into staging Mar 12, 2020
@bencodeorg bencodeorg deleted the contact-rollups-raw-logic branch March 12, 2020 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants