CRv2: Enable the full pipeline

[hacodeorg]

PLC-839 Enable the full contact rollups V2 pipeline, which includes the 3 main steps:

1. Collect and process contact data
2. Sync new contacts to Pardot (creating new Pardot prospects).
3. Sync updated contacts to Pardot (updating existing Pardot prospects).

Monitoring

Monitor Pardot

• Monitor Pardot updates at the Website tracking campaign, which is the campaign assigned to contact rollups. Use filters = 'all prospects' + 'created/updated' + 'today'
• To see updates from all campaigns, go to here.

Monitor the production database impact

• CloudWatch -> Dashboard -> Aurora-Database metrics. Metrics we care about: Write Latency/Throughput, Read Latency/Throughput and Query load.
• Use Percona Query Analytics to monitor query performance.

Testing story

Manually test the full pipeline in production-daemon on 5/28/2020.

# Sync deleted and active prospects from Pardot to the production database.
# Note: save the current maximum prospect ID before syncing so we have a 
# milestone to redo sync operations if something fails.
max_id = ContactRollupsPardotMemory.maximum(:pardot_id)
ContactRollupsPardotMemory.download_deleted_pardot_prospects
ContactRollupsPardotMemory.download_pardot_prospects(max_id)

# Dry-run
# Note: Set `is_dry_run` to false to really send data to Pardot.
cr = ContactRollupsV2.new(is_dry_run: true)  # use false later
cr.collect_and_process_contacts
cr.sync_new_contacts_with_pardot  # add 165,810 prospects
cr.sync_updated_contacts_with_pardot   # update 43 prospects
cr.report_results

Links

• Slack discussion with Marketing on deleting 700K prospects before running Contact Rollups so we don't go over the 3.2M mailable prospects limit.
• Resolved Honeybadger error for query time-out issue.

Reviewer Checklist:

• Tests provide adequate coverage
• Code is well-commented
• New features are translatable or updates will not break translations
• Relevant documentation has been added or updated
• User impact is well-understood and desirable
• Pull Request is labeled appropriately
• Follow-up work items (including potential tech debt) are tracked and linked

[islemaster]

🎉

[bencodeorg]

I think typo in description but looks great!

# Note: Set is_dry_run to true to really send data to Pardot.

[hacodeorg]

I think typo in description but looks great!

# Note: Set is_dry_run to true to really send data to Pardot.

Fixed ⚔️