Further-deduplicate-the-incremental-rows

[Tian-2017]

• Deduplicate by ID and last updated.
• use standard logger instead of glue logger, otherwise it causes the unit test error because prepare_increments doesn't have logger;
• Remove data quality keyword from the finally clause to avoid the confusion
• Add coalesce(300) to avoid causing (Service: Amazon S3; Status Code: 503) when have many workers

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Tian-2017]

Hey @timburke-hackit could you review it again? I've checked the data - looks good now.
Adding a new commit - this is the version running in the prod :

• Changed duplicate_ids.count() > 0 to duplicate_ids.limit(1).count() > 0 (to reduce shuffle)
• Added snapshot_df = snapshot_df.coalesce(300) (to reduce small files reaching the throttle of S3)

Thanks.