Command to deduplicate contacts #5344

alanhartless · 2017-11-17T00:07:17Z

Q	A
Bug fix?
New feature?	y
Automated tests included?	y
Related user documentation PR URL	todo
Related developer documentation PR URL
Issues addressed (#s or URLs)
BC breaks?
Deprecations?

Description:

This PR adds a command php app/console mautic:contacts:dedup to find contacts with the same unique identifiers and merge them. By default, the records are merged from oldest into newest but that can be changed with the flag --newest-into-oldest.

Regardless of which record is merged into which, the latest modified record's custom field data is used (except where it is empty as we will error on the side of not completely losing data). Also the latest last active date is preserved and the oldest date identified date.

Contacts with conflicting unique identifiers when there are more than one unique identifier are not merged.

Steps to test this PR:

Hack the database to update multiple contacts with the same email OR manually edit through the UI to ensure there are multiple contacts with the same unique identifiers.
Run the command and they should be merged along with all of their behavior data.

List deprecations along with the new alternative:

LeadModel::mergeLeads is deprecated. ContactMerger should be used instead

dongilbert · 2017-11-17T01:04:39Z

+1 works well. Tentative approval pending UT's

javjim · 2017-11-17T01:40:46Z

works properly for me

luizeof · 2017-12-07T13:55:35Z

@alanhartless .... 2.12.0 fix duplicating contacts on import .... this pull will be released on 2.12.0 too?

Thanks ;)

vesper8 · 2018-04-07T14:25:29Z

I need to use this, I have a bunch of duplicate contacts and am looking for a safe way to merge them

2.12 came out already but this didn't make it in? How can I use this now? I must create a fork or is there another way?

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Command/DedupCommand.php

+    protected function execute(InputInterface $input, OutputInterface $output)
+    {
+        /** @var ContactDeduper $deduper */
+        $deduper        = $this->getContainer()->get('mautic.lead.deduper');


Can we use DI for new commands?

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Command/DedupCommand.php

+use Symfony\Component\Console\Input\InputOption;
+use Symfony\Component\Console\Output\OutputInterface;
+
+class DedupCommand extends ModeratedCommand


Is the dedup real word in English? I think DeduplicateCommand would be a better name?

Probably more slang but I can rename it.

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Deduplicate/ContactDeduper.php

+    /**
+     * @var ContactMerger
+     */
+    private $merger;


Any reason why not to use $contactMerger?

No reason why not. I'll change it.

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Deduplicate/ContactDeduper.php

+    /**
+     * @var LeadRepository
+     */
+    private $repository;


$leadRepository?

No reason why not. I'll change it.

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Exception/ValueNotMergeable.php

+
+namespace Mautic\LeadBundle\Exception;
+
+class ValueNotMergeable extends \Exception


Should be ValueNotMergeableException

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Exception/ValueNotMergeable.php

+     */
+    public function __construct($newerValue, $olderValue)
+    {
+        parent::__construct(var_export($newerValue, true).' / '.var_export($olderValue, true));


Why is var_export here? Should be a comment here I think, if there is a real reason.

Just to add in the message what the value actually was. $newerValue can be anything so var_export converts it to a string for the exception message.

Maxell92 · 2018-04-26T17:21:00Z

app/bundles/LeadBundle/Model/LeadModel.php

+     *
+     * @return Lead
+     */
+    public function mergeLeads(Lead $lead, Lead $lead2, $autoMode = true)


Why is this change here? I think this method just should call the new method, right?

It was relocated from elsewhere in the class. But there is a circular dependency if I use the new service here. Will have to create some kind of legacy class just to get around that.

I had to address this by passing the container to a legacy class to keep from having the circular dependency. Will likely need to leverage this class for some other methods in the LeadModel to remove more deprecated code/circular dependencies. I want to get rid of checkForDuplicateContact as well and use the ContactDeduper class instead.

…leads

Maxell92

Works for me

escopecz

The code looks nice. Will test it. Just found minor issues in the doc blocks.

escopecz · 2018-05-08T15:39:43Z

app/bundles/LeadBundle/Command/DeduplicateCommand.php

+<?php
+
+/*
+ * @copyright   2017 Mautic Contributors. All rights reserved


should read 2018. Also check other file annotations.

escopecz · 2018-05-08T15:39:43Z

app/bundles/LeadBundle/Command/DeduplicateCommand.php

+
+/*
+ * @copyright   2017 Mautic Contributors. All rights reserved
+ * @author      Mautic, Inc.


Shouldn't be Inc, or should it? Also check other file annotations.

Well technically it is Mautic, Inc contributing this code. So I think this is fine.

Do we want every company/individual who creates/modifies the code to add itself as the author? It says "Mautic" on all files I randomly checked. Maybe @dbhurley could way in.

I just saw Mautic, Inc. on another file from 2014. So if that's not new, let's ignore this discussion.

escopecz · 2018-05-08T15:39:43Z

app/bundles/LeadBundle/Deduplicate/ContactMerger.php

+     * @param LeadModel             $leadModel
+     * @param MergeRecordRepository $repo
+     * @param LoggerInterface       $logger
+     */


Dispatcher param is missing in the docblock

escopecz · 2018-05-09T07:33:51Z

app/bundles/LeadBundle/Deduplicate/ContactDeduper.php

+<?php
+
+/*
+ * @copyright   2017 Mautic Contributors. All rights reserved


Sorry to be PITA, but could you check all the file annotations?

I think I technically wrote these files in 2017 :-)

escopecz

Works fine for me. Thanks Alan!

alanhartless added feature A new feature for inclusion in minor or major releases needs-automated-tests PR's that need automated tests before they can be merged WIP PR's that are not ready for review and are currently in progress labels Nov 17, 2017

alanhartless changed the title ~~Command to deduplicate contacts~~ WIP - Command to deduplicate contacts Nov 17, 2017

alanhartless force-pushed the feature-dedup-command branch 2 times, most recently from 789cf21 to 9ab8020 Compare November 23, 2017 16:37

alanhartless force-pushed the feature-dedup-command branch from 123476f to 5ed1e32 Compare April 17, 2018 21:25

alanhartless added this to the 2.14.0 milestone Apr 17, 2018

alanhartless changed the title ~~WIP - Command to deduplicate contacts~~ Command to deduplicate contacts Apr 19, 2018

alanhartless force-pushed the feature-dedup-command branch from 21bb42f to f103a5b Compare April 19, 2018 14:24

mautibot added the pending-test-confirmation PR's that require one test before they can be merged label Apr 26, 2018

Maxell92 suggested changes Apr 26, 2018

View changes

alanhartless added 10 commits April 26, 2018 14:56

Command to deduplicate contacts

6d3cdea

Plugged a memory leak due to Doctrine query caching

8ce2aeb

Prevent a PHP notice with unpublished fields

f4699a0

Don't allow overwriting ID

8fbd95f

Don't merge id

2d9152a

Return a count of merged contacts

cbe7d5a

Docblock update

9ad5592

Prevent overwriting of values

527719c

Don't use blank values for comparison when searching for dups

234e54e

Increment duplicate count upon merge

e95e085

alanhartless added 9 commits April 26, 2018 14:56

Added debugging

c3754dd

Refactored a bit and added tests

6273cd5

Fixed variable

d850c77

CS fixes

ddb139c

Use DI for command

ce3646c

Renamed class variables

b52ae16

Moved and renamed ValueNotMergeable exception

7be6d90

Fixed docblock

6bad9a0

Use a LegacyLeadModel to work around circular dependency for merging …

3a7151a

…leads

alanhartless force-pushed the feature-dedup-command branch from bdc07ef to 3a7151a Compare April 26, 2018 19:57

Fixed DI

0d1072e

Maxell92 approved these changes May 8, 2018

View changes

mautibot added the code-review PR's that require a code review before merging label May 8, 2018

escopecz self-assigned this May 8, 2018

escopecz reviewed May 8, 2018

View changes

Fixed a couple docblock issues

ef02332

escopecz reviewed May 9, 2018

View changes

Updated copyright date

ce4d197

mautibot removed the code-review PR's that require a code review before merging label May 9, 2018

escopecz approved these changes May 9, 2018

View changes

escopecz added ready-to-commit PR's with 2 successful tests, 1 approval, automated tests and docs and is ready to be merged and removed pending-test-confirmation PR's that require one test before they can be merged ready-to-test PR's that are ready to test labels May 9, 2018

dbhurley added Backlog and removed ready-to-commit PR's with 2 successful tests, 1 approval, automated tests and docs and is ready to be merged labels May 18, 2018

dbhurley removed this from the 2.14.0 milestone May 18, 2018

alanhartless removed the Backlog label May 25, 2018

alanhartless added this to the 2.14.0 milestone May 25, 2018

alanhartless merged commit a3427ee into mautic:staging May 25, 2018

kuzmany mentioned this pull request Sep 7, 2019

Mautic site 2.15.1 error after upload MauticCustomDeduplicateBundle mtcextendee/mautic-custom-deduplicate-bundle#1

Closed

Command to deduplicate contacts #5344

Command to deduplicate contacts #5344

alanhartless commented Nov 17, 2017 •

edited

dongilbert commented Nov 17, 2017

javjim commented Nov 17, 2017

luizeof commented Dec 7, 2017

vesper8 commented Apr 7, 2018

Maxell92 Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

Maxell92 Apr 26, 2018

alanhartless Apr 26, 2018

alanhartless Apr 26, 2018 •

edited

Maxell92 left a comment

escopecz left a comment

escopecz May 8, 2018

escopecz May 8, 2018

alanhartless May 9, 2018

escopecz May 9, 2018

escopecz May 9, 2018

escopecz May 8, 2018

escopecz May 9, 2018

alanhartless May 9, 2018

escopecz left a comment


		namespace Mautic\LeadBundle\Exception;

		class ValueNotMergeable extends \Exception

Command to deduplicate contacts #5344

Command to deduplicate contacts #5344

Conversation

alanhartless commented Nov 17, 2017 • edited

Description:

Steps to test this PR:

List deprecations along with the new alternative:

dongilbert commented Nov 17, 2017

javjim commented Nov 17, 2017

luizeof commented Dec 7, 2017

vesper8 commented Apr 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanhartless Apr 26, 2018 • edited

Choose a reason for hiding this comment

Maxell92 left a comment

Choose a reason for hiding this comment

escopecz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

escopecz left a comment

Choose a reason for hiding this comment

alanhartless commented Nov 17, 2017 •

edited

alanhartless Apr 26, 2018 •

edited