Handle Pipedrive merging of Org, Person or Deal #227

PrenSJ2 · 2024-05-23T11:41:23Z

The Problem

The sales team often merge Pipedrive Objects (Organizations, Persons, Deals)

this causes the custom field hermes_id to be combined 123 --> '123, 456'

which in turn causes a validation error as we are expecting the hermes_id to be type FK_FIELD which is an int

Solution

In PipedriveEvent vaidate_obj, if hermes_id is a str we use the newly created handle_duplicate_hermes_ids to select a main object and update all the associated objects to have the main object id

Testing

run make restore-from-live
run uvicorn - python -m uvicorn app.main:app --reload
setup a postman request to post to localhost:8000/pipedrive/callback/

Test Deal

we need to create a new deal, do this on the hermes admin page
create a new meeting on hermes admin, and set the deal id to the new deal - meeting/create
then post to /pipedrive/callback/ with:

{
    "v": 1,
    "matches_filters": {
        "current": []
    },
    "meta": {
        "action": "updated",
        "object": "deal"
    },
    "current": {
        "id": 5712,
        "acv": null,
        "arr": null,
        "mrr": null,
        "label": "58",
        "title": "Tutor Company",
        "value": 1,
        "active": true,
        "org_id": 7663,
        "origin": "API",
        "status": "open",
        "channel": null,
        "deleted": false,
        "user_id": 15761943,
        "add_time": "2024-05-23 07:42:35",
        "cc_email": "tutorcruncher+deal5712@pipedrivemail.com",
        "currency": "GBP",
        "obj_type": "deal",
        "org_name": "Tutor Company",
        "stage_id": 72,
        "won_time": null,
        "lost_time": null,
        "origin_id": null,
        "person_id": 30169,
        "channel_id": null,
        "close_time": null,
        "org_hidden": false,
        "owner_name": "Sam",
        "visible_to": "3",
        "files_count": 0,
        "lost_reason": null,
        "notes_count": 0,
        "person_name": "Tutor Company Owner",
        "pipeline_id": 7,
        "probability": null,
        "rotten_time": null,
        "update_time": "2024-05-23 15:02:03",
        "acv_currency": null,
        "arr_currency": null,
        "mrr_currency": null,
        "person_hidden": false,
        "first_won_time": null,
        "local_won_date": null,
        "products_count": 0,
        "stage_order_nr": 4,
        "weighted_value": 1,
        "creator_user_id": 15708604,
        "followers_count": 1,
        "formatted_value": "£1",
        "local_lost_date": null,
        "activities_count": 0,
        "last_activity_id": null,
        "local_close_date": null,
        "next_activity_id": null,
        "stage_change_time": "2024-05-23 07:42:40",
        "last_activity_date": null,
        "next_activity_date": null,
        "next_activity_note": null,
        "next_activity_time": null,
        "next_activity_type": null,
        "participants_count": 1,
        "expected_close_date": "2024-06-06",
        "email_messages_count": 4,
        "done_activities_count": 0,
        "next_activity_subject": null,
        "next_activity_duration": null,
        "last_incoming_mail_time": "2024-05-23 14:52:21",
        "last_outgoing_mail_time": "2024-05-23 14:19:38",
        "undone_activities_count": 0,
        "weighted_value_currency": "GBP",
        "formatted_weighted_value": "£1",
        "177e444cef4dc6f0101a3ee51276eab1b02d335d": null,
        "306f085fd53c5ae792aeed30e3978484a26689d6": null,
        "5be1188db52a8c7f0ea49331eb391ae54aeabafc": "8303",
        "8d3fb70224a8f613a6cc7416e121996606868929": null,
        "90e1589738e5dc4ecec002c6f8424348f0c17143": null,
        "9d1836a71ef185ab2103eae0c902b4f4af6e1b03": null,
        "a8fb293ba4cfedc80a733b3411620e00921ba7d8": null,
        "abec03eee90b12d8b5658b7f988126e254a4bd79": null,
        "b0d672e094f9057c86f31a3383bedbb163ce718f": null,
        "c87e16b3aa96fb56c6d4bacf824d9a4fae2611d9": null,
        "f0a2dc95b937e2ec8ad860f0dd9cbfe4a338f818": null
    },
    "previous": {
        "id": 5712,
        "acv": null,
        "arr": null,
        "mrr": null,
        "label": "58",
        "title": "Tutor Company",
        "value": 1,
        "active": true,
        "org_id": 7663,
        "origin": "API",
        "status": "open",
        "channel": null,
        "deleted": false,
        "user_id": 15761943,
        "add_time": "2024-05-23 07:42:35",
        "cc_email": "tutorcruncher+deal5712@pipedrivemail.com",
        "currency": "GBP",
        "obj_type": "deal",
        "org_name": "Tutor Company",
        "stage_id": 72,
        "won_time": null,
        "lost_time": null,
        "origin_id": null,
        "person_id": 30169,
        "channel_id": null,
        "close_time": null,
        "org_hidden": false,
        "owner_name": "Sam",
        "visible_to": "3",
        "files_count": 0,
        "lost_reason": null,
        "notes_count": 0,
        "person_name": "Tutor Company Owner",
        "pipeline_id": 7,
        "probability": null,
        "rotten_time": null,
        "update_time": "2024-05-23 15:02:03",
        "acv_currency": null,
        "arr_currency": null,
        "mrr_currency": null,
        "person_hidden": false,
        "first_won_time": null,
        "local_won_date": null,
        "products_count": 0,
        "stage_order_nr": 4,
        "weighted_value": 1,
        "creator_user_id": 15708604,
        "followers_count": 1,
        "formatted_value": "£1",
        "local_lost_date": null,
        "activities_count": 0,
        "last_activity_id": null,
        "local_close_date": null,
        "next_activity_id": null,
        "stage_change_time": "2024-05-23 07:42:40",
        "last_activity_date": null,
        "next_activity_date": null,
        "next_activity_note": null,
        "next_activity_time": null,
        "next_activity_type": null,
        "participants_count": 1,
        "expected_close_date": "2024-06-06",
        "email_messages_count": 4,
        "done_activities_count": 0,
        "next_activity_subject": null,
        "next_activity_duration": null,
        "last_incoming_mail_time": "2024-05-23 14:52:21",
        "last_outgoing_mail_time": "2024-05-23 14:19:38",
        "undone_activities_count": 0,
        "weighted_value_currency": "GBP",
        "formatted_weighted_value": "£1",
        "177e444cef4dc6f0101a3ee51276eab1b02d335d": null,
        "306f085fd53c5ae792aeed30e3978484a26689d6": null,
        "5be1188db52a8c7f0ea49331eb391ae54aeabafc": "8303, {new_deal.id}", // this is the hermes deal id
        "8d3fb70224a8f613a6cc7416e121996606868929": null,
        "90e1589738e5dc4ecec002c6f8424348f0c17143": null,
        "9d1836a71ef185ab2103eae0c902b4f4af6e1b03": null,
        "a8fb293ba4cfedc80a733b3411620e00921ba7d8": null,
        "abec03eee90b12d8b5658b7f988126e254a4bd79": null,
        "b0d672e094f9057c86f31a3383bedbb163ce718f": null,
        "c87e16b3aa96fb56c6d4bacf824d9a4fae2611d9": null,
        "f0a2dc95b937e2ec8ad860f0dd9cbfe4a338f818": null
    },
    "event": "updated.deal"
}

check that the deal id for the newly created meeting is now 5712
check that the deal you created has been deleted

Testing Organisation

Create a new Company 'A' in hermes admin
Create a new Contact in hermes admin and set the company.id to A - /contact/create
Create a new Deal in hermes admin and set the company.id to A - /deal/create
post to /pipedrive/callback/

{
  "v": 1,
  "matches_filters": {
    "current": []
  },
  "meta": {
    "action": "updated",
    "object": "organization"
  },
  "current": {
  "id": 6998,
  "name": "new Tutor Company",
  "label": null,
  "address": null,
  "add_time": "2024-02-23 14:55:34",
  "cc_email": "tutorcruncher@pipedrivemail.com",
  "obj_type": "organization",
  "owner_id": 15761943,
  "edit_name": true,
  "label_ids": [],
  "company_id": 11324733,
  "first_char": "a",
  "owner_name": "Sam",
  "picture_id": null,
  "visible_to": "3",
  "active_flag": true,
  "category_id": null,
  "delete_time": null,
  "files_count": 0,
  "notes_count": 1,
  "update_time": "2024-05-23 15:33:09",
  "country_code": null,
  "people_count": 1,
  "address_route": null,
  "address_country": "GB",
  "followers_count": 1,
  "won_deals_count": 0,
  "activities_count": 0,
  "address_locality": null,
  "last_activity_id": null,
  "lost_deals_count": 0,
  "next_activity_id": null,
  "open_deals_count": 1,
  "address_subpremise": null,
  "closed_deals_count": 0,
  "last_activity_date": null,
  "next_activity_date": null,
  "next_activity_time": null,
  "address_postal_code": null,
  "address_sublocality": null,
  "email_messages_count": 214,
  "address_street_number": null,
  "done_activities_count": 0,
  "related_won_deals_count": 0,
  "undone_activities_count": 0,
  "related_lost_deals_count": 0,
  "related_open_deals_count": 0,
  "address_formatted_address": null,
  "address_admin_area_level_1": null,
  "address_admin_area_level_2": null,
  "related_closed_deals_count": 0,
  "067f5298c65be333811f54203503839b9d9341ee": null,
  "0dce4a9cac0177a95c0233756b8cd045bf7e28f2": null,
  "1e09f790cd937cfc1c34f8a8b664a8b0293253c4": null,
  "385e5989540972c85c9659a62a8ab82fd8c6c5fa": null,
  "45f62b0fc120d201ea02fdfa5e7282273add2f20": null,
  "4be5bf6e60e2a01e2653532e872cd15b5308da23": null,
  "57170eb130b8fa45925381623c86011e4e598e21": "pending_email_conf",
  "5ce5a41297410f570d97d78341aa2fbcf5801012": 3,
  "70527310be44839c869854b055788a69ecbbab66": "0",
  "770b2fee9c89906b60a74057719509e087342ae9": "http://bulldog-manatee-nbrd.squarespace.com",
  "7c18fff1ec6b2ddf7e22a4571d24a511079d2ec8": null,
  "7f8959760703808f36b3795c15310566b74f5134": "48553, {A.id}", // this is the hermes id
  "8c5b9aa3af06e36d7136bddf32d55e04bd9508b5": null,
  "bdef8d12d2f2af6a61e907b6296e410fdfbef9e3": null,
  "d30bf32a173cdfa780901d5eeb92a8f2d1ccd980": null,
  "d8b615ce8885544ed228feaae3ce9f28dcf04531": "https://secure.tutorcruncher.com/clients/2755259/",
  "dbe81f3fdf69ce3dfbfc7609caee68f1654c901a": "just starting out",
  "f6a981f2aa1199ecbb7fda5aa39a2d9ad672234b": null
},
  "previous": {
  "id": 6998,
  "name": "Tutor Company",
  "label": null,
  "address": null,
  "add_time": "2024-02-23 14:55:34",
  "cc_email": "tutorcruncher@pipedrivemail.com",
  "obj_type": "organization",
  "owner_id": 15761943,
  "edit_name": true,
  "label_ids": [],
  "company_id": 11324733,
  "first_char": "a",
  "owner_name": "Sam",
  "picture_id": null,
  "visible_to": "3",
  "active_flag": true,
  "category_id": null,
  "delete_time": null,
  "files_count": 0,
  "notes_count": 1,
  "update_time": "2024-05-23 15:33:09",
  "country_code": null,
  "people_count": 1,
  "address_route": null,
  "address_country": "GB",
  "followers_count": 1,
  "won_deals_count": 0,
  "activities_count": 0,
  "address_locality": null,
  "last_activity_id": null,
  "lost_deals_count": 0,
  "next_activity_id": null,
  "open_deals_count": 1,
  "address_subpremise": null,
  "closed_deals_count": 0,
  "last_activity_date": null,
  "next_activity_date": null,
  "next_activity_time": null,
  "address_postal_code": null,
  "address_sublocality": null,
  "email_messages_count": 214,
  "address_street_number": null,
  "done_activities_count": 0,
  "related_won_deals_count": 0,
  "undone_activities_count": 0,
  "related_lost_deals_count": 0,
  "related_open_deals_count": 0,
  "address_formatted_address": null,
  "address_admin_area_level_1": null,
  "address_admin_area_level_2": null,
  "related_closed_deals_count": 0,
  "067f5298c65be333811f54203503839b9d9341ee": null,
  "0dce4a9cac0177a95c0233756b8cd045bf7e28f2": null,
  "1e09f790cd937cfc1c34f8a8b664a8b0293253c4": null,
  "385e5989540972c85c9659a62a8ab82fd8c6c5fa": null,
  "45f62b0fc120d201ea02fdfa5e7282273add2f20": null,
  "4be5bf6e60e2a01e2653532e872cd15b5308da23": null,
  "57170eb130b8fa45925381623c86011e4e598e21": "pending_email_conf",
  "5ce5a41297410f570d97d78341aa2fbcf5801012": 3,
  "70527310be44839c869854b055788a69ecbbab66": "0",
  "770b2fee9c89906b60a74057719509e087342ae9": "http://bulldog-manatee-nbrd.squarespace.com",
  "7c18fff1ec6b2ddf7e22a4571d24a511079d2ec8": null,
  "7f8959760703808f36b3795c15310566b74f5134": "48553, {A.id}", // this is the hermes id
  "8c5b9aa3af06e36d7136bddf32d55e04bd9508b5": null,
  "bdef8d12d2f2af6a61e907b6296e410fdfbef9e3": null,
  "d30bf32a173cdfa780901d5eeb92a8f2d1ccd980": null,
  "d8b615ce8885544ed228feaae3ce9f28dcf04531": "https://secure.tutorcruncher.com/clients/2755259/",
  "dbe81f3fdf69ce3dfbfc7609caee68f1654c901a": "just starting out",
  "f6a981f2aa1199ecbb7fda5aa39a2d9ad672234b": null
},
  "event": "updated.organization"
}

test that the contact's company was updated to 6998
test that the deal's company was updated to 6998
test that company A was deleted

Test Contact

Create a new contact 'A' in hermes admin
create a new deal with the contact = A
create a new meeting with the contact = A
post to /pipedrive/callback/ with

{
  "v": 1,
  "matches_filters": {
    "current": []
  },
  "meta": {
    "action": "updated",
    "object": "person"
  },
  "current": {
  "id": 29664,
  "im": [
    {
      "value": "",
      "primary": true
    }
  ],
  "name": "Test Person",
  "email": [
    {
      "label": "work",
      "value": "test_person@example.com",
      "primary": true
    }
  ],
  "label": null,
  "notes": null,
  "phone": [
    {
      "label": "",
      "value": "61 666666666",
      "primary": true
    }
  ],
  "org_id": 7578,
  "add_time": "2024-05-13 06:40:18",
  "birthday": null,
  "cc_email": "tutorcruncher@pipedrivemail.com",
  "obj_type": "person",
  "org_name": "Tutor Company",
  "owner_id": 15761943,
  "job_title": null,
  "label_ids": [],
  "last_name": "Owner",
  "company_id": 11324733,
  "doi_status": 1,
  "first_char": "p",
  "first_name": "Tutor",
  "owner_name": "Sam",
  "picture_id": null,
  "visible_to": "3",
  "active_flag": true,
  "delete_time": null,
  "files_count": 0,
  "notes_count": 1,
  "update_time": "2024-05-23 15:42:03",
  "postal_address": null,
  "followers_count": 1,
  "picture_128_url": null,
  "won_deals_count": 0,
  "activities_count": 0,
  "last_activity_id": null,
  "lost_deals_count": 0,
  "marketing_status": "no_consent",
  "next_activity_id": null,
  "open_deals_count": 1,
  "closed_deals_count": 0,
  "last_activity_date": null,
  "next_activity_date": null,
  "next_activity_time": null,
  "email_messages_count": 3,
  "postal_address_route": null,
  "done_activities_count": 0,
  "postal_address_country": null,
  "last_incoming_mail_time": "2024-05-13 10:32:54",
  "last_outgoing_mail_time": "2024-05-23 15:41:56",
  "postal_address_locality": null,
  "related_won_deals_count": 0,
  "undone_activities_count": 0,
  "related_lost_deals_count": 0,
  "related_open_deals_count": 0,
  "postal_address_subpremise": null,
  "postal_address_postal_code": null,
  "postal_address_sublocality": null,
  "related_closed_deals_count": 0,
  "participant_open_deals_count": 0,
  "postal_address_street_number": null,
  "participant_closed_deals_count": 0,
  "postal_address_formatted_address": null,
  "postal_address_admin_area_level_1": null,
  "postal_address_admin_area_level_2": null,
  "00230e1468e98ca253e39870d14256cfc8b77514": null,
  "5b1e493cfb81e5817c481b149b3b32891bdf3312": null,
  "8c2f326b6be255cd3d5cf4ee7385eaf544a47f1d": "89348",
  "c1c3c51f1a26713b786a5274e450e997fd0c813b": null,
  "c2cce53d5a10e5124154a4cabc0aaf4199e4d94d": null,
  "c52a0f158c49ae8a2c428ac4fdc5575554e3760c": null,
  "c608baee9f204e6a09e5c741dc8159e9295c1dbb": null,
  "ccefbc26a1f2c753e467440c7420e7342e761c44": null,
  "f11fc7f795d827ef2edb02e729599ca291b77311": null
}
,
  "previous": {
  "id": 29664,
  "im": [
    {
      "value": "",
      "primary": true
    }
  ],
  "name": "Test Person",
  "email": [
    {
      "label": "work",
      "value": "test_person@example.com",
      "primary": true
    }
  ],
  "label": null,
  "notes": null,
  "phone": [
    {
      "label": "",
      "value": "61 666666666",
      "primary": true
    }
  ],
  "org_id": 7578,
  "add_time": "2024-05-13 06:40:18",
  "birthday": null,
  "cc_email": "tutorcruncher@pipedrivemail.com",
  "obj_type": "person",
  "org_name": "Tutor Company",
  "owner_id": 15761943,
  "job_title": null,
  "label_ids": [],
  "last_name": "Owner",
  "company_id": 11324733,
  "doi_status": 1,
  "first_char": "p",
  "first_name": "Tutor",
  "owner_name": "Sam",
  "picture_id": null,
  "visible_to": "3",
  "active_flag": true,
  "delete_time": null,
  "files_count": 0,
  "notes_count": 1,
  "update_time": "2024-05-23 15:42:03",
  "postal_address": null,
  "followers_count": 1,
  "picture_128_url": null,
  "won_deals_count": 0,
  "activities_count": 0,
  "last_activity_id": null,
  "lost_deals_count": 0,
  "marketing_status": "no_consent",
  "next_activity_id": null,
  "open_deals_count": 1,
  "closed_deals_count": 0,
  "last_activity_date": null,
  "next_activity_date": null,
  "next_activity_time": null,
  "email_messages_count": 3,
  "postal_address_route": null,
  "done_activities_count": 0,
  "postal_address_country": null,
  "last_incoming_mail_time": "2024-05-13 10:32:54",
  "last_outgoing_mail_time": "2024-05-23 15:41:56",
  "postal_address_locality": null,
  "related_won_deals_count": 0,
  "undone_activities_count": 0,
  "related_lost_deals_count": 0,
  "related_open_deals_count": 0,
  "postal_address_subpremise": null,
  "postal_address_postal_code": null,
  "postal_address_sublocality": null,
  "related_closed_deals_count": 0,
  "participant_open_deals_count": 0,
  "postal_address_street_number": null,
  "participant_closed_deals_count": 0,
  "postal_address_formatted_address": null,
  "postal_address_admin_area_level_1": null,
  "postal_address_admin_area_level_2": null,
  "00230e1468e98ca253e39870d14256cfc8b77514": null,
  "5b1e493cfb81e5817c481b149b3b32891bdf3312": null,
  "8c2f326b6be255cd3d5cf4ee7385eaf544a47f1d": "89348, {new_person_id}", // hermes id
  "c1c3c51f1a26713b786a5274e450e997fd0c813b": null,
  "c2cce53d5a10e5124154a4cabc0aaf4199e4d94d": null,
  "c52a0f158c49ae8a2c428ac4fdc5575554e3760c": null,
  "c608baee9f204e6a09e5c741dc8159e9295c1dbb": null,
  "ccefbc26a1f2c753e467440c7420e7342e761c44": null,
  "f11fc7f795d827ef2edb02e729599ca291b77311": null
},
  "event": "updated.person"
}

codecov-commenter · 2024-05-23T14:30:09Z

Codecov Report

Attention: Patch coverage is 93.90244% with 5 lines in your changes missing coverage. Please review.

Project coverage is 93.86%. Comparing base (ed2cefa) to head (fd16a05).
Report is 11 commits behind head on main.

Files	Patch %	Lines
app/pipedrive/_schema.py	92.98%	1 Missing and 3 partials ⚠️
app/pipedrive/views.py	96.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #227      +/-   ##
==========================================
+ Coverage   93.76%   93.86%   +0.10%     
==========================================
  Files          29       29              
  Lines        1652     1712      +60     
  Branches      211      228      +17     
==========================================
+ Hits         1549     1607      +58     
  Misses         66       66              
- Partials       37       39       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DanJezeph15

While testing, the info was in current and I was getting errors because there was nothing in the previous section which is referenced by the new prepare_event_data function

app/pipedrive/views.py

DanJezeph15

lgtm!

HenryTraill

else lgtm

app/pipedrive/_schema.py

PrenSJ2 added 2 commits May 23, 2024 12:33

start, created handle_duplicate_hermes_ids

577fe32

tests added

763959a

PrenSJ2 self-assigned this May 23, 2024

PrenSJ2 added bug Something isn't working and removed bug Something isn't working labels May 23, 2024

PrenSJ2 assigned DanJezeph15 and unassigned PrenSJ2 May 23, 2024

DanJezeph15 assigned PrenSJ2 and unassigned DanJezeph15 May 24, 2024

DanJezeph15 reviewed May 24, 2024

View reviewed changes

app/pipedrive/views.py Outdated Show resolved Hide resolved

PrenSJ2 added 4 commits May 24, 2024 12:40

change prepare_event_data to check the hermes_id, pd_field_ids

f82a00e

improve prepare_event_data to handle both current and previous

199dea9

fix test

b038f30

fix test add enums

e4555af

PrenSJ2 assigned DanJezeph15 and unassigned PrenSJ2 May 24, 2024

DanJezeph15 approved these changes May 28, 2024

View reviewed changes

DanJezeph15 assigned HenryTraill and unassigned DanJezeph15 May 28, 2024

HenryTraill requested changes Jun 4, 2024

View reviewed changes

app/pipedrive/_schema.py Outdated Show resolved Hide resolved

HenryTraill approved these changes Jun 4, 2024

View reviewed changes

HenryTraill assigned PrenSJ2 and unassigned HenryTraill Jun 4, 2024

PrenSJ2 added 4 commits June 4, 2024 17:24

blank

678e6ef

henrys comment

1415b66

lint

89032da

fix tests

fd16a05

PrenSJ2 assigned HenryTraill and unassigned PrenSJ2 Jun 10, 2024

PrenSJ2 merged commit fb0b487 into main Jun 18, 2024
2 checks passed

PrenSJ2 deleted the handle-merged-pipedrive-org-hermes-ids branch June 18, 2024 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle Pipedrive merging of Org, Person or Deal #227

Handle Pipedrive merging of Org, Person or Deal #227

PrenSJ2 commented May 23, 2024 •

edited

Loading

codecov-commenter commented May 23, 2024 •

edited

Loading

DanJezeph15 left a comment

DanJezeph15 left a comment

HenryTraill left a comment

Handle Pipedrive merging of Org, Person or Deal #227

Handle Pipedrive merging of Org, Person or Deal #227

Conversation

PrenSJ2 commented May 23, 2024 • edited Loading

The Problem

Solution

Testing

Test Deal

Testing Organisation

Test Contact

codecov-commenter commented May 23, 2024 • edited Loading

Codecov Report

DanJezeph15 left a comment

Choose a reason for hiding this comment

DanJezeph15 left a comment

Choose a reason for hiding this comment

HenryTraill left a comment

Choose a reason for hiding this comment

PrenSJ2 commented May 23, 2024 •

edited

Loading

codecov-commenter commented May 23, 2024 •

edited

Loading