Create parent company extended fields #1602

jwalgran · 2022-01-25T21:52:48Z

Overview

Create parent company extended fields when a contributor includes a parent_company in their CSV or API submission. Use a trigram search of existing contributor names to try an link the submitted value to a known contributor.

Connects #1584

Demo

Testing Instructions

These instruction assume ./scripts/resetdb has been run.

Browse http://localhost:8081/api/docs/#!/facilities/facilities_create and authenticate with Token 1d18b962d6f976b0b7e8fcf9fcc39b56cf278051
Submit this JSON

{
  "country": "US",
  "name": "Azavea",
  "address": "990 Spring Garden Street, 5th Floor Philadelphia, PA 19123",
  "parent_company": "A non matching value"
}

Browse the facility and verify that the extended field value is displayed in the sidebar
In a separate browser session browse http://localhost:8081/admin/api/extendedfield/1/change/ , log in as c1@example.com and verify that the extended field created includes but the raw value and a name key set to the same value.
Log in as c3@example.com and contribute parent-company.csv
Fully process the list

./scripts/manage batch_process --list-id 16 --action parse
./scripts/manage batch_process --list-id 16 --action geocode
./scripts/manage batch_process --list-id 16 --action match

Browse the new facility and verify that the parent company was fuzzy matched to an existing contributor and that contributor is displayed in the sidebar.
In a the admin browser session browse http://localhost:8081/admin/api/extendedfield/2/ and verify that the extended field created includes the raw value and matched contributor details.

Checklist

fixup! commits have been squashed
CI passes after rebase
CHANGELOG.md updated with summary of features or fixes, following Keep a Changelog guidelines

jwalgran · 2022-01-26T23:38:13Z

src/django/api/models.py

@@ -85,6 +86,33 @@ def create_superuser(self, email, password, **extra_fields):
        return self._create_user(email, password, **extra_fields)


+class ContributorManager(models.Manager):
+    TRIGRAM_SIMILARY_THRESHOLD = 0.5


This was chosen via trial an error with development data. We may adjust this in the future if non enough matching occurs or an excessive number of mismatches need to be corrected.

TaiWilkin

This is working well and has a thorough test set. I tested a few different variants of parent company names and the results seemed reasonable in terms of matching threshold.

I left a few suggestions regarding variable names, but otherwise this is good to go.

TaiWilkin · 2022-01-27T14:03:05Z

src/django/api/models.py

+        False is less than True so we order_by boolean fields in descending
+        order
+        """
+        threhold = ContributorManager.TRIGRAM_SIMILARY_THRESHOLD


Can we rename this threshold?

Corrected in fixup 945925c

TaiWilkin · 2022-01-27T14:04:31Z

src/django/api/models.py

@@ -85,6 +86,33 @@ def create_superuser(self, email, password, **extra_fields):
        return self._create_user(email, password, **extra_fields)


+class ContributorManager(models.Manager):
+    TRIGRAM_SIMILARY_THRESHOLD = 0.5


Can we rename this variable to TRIGRAM_SIMILARITY_THRESHOLD?

Corrected in fixup 75954e9 and a44f6f5

TaiWilkin · 2022-01-27T15:03:54Z

src/django/api/models.py

+            .annotate(active_source_count=models.Count(
+                Q(source__is_active=True))) \
+            .annotate(
+                has_active_sources=ExpressionWrapper(


Nice two-step annotation

TaiWilkin · 2022-01-27T15:09:33Z

src/django/api/extended_fields.py

+            else:
+                field_value = {
+                    'raw_value': field_value,
+                    'name': field_value


I like using different values here (contributor_name and _id vs just name) to indicate whether we've matched it directly to a contributor or not. If we want to make the list items links in the future, we will be able to do that pretty easily.

To support fuzzy matching contributors by a submitted parent company extended field we add a custom manager to the Contributor model. We set the TRIGRAM_SIMILARY_THRESHOLD by trial and error testing with development data. In the real world multiple people register accounts with the same contributor name to we prioritize matches to a verified account, then to one under which data has been submitted.

Update the extended field handling to process `parent_company` values by using the custom `filter_by_name` manager method and setting the appropriate keys in the `ExtendedField.value` dictionary based on whether there is a match or not.

These were mistakenly left in the code after testing.

jwalgran · 2022-01-27T18:29:35Z

Thanks for the review.

jwalgran force-pushed the feature/jcw/parent-company branch 2 times, most recently from 3e85fef to ca9af42 Compare January 26, 2022 22:02

jwalgran commented Jan 26, 2022

View reviewed changes

jwalgran requested a review from TaiWilkin January 26, 2022 23:39

jwalgran assigned TaiWilkin Jan 26, 2022

jwalgran marked this pull request as ready for review January 26, 2022 23:39

TaiWilkin approved these changes Jan 27, 2022

View reviewed changes

TaiWilkin assigned jwalgran and unassigned TaiWilkin Jan 27, 2022

jwalgran force-pushed the feature/jcw/parent-company branch from 75954e9 to a44f6f5 Compare January 27, 2022 16:48

jwalgran added 4 commits January 27, 2022 09:49

Create parent company extended field

2bf18d4

Update the extended field handling to process `parent_company` values by using the custom `filter_by_name` manager method and setting the appropriate keys in the `ExtendedField.value` dictionary based on whether there is a match or not.

Remove debug print statements from migrations

2acf723

These were mistakenly left in the code after testing.

Update CHANGELOG

1f7a7b0

jwalgran force-pushed the feature/jcw/parent-company branch from a44f6f5 to 1f7a7b0 Compare January 27, 2022 16:49

jwalgran merged commit 4c30cf8 into develop Jan 27, 2022

jwalgran deleted the feature/jcw/parent-company branch January 27, 2022 18:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create parent company extended fields #1602

Create parent company extended fields #1602

jwalgran commented Jan 25, 2022 •

edited

jwalgran Jan 26, 2022

TaiWilkin left a comment

TaiWilkin Jan 27, 2022

jwalgran Jan 27, 2022

jwalgran Jan 27, 2022

TaiWilkin Jan 27, 2022

jwalgran Jan 27, 2022

jwalgran Jan 27, 2022 •

edited

TaiWilkin Jan 27, 2022

TaiWilkin Jan 27, 2022

jwalgran commented Jan 27, 2022

Create parent company extended fields #1602

Create parent company extended fields #1602

Conversation

jwalgran commented Jan 25, 2022 • edited

Overview

Demo

Testing Instructions

Checklist

Choose a reason for hiding this comment

TaiWilkin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwalgran Jan 27, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwalgran commented Jan 27, 2022

jwalgran commented Jan 25, 2022 •

edited

jwalgran Jan 27, 2022 •

edited