Add facility/processing type taxonomy and look-up function #1601

caseycesari · 2022-01-25T18:41:44Z

Overview

The look-up function takes the input value, cleans it up by removing non-letter characters and extra spaces, and then attempts to find a match in the taxonomy using various methods.

Connects #1582

Demo

>>> get_facility_and_processing_type('shredding')
('PROCESSING_TYPE', 'EXACT', 'Raw Material Processing or Production', 'Shredding')
>>> get_facility_and_processing_type('pulp')
('PROCESSING_TYPE', 'FUZZY', 'Raw Material Processing or Production', 'Pulp making')
>>> get_facility_and_processing_type('wet painting')
('PROCESSING_TYPE', 'FUZZY', 'Printing, Product Dyeing and Laundering', 'Wet roller printing')
>>> get_facility_and_processing_type('wet printing')
('PROCESSING_TYPE', 'ALIAS', 'Printing, Product Dyeing and Laundering', 'Wet roller printing')
>>> get_facility_and_processing_type('assembly')
('PROCESSING_TYPE', 'EXACT', 'Final Product Assembly', 'Assembly')
>>> get_facility_and_processing_type('assemble')
('PROCESSING_TYPE', 'FUZZY', 'Final Product Assembly', 'Assembly')
>>> get_facility_and_processing_type('final assembly')
('FACILITY_TYPE', 'ALIAS', 'Final Product Assembly', 'Final Product Assembly')

Note

FUZZY_ALIAS_MATCH was not implemented because it ended up not being useful. All non-exact entries were either matched using the alias or fuzzy methods.

Testing Instructions

Run ./scripts/update to install the new pip dependency.
Run ./scripts/test to ensure the new tests pass.
Start a shell (./scripts/manage shell) and manually experiment with the matching function to see if it otherwise is working as intended.
Review the taxonomy and suggest any edits.

Checklist

fixup! commits have been squashed
CI passes after rebase
CHANGELOG.md updated with summary of features or fixes, following Keep a Changelog guidelines

caseycesari · 2022-01-31T15:29:30Z

src/django/api/facility_type_processing_type.py

@@ -0,0 +1,397 @@
+from api.matching import clean


I put this stuff in it's own file since there are so many items.

caseycesari · 2022-01-31T15:30:16Z

src/django/api/tests.py

@@ -600,6 +608,68 @@ def test_incorrect_country_code_raises_error(self, mock_get):
        )


+class FacilityAndProcessingTypeTest(TestCase):


I know the issue said to create tests using a loop, but I found it easier to incrementally add functionality to the look up function with individual tests. I can condense if need be.

jwalgran

Nice implementation. I was excited to see how well this works. I made one suggestion on how we return when there is not match but we could defer that until we actually have client code using it.

jwalgran · 2022-01-31T17:20:42Z

src/django/api/facility_type_processing_type.py

+
+            # Match must score 85 or higher to be considered usable.
+            if not matched_value or matched_value[1] < 85:
+                return None


We may want to consider returning (None, None, None, None) here so that the "shape" of the return values from the function is always the same, which may make the calling code a litter clearer.

>>> (a, b, c, d) = get_facility_and_processing_type('WILL NOT MATCH ANYTHING') Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: cannot unpack non-iterable NoneType object

I had that at first, but changed it to None because (None, None, None, None) evaluates to not None. But as you point out, I have some questions about the client code implementation that may require changing this.

Updated this. Thanks.

jwalgran · 2022-01-31T17:21:41Z

src/django/api/facility_type_processing_type.py

+FUZZY_MATCH = 'FUZZY'
+
+ALL_PROCESSING_TYPES = {
+    **OFFICE_PROCESSING_TYPES,


Nice use of ** here to compactly build these derived data structures.

The look-up function takes the input value, cleans it up by removing non-letter characters and extra spaces, and then attempts to find a match in the taxonomy using various methods. Refs #1582

caseycesari force-pushed the cpc/add-facility-production-type branch 5 times, most recently from 106e151 to e7c1f37 Compare January 28, 2022 16:03

caseycesari assigned jwalgran Jan 28, 2022

caseycesari requested a review from jwalgran January 28, 2022 16:12

caseycesari changed the title ~~WIP: Add facility type and production type taxonomy~~ Add facility/processing type taxonomy and look-up function Jan 28, 2022

caseycesari marked this pull request as ready for review January 28, 2022 16:13

caseycesari commented Jan 31, 2022

View reviewed changes

jwalgran approved these changes Jan 31, 2022

View reviewed changes

jwalgran assigned caseycesari and unassigned jwalgran Jan 31, 2022

caseycesari added 2 commits January 31, 2022 15:07

Add facility/processing type taxonomy and look-up function

cb7ff7b

The look-up function takes the input value, cleans it up by removing non-letter characters and extra spaces, and then attempts to find a match in the taxonomy using various methods. Refs #1582

Update CHANGELOG

29de8fa

caseycesari force-pushed the cpc/add-facility-production-type branch from e7c1f37 to 29de8fa Compare January 31, 2022 20:07

caseycesari merged commit 5593ecd into develop Jan 31, 2022

caseycesari deleted the cpc/add-facility-production-type branch January 31, 2022 20:16

emilyhu0106 mentioned this pull request Feb 11, 2022

Update final facility_type/processing_type taxonomy #1644

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add facility/processing type taxonomy and look-up function #1601

Add facility/processing type taxonomy and look-up function #1601

caseycesari commented Jan 25, 2022 •

edited

caseycesari Jan 31, 2022

caseycesari Jan 31, 2022

jwalgran left a comment

jwalgran Jan 31, 2022

caseycesari Jan 31, 2022

caseycesari Jan 31, 2022

jwalgran Jan 31, 2022

		@@ -600,6 +608,68 @@ def test_incorrect_country_code_raises_error(self, mock_get):
		)


		class FacilityAndProcessingTypeTest(TestCase):

Add facility/processing type taxonomy and look-up function #1601

Add facility/processing type taxonomy and look-up function #1601

Conversation

caseycesari commented Jan 25, 2022 • edited

Overview

Demo

Note

Testing Instructions

Checklist

caseycesari Jan 31, 2022

Choose a reason for hiding this comment

caseycesari Jan 31, 2022

Choose a reason for hiding this comment

jwalgran left a comment

Choose a reason for hiding this comment

jwalgran Jan 31, 2022

Choose a reason for hiding this comment

caseycesari Jan 31, 2022

Choose a reason for hiding this comment

caseycesari Jan 31, 2022

Choose a reason for hiding this comment

jwalgran Jan 31, 2022

Choose a reason for hiding this comment

caseycesari commented Jan 25, 2022 •

edited