Map Aardvark.dct_spatial_sm to Locations@kind="Place Name"

[jonavellecuerdo]

Purpose and background context

Map Aardvark.dct_spatial_sm to Locations@kind="Place Name" to support the use of place names
for location-based searching for GeoData.

Note: MITAardvark is the only source transformer that writes Subject@kind="Dublin Core; Spatial" (i.e., what Aardvark.dct_spatial_sm currently maps to), so the updated mapping is only applied to this transformer.

How can a reviewer manually see the effects of these changes?

1. Temporarily set AWS credentials for TimdexManagers for Dev1 in your terminal.

2. Transform gismit records

   • Run the following command from your local clone of transmogrifier:

     pipenv run transform -i s3://timdex-extract-dev-222053980223/gismit/gismit-2024-03-11-full-extracted-records-to-index.jsonl -o output/output_gismit.json -s gismit -v
     

   • View output/output_gismit.json. Lines 67-88 should read:
     

3. Transform gisogm records (a partial run, at least!)

   • Run the following command from your local clone of transmogrifier:

     pipenv run transform -i s3://timdex-extract-dev-222053980223/gisogm/gisogm-2024-03-01-full-extracted-records-to-index.jsonl -o output/output_gisogm.json -s gisogm -v
     

   • View output/output_gisogm.json. Lines 49-66 should read:
     

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

https://mitlibraries.atlassian.net/browse/GDT-217

Developer

• All new ENV is documented in README
• All new ENV has been added to staging and production environments
• All related Jira tickets are linked in commit message(s)
• Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

• The commit message is clear and follows our guidelines (not just this PR message)
• There are appropriate tests covering any new functionality
• The provided documentation is sufficient for understanding any new functionality introduced
• Any manual tests have been performed and verified
• New dependencies are appropriate or there were no changes

[ghukill]

The update looks good, and I think achieves what we need, but I might propose that we lean into the post-transform "hook" or "enrichment" (name TBD) pattern that was established in #141.

While I think it's agreed that we'll want to revist how to orchestrate these hooks, this second case of needing to do something very similar suggests there is a pattern that we can identify and abstract out.

Until then, I think we'd benefit from not locating this Subject --> Location data extraction/duplication in the MITAardvark transform only, lest we forget about it later and accidentally duplicate it's functionality. Moreover, if we lean into this pattern now and define it at the Transformer level, I think it provides more concrete use cases to consider when that refactor happens in earnest.

More concretely, where we currently have:

fields = self.create_dates_and_locations_from_publishers(fields)

I might propose we add another like:

# post transform hooks and enrichments
fields = self.create_dates_and_locations_from_publishers(fields)
fields = self.create_locations_from_spatial_subjects(fields)  # <--- new

The needle I think we want to thread is: making changes now that are solid and understandable, while understanding a future refactor may combine and extend this pattern that is emerging from these first two use cases.

[jonavellecuerdo]

@ghukill Thank you for clarifying! I undid the changes to the MITAardvark transformer and implemented the solution as a post-transform hook in the transformer module.

[ghukill]

General structure, looks good! Thanks for going with this pattern of locating these "post hook" work together.

I left a syntactical / stylistic comment for the filtering of subjects down to spatial ones. I find list comprehensions easier to scan than filter + lambda, but open to pushback!

[ghukill]

I think that I see the logic here, refining a list of potential subjects down using filter + a lambda, but perhaps this could be a simpler list comprehension?

# early return; handles edge case where subjects is value None and would return as such from .get()
if not fields.get("subjects"):
    return fields

# filters list, potentially to empty list which is okay
spatial_subjects = [
    subject
    for subject in fields.get("subjects", []) # defaults to empty list
    if subject.kind == "Dublin Core; Spatial"
    and subject.value is not None
]

[ehanson8]

Agree on list comprehension, that's what we've used throughout the app and I find it more readable as well

[ghukill]

Looks good!

[ehanson8]

Looks good!