Normalize GBL1 and Aardvark records for OGM harvests

[ghukill]

Purpose and background context

This PR adds the ability to normalize GBL1 and Aardvark records for OGM harvests.

During this work there multiple threads, with multiple groups, that resulted in some interconnected changes. But the work can priarmily be broken down into two major chunks:

1- Expose OGM repository configurations to the SourceRecord class to use during normalization (commit)

Normalization for OGM records sometimes requires information (e.g. common, consistent name for the institution) about the repository that is not explicitly present in the harvested metadata record. The configuration YAML is the best choice for storing and retrieving this information.

2- Normalization of OGM GBL1 and Aardvark records (commit)

This work was very similar to FGDC and ISO19139, but overall much simpler. It was determined that ALL harvested OGM records could be either GBL1 or Aardvark, making these two source classes very OGM centric.

Furthermore, GBL1 often was just renaming and re-typing field values, while many Aardvark fields were direct copies of the original record.

The bulk of this work is setting up field methods, ensuring correct types, and adequate testing; the approach is nearly identical to FGDC and ISO19139 source classes and normalization.

NOTE: It was determined that OGM harvests will not need to write source and normalized metadata to our S3 CDN bucket. This change will be handled in the next PR.

How can a reviewer manually see the effects of these changes?

As alluded to above, the normal flow of the CLI or deployed ECS task will attempt to write OGM records to S3 CDN, which is not needed or ideal.

Therefore, the best way to test OGM harvesting is still directly in python. The following performs a full harvest of:

• edu.uchicago: GBL1
• edu.cornell: Aardvark, and example of explicit, alternate strategy of getting an external URL for each record

import logging
import time

from harvester.config import Config, configure_logger
from harvester.harvest.ogm import OGMHarvester

logger = logging.getLogger("harvester")
configure_logger(logger, True) # verbose
CONFIG = Config()

harvester = OGMHarvester(
    harvest_type="full",
    output_file="output/chicago_and_cornell_normalized.jsonl",
    include_repositories=[
        "edu.uchicago", # Aardvark
        "edu.cornell", # GBL1, example of explicit external URL strategy
    ],
    remove_local_repos=False
)
t0 = time.time()
results = harvester.harvest()
print(results)
print(f"total time: {time.time()-t0}")

Once harvest is complete, you can see the results of normalized files at output/chicago_and_cornell_normalized.jsonl.

Includes new or updated dependencies?

NO

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/GDT-109

Developer

• All new ENV is documented in README
• All new ENV has been added to staging and production environments
• All related Jira tickets are linked in commit message(s)
• Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

• The commit message is clear and follows our guidelines (not just this PR message)
• There are appropriate tests covering any new functionality
• The provided documentation is sufficient for understanding any new functionality introduced
• Any manual tests have been performed and verified
• New dependencies are appropriate or there were no changes

[ehanson8]

Looking good but some questions and changes requested!

[ghukill]

Looking good but some questions and changes requested!

Thanks for the review @ehanson8 - I've responded with additional commits!

[ehanson8]

Approved but commenting to defer final approval to @jonavellecuerdo

[jonavellecuerdo]

Hi, @ghukill ! Great work here. I appreciate the detail in doc strings for field mappings that required additional steps--every time I question came to mind, there was a comment for it. 💯

I just have one clarification question/request for change (at most, to add a docstring); see my second comment regarding Aardvark._gbl_resourceClass_sm. 🤔

[jonavellecuerdo]

How did you determine what key values to use? 🤔

[ghukill]

@jonavellecuerdo - not a precise science here. Knowing the controlled terms we want end up with, just slowly adding instances that we encounter.

[jonavellecuerdo]

To clarify, if the original Aardvark record (in self.parsed_data) does not have gbl_resourceClass_sm assigned, the function would return ["Other"], right? Is that the expected behavior? If so, could we provide an explanation as to what "Other" means in this scenario? 🤔

It seems unlikely that we'll encounter this issue, given that gbl_resourceClass_sm is required for the Aardvark schema. Technically, if this Aardvark record was missing a value AND it was run through the schema, it would fail and throw a validation error...

[ghukill]

Yes, that's correct: if the original Aardvark file didn't have it, it'd get set to Other.

As much a fan of docstrings as I am, I feel like it's okay without? I think in metadata-land it's fairly common to have a default value like Other, particularly when the field is required, but there is no data to support it.

And to your point, we really should never encounter this situation.

[jonavellecuerdo]

Sounds good! I'd be curious if any records that end up meeting this condition (i.e., returning ["Other"]) and maybe only then revisiting whether "Aardvark" records missing this required field should be kept or ignored (I have some difficulty semantically accepting ["Other"] as "no data" 😅). That said, it seems like a low chance that we'll encounter this case, so I think we can proceed forward!