TIMX 406 - add provenance data to transformed records

[ghukill]

Purpose and background context

This PR adds the field timdex_provenance to the TIMDEX data model. As outlined in the original engineering plan, this addition to TIMDEX records has multiple uses:

1. providing quick, human readable information about the record's origins (e.g. run_date)
2. embed enough information in the indexed Opensearch record to efficiently find the record in the parquet dataset

As outlined in a recent timdex-dataset-api PR, we need to know a few things to quickly retrieve a record from the dataset: [run_date, run_type, timdex_record_id, run_record_offset] where timdex_record_id is actually kind of optional, but helpful for confirmation. This new field contains this information, which is known briefly during write to the dataset in Transmogrifier.

How can a reviewer manually see the effects of these changes?

Run a couple of transformations to establish a toy dataset at /tmp/dataset-provenance:

pipenv run transform \
 -s libguides \
 -i tests/fixtures/dataset/libguides-2024-06-03-full-extracted-records-to-index.xml \
 -o /tmp/dataset-provenance \
 --run-id run-abc-123

 pipenv run transform \
 -s libguides \
 -i tests/fixtures/dataset/libguides-2025-01-09-full-extracted-records-to-index.xml \
 -o /tmp/dataset-provenance \
 --run-id run-def-4456

Load the dataset:

import json
from timdex_dataset_api import  TIMDEXDataset

td = TIMDEXDataset('/tmp/dataset-provenance')
td.load()

Look at a couple of timdex_provenance sections from run run-abc-123 (3 total records):

for record in td.read_transformed_records_iter(run_id='run-abc-123'):
    print(record['timdex_provenance'])
"""
{'source': 'libguides', 'run_date': '2024-06-03', 'run_id': 'run-abc-123', 'run_record_offset': 0}
{'source': 'libguides', 'run_date': '2024-06-03', 'run_id': 'run-abc-123', 'run_record_offset': 2}
{'source': 'libguides', 'run_date': '2024-06-03', 'run_id': 'run-abc-123', 'run_record_offset': 3}
"""

Note the absence of run_record_offset=1. The method read_transformed_records_iter() only returns records with a transformed record, where we're expecting to see this provenance object, so it didn't show up.

But run_record_offset=1 actually does exist in the dataset, just for a record that was action="delete":

td.read_dataframe(columns=['timdex_record_id','action','run_record_offset'], run_id='run-abc-123')
"""
Out[14]: 
          timdex_record_id  action  run_record_offset
0  libguides:guides-175846   index                  0
1  libguides:guides-175849  delete                  1              <-------------
2  libguides:guides-175853   index                  2
3  libguides:guides-175855   index                  3
"""

This kind of a nicely demonstrates how the offset is calculated relative to the source record encountered during iteration in Transmogrifier. If records are skipped, they "consume" an offset number, and it keeps incrementing. In this way, the offset number is wholly artibrary, but by being present in the dataset and the provenance object, it provides a link.

Observe the way run_record_offset is written and incremented per run:

# get dataframe of all records in dataset
df = td.read_dataframe()

# get counts of column run_record_offset
# note that offsets 0-3 are repeated twice, this is because the first run only had
# 4 records written, and so offsets 0-3 were repeated for BOTH runs
df.run_record_offset.value_counts()
"""
Out[11]: 
run_record_offset
0      2   <-----
2      2   <-----
3      2   <-----
1      2   <-----
665    1
      ..
338    1
339    1
340    1
341    1
999    1
Name: count, Length: 1001, dtype: int64
"""

Includes new or updated dependencies?

YES

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/TIMX-406

Developer

• All new ENV is documented in README
• All new ENV has been added to staging and production environments
• All related Jira tickets are linked in commit message(s)
• Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

• The commit message is clear and follows our guidelines (not just this PR message)
• There are appropriate tests covering any new functionality
• The provided documentation is sufficient for understanding any new functionality introduced
• Any manual tests have been performed and verified
• New dependencies are appropriate or there were no changes

[ghukill]

I had considered locating this in the Transformer.transform() method, where the TimdexRecord instance is built, but opted not to for two reasons:

1. the transform method is still v1 and v2 compatible, whereas this would introduce feature flagging
2. the better of the two reasons, I think it's more associated with the orchestration of the run, the __next__ iterator, etc., and makes sense in this part of the code vs the transform method which feels primarily focused on the metadata fields

[ghukill]

Not quite - the model is updated to allow it (see this related response), but only the v2 feature flagged method _etl_v2_next_iter_method() will actually add it to the record.

[ehanson8]

Looks good!

[jonavellecuerdo]

@ghukill Looks good to me! Just one clarification question.

[jonavellecuerdo]

Why is this field optional? 🤔 In what cases is this field set to None?

[ghukill]

Bad answer: if it were a "v1" record, it wouldn't have it. So this simplifies backwards cmpatibility.

Better answer: it allows for creating the TimdexRecord incrementally. You don't have to initialize a record with it set, which allows for setting that property/field more dynamically as the object moves through it's init and handling. Even better, we somehow validate required fields during serialization. But doesn't feel that critical.