TIMX 489 - remove parquet feature flags #249
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Why these changes are being introduced: As of pipenv 2025.0.1 the use of `pipenv check` would throw an error, indicating that the library `safety` was not installed. It worked to run `pipenv check --auto-install` which would temporarily install `safety`, but this was not ideal for multiple reasons. First, we anticipate potentially moving away from `pipenv`. Second, it appears that `safety` is moving to a pay / subscription model. Third, it remains a little obfuscated what `pipenv check` is actually doing. As this new situation affects all builds in Github Actions CI, we need a way to scan for vulnerabilities that ideally is not a massive overhaul of our vulnerability scanning approach. How this addresses that need: `pip-audit` is a nice standalone, open-source library that performs very similar work to `safety`. This commit replaces `pipenv check` (which was `safety` under the hood) with `pip-audit`. Side effects of this change: * Builds will be successful in Github Actions Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/IN-1240
Why these changes are being introduced: At the onset of the parquet dataset work, feature flags were used to support both pre-parquet and parquet code pathways. With the parquet dataset working and fully utilized now, we can remove the feature flags. This project required some additional tweaks when removing the flags, namely updating tests to be fully parquet compliant. How this addresses that need: * Remove all feature flag code branching * Update Transformer class to support missing 'source_file', if testing environment detected, to remain backwards compatible with many tests Side effects of this change: * Original "v1" support is ended. Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TIMX-489
ghukill
force-pushed
the
TIMX-489-remove-parquet-feature-flags
branch
from
91766d6 to
aa1e78b
Compare
jonavellecuerdo
requested changes
May 7, 2025
jonavellecuerdo
approved these changes
May 7, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose and background context
This PR removes feature flags that were used during development work moving to a parquet dataset for ETL.
The removal of feature flag code branching was fairly minimal, but some followup work was required for the testing suite.
Many, if not most, of the unit tests are exercising transformation logic which is the core of Transmogrifier. There was a pattern throughout where a
Transformerwas instantiated and then a single transformed record was asserted, psuedo-code roughly:Now, the
next()of aTransformerinstance returns a TDADatasetRecordnot aTIMDEXRecord. The tests needed to be updated to follow this pattern:The changes to the testing suite were primarily of this nature. Ignoring the changes in the
/testsfolder will help review the actual feature flag removal changes.How can a reviewer manually see the effects of these changes?
The normal
make test+make lint.And, setting AWS Dev
TimdexManagerscredentials and running the following:Includes new or updated dependencies?
NO
Changes expectations for external applications?
YES: Transmogrifier no longer supports the "v1" JSON file output.
What are the relevant tickets?
Developer
Code Reviewer(s)