DM-42220: Incorporate ModelPackage Butler datasets into Prompt Processing #125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The main thing we want to do with the datasetRef collections is test for membership, for which sets are ideal.
kfindeisen
force-pushed
the
tickets/DM-42220
branch
from
5c492ac
to
2243087
Compare
hsinfang
approved these changes
Feb 20, 2024
Previously standard collection names were represented partly by constants, and partly by methods. Making them all attributes/properties, with the same naming convention, provides a more uniform API.
A model is quite large (hundreds of MB), but because the same model is used regardless of data ID, it should be a one-time overhead for the pod. This commit also chains the dummy model to the test repo's DECam/defaults, something that should have been done when it was added.
Now that the model is loaded in prep_butler (in both unit and integration tests), we can build a graph for the complete pipeline without getting missing dataset errors.
kfindeisen
force-pushed
the
tickets/DM-42220
branch
from
2243087
to
aab641c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for machine learning models (specifically, the
pretrainedModelPackagedataset type) toprep_butler. Applicable models have already been added to/repo/embargoand our integration testing repos.