DM-43188: Add a nightly chained collection to Prompt Processing output #136
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kherner
approved these changes
Mar 20, 2024
| self._count_datasets(central_butler, ["raw", "calexp"], f"{instname}/defaults"), | ||
| 0) | ||
|
|
||
| def test_export_outputs_retry(self): |
There was a problem hiding this comment.
Just verifying that is was intentional to eliminate the test_export_outputs_retry function entirely and move the new line into test_export_outputs. Nothing actually called the retry function anyway, right?
There was a problem hiding this comment.
@kherner, does that answer your question, or are there outstanding issues with the review?
This test became obsolete with the removal of raw export on DM-39188.
Some of the code for defining an output run has been factored into the chain definition.
The chain will make it easier for people to do real-time analysis of the pipeline outputs, since it depends only on the instrument and processing date. Each chain should be small enough (a few collections per pipeline per deployment) not to cause performance problems.
Currently, the two collections are identically named, but this will be important for forward-compatibility.
This run is not used yet (except as a mock for the preload metrics), but will be needed once we have preload tasks/pipelines.
In addition to setting up access to the buckets, Butler operations need access to the registry DB.
kfindeisen
force-pushed
the
tickets/DM-43188
branch
from
5943a7e to
ebd6ec4
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a daily chained collection to the central repo, and refactors some other collection-management code.