DM-36999: Complete integration test at USDF #38
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This appears to be a limit of the boto3 API; see #35 for discussion.
The Contents field does not exist if no files are found, so the old code would crash with a KeyError.
The topic is still hard-coded into the file, until we can decide how we want topics to be organized.
kfindeisen
force-pushed
the
tickets/DM-36999
branch
from
ad32cb8 to
9b4a6e7
Compare
hsinfang
reviewed
Nov 17, 2022
hsinfang
reviewed
Nov 17, 2022
hsinfang
reviewed
Nov 17, 2022
| _log.info(f"No raw files found for {instrument}, generating dummy files instead.") | ||
| upload_from_random(producer, instrument, dest_bucket, n_groups, new_group_base) | ||
| finally: | ||
| producer.flush(30.0) |
There was a problem hiding this comment.
I see the warning but I'm not sure if it matters to flush and wait here. Don't they get sent out still?
There was a problem hiding this comment.
Yes, the messages get sent, but such warnings usually mean that a class finalizer is doing cleanup that it can't be trusted to do. Also, the correctness of not flushing depends on the lower-level code not using callbacks, and there's no need for main to assume that.
As for waiting, the delay when I tested it was too small to measure.
hsinfang
approved these changes
Nov 17, 2022
The test assertion for raw export was originally written in terms of avoiding the standard behavior, but there's no reason not to test for the behavior we want instead, and it's easier to read.
This bug is currently caused by repetition of input exposure IDs, but it may also happen in the future as a result of retries, especially late-stage ones. Solution is to have the export code check for and only export the latest run, though this comes at the cost of not being able to retry previously failed exports.
We're not currently using callbacks, but Producer emits a warning if we don't flush anyway.
kfindeisen
force-pushed
the
tickets/DM-36999
branch
from
9b4a6e7 to
4bbce1b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains a number of miscellaneous fixes that prevented an end-to-end run on USDF. Many of these are preexisting bugs that were never exercised on GCP.