[QQC-2355] Limit number of data rows to check for processing status at once #1218
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| wait_processing_max_seconds: int = _wait_processing_max_seconds, | ||
| sleep_interval=30): | ||
| """ Wait until all the specified data rows are processed""" | ||
| start_time = datetime.now() |
There was a problem hiding this comment.
Since the start_time is set to the beginning of this function, I think _poll_data_row_processing_status may not wait the full wait_processing_max_seconds per chunk
There was a problem hiding this comment.
My understanding was that the wait_processing_max_seconds was to be applied to the entire call for _wait_until_data_rows_are_processed, and not the individual chunks. This should preserve original behavior
There was a problem hiding this comment.
Ah gotcha makes sense
vbrodsky
reviewed
Aug 22, 2023
|
|
||
| max_data_rows_per_poll = 100_000 | ||
| if data_row_ids is not None: | ||
| for i in range(0, len(data_row_ids), max_data_rows_per_poll): |
There was a problem hiding this comment.
nice use of step for chunking
labelbox/schema/project.py
Outdated
| "Maximum wait time exceeded while waiting for data rows to be processed. Try creating a batch a bit later" | ||
| ) | ||
| "Maximum wait time exceeded while waiting for data rows to be processed. " | ||
| "Try creating a batch a bit later") |
There was a problem hiding this comment.
is it supposed to be a multi-line string? does not work? I though smth like
Suggested change
| "Try creating a batch a bit later") | |
| raise ProcessingWaitTimeout("""Maximum wait time exceeded while waiting for data rows to be processed. | |
| Try creating a batch a bit later""") |
kkim-labelbox
approved these changes
Aug 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a limit of 100k data rows that can be checked at once for processing status. Split this call to the backend into chunks of 100k