Skip to content
This repository has been archived by the owner on Nov 23, 2023. It is now read-only.

feat: Dynamic batch job array size #278

Merged
merged 5 commits into from
Feb 25, 2021
Merged

Conversation

l0b0
Copy link
Contributor

@l0b0 l0b0 commented Feb 18, 2021

Closes #269

Challenges:

@l0b0 l0b0 force-pushed the dynamic-batch-job-array-size branch 6 times, most recently from 3b1ae3a to c17dcf0 Compare February 21, 2021 22:17
@l0b0 l0b0 changed the title Dynamic batch job array size feat: Dynamic batch job array size Feb 22, 2021
This means we don't need to treat non-existing assets as "normal" within
check_files_checksums.

Also remove write access to the assets table from the content iterator -
it doesn't need it.
By randomly using one or two assets we test both branches *eventually,*
and since this is a particularly expensive test the tradeoff between
learning about any issues early and a faster run is probably worth it at
least in the short term.

Had to make `S3Object` a proper `ContextManager` subclass after working
around a `mypy` limitation related to grouping brackets
<python/mypy#10109 (comment)>.
@l0b0 l0b0 force-pushed the dynamic-batch-job-array-size branch from c17dcf0 to 264a331 Compare February 23, 2021 22:10
@l0b0 l0b0 marked this pull request as ready for review February 23, 2021 22:16
@l0b0 l0b0 added the automerge label Feb 25, 2021
@kodiakhq kodiakhq bot merged commit fc776fe into master Feb 25, 2021
@kodiakhq kodiakhq bot deleted the dynamic-batch-job-array-size branch February 25, 2021 01:47
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

Successfully merging this pull request may close these issues.

Implement dynamic batch job array size
2 participants