[WIP] Temporarily remove transactions & batching in staging. #583
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he problem is somehow linked to calling the process from the task scheduler.
jrochkind
added a commit
that referenced
this pull request
Jan 28, 2020
Accomplish the same thing more manually -- fetching in batches of 1000, but still yielding in a single stream. We think find_each race condition may have been responsible for #498, and this may solve it. @eddierubeiz 's debugging in #583 was crucial for figuring out what was going wrong and coming up with possible solution. Possibly we're still being too clever -- but I just didn't want to give up on avoiding fetching al objects or IDs into memory, to have an implementation that should be able to scale to orders of magnitude more assets no problem.
jrochkind
added a commit
that referenced
this pull request
Jan 28, 2020
Accomplish the same thing more manually -- fetching in batches of 1000, but still yielding in a single stream. We think find_each race condition may have been responsible for #498, and this may solve it. @eddierubeiz 's debugging in #583 was crucial for figuring out what was going wrong and coming up with possible solution. Possibly we're still being too clever -- but I just didn't want to give up on avoiding fetching al objects or IDs into memory, to have an implementation that should be able to scale to orders of magnitude more assets no problem.
|
This is no longer needed! @jrochkind fixed the problem decisively in #605. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please do not merge.
Temporarily removing transactions and batching in staging, to help diagnose the bug described in #498 .