propagate and handle remote file resolution errors #741
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #480.
Fixes #668.
Fixes #728.
Fixes #738.
Ensure that we propagate any remote file resolution errors
during pagination.
Previously pagination was done in a goroutine that just sent
back a channel of files. However, this design meant that
if errors were encountered during the resolution, the channel
would close after just
and then close the channel.
Meanwhile the problem then bubbled up to the resolver
that determined which files exist and which don't
thereby triggering deletion operations claiming the remote
files didn't exist say when we Google Drive encountered a 500 or a 504.
This CL fixes these problems by ensuring that we return a paginationPair
which contains two channels:
a) a channel for encountered errors
b) a channel for the actual files as was before.
This then translates to consumers of remote file pagination have to
multiplex on both channels with a select loop and watching if any
errors came through and then propagate that along to the callers
to avoid making a simple bad situation, a total disaster.
To demonstrate the problems with the old and new behavior, let's make a
simulated remote error after resolving a few files, the typical would
be a 500 or 504 from Google Drive.
As you'll see, the old behavior would report a new for a total
addition/deletion which is wrong/faulty behavior causing duplicates
for additions, and spurious additions on remote/local, but also
worse causing deletion of files.
In the new behavior, those problems are trivially handled which is
the correct behavior.