Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: do not close delegate rs in callback runnable #425

Merged

Conversation

olavloite
Copy link
Contributor

@olavloite olavloite commented Sep 14, 2020

AsyncResultSetImpl should not close its delegate ResultSet in the CallbackRunnable, but should leave that to the ProduceRowsCallable and instead only set the flag cursorReturnedDoneOrException. Otherwise, the main loop that fetches and produces rows could get stuck in an infinite loop.

@olavloite olavloite requested a review from thiagotnunes Sep 14, 2020
@google-cla google-cla bot added the cla: yes label Sep 14, 2020
@codecov
Copy link

@codecov codecov bot commented Sep 14, 2020

Codecov Report

Merging #425 into master will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #425   +/-   ##
=========================================
  Coverage     82.16%   82.16%           
  Complexity     2455     2455           
=========================================
  Files           136      136           
  Lines         13589    13589           
  Branches       1307     1307           
=========================================
  Hits          11166    11166           
  Misses         1895     1895           
  Partials        528      528           
Impacted Files Coverage Δ Complexity Δ
...a/com/google/cloud/spanner/AsyncResultSetImpl.java 91.43% <100.00%> (ø) 31.00 <0.00> (ø)
...m/google/cloud/spanner/spi/v1/GapicSpannerRpc.java 82.03% <0.00%> (ø) 81.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b312091...368bda6. Read the comment docs.

@thiagotnunes
Copy link
Contributor

@thiagotnunes thiagotnunes commented Sep 15, 2020

I would like to understand a bit more about this before approving. Could you point me to where the infinite loop would be?

@product-auto-label product-auto-label bot added the api: spanner label Sep 15, 2020
@olavloite
Copy link
Contributor Author

@olavloite olavloite commented Sep 15, 2020

I would like to understand a bit more about this before approving. Could you point me to where the infinite loop would be?

Yeah, I should have included that in the initial commit. The problem is as follows:

  1. The ProduceRowsCallable has two loops.
  2. The first loop iterates over the underlying ResultSet and fetches and buffers the rows, and initiates callbacks if one is needed. It will stop if the underlying ResultSet does not have any further rows, if the AsyncResultSet reaches a stoppable state (DONE, CANCELLED), or if an error occurs.
  3. The second loop initiates callbacks for the remaining rows in the buffer until the variable cursorReturnedDoneOrException is true. This will be set to true in the following cases:
    3.1. If the AsyncResultSet is cancelled.
    3.2. If an error occurred during a callback.
    3.3. If tryNext() returned DONE to the callback.
    3.4. It was however NOT set if the callback returned DONE before all rows had been consumed.
  4. The second loop will only finish if cursorReturnedDoneOrException set to true, which is where the infinite loop will occur.

This problem was introduced by fb26abe#diff-df5cefcf33d6f54a3dbfc8ac52ee50f3. Before that change, the loop would also finish if the state of the AsyncResultSet was DONE, but that was removed as it could cause the AsyncResultSet to miss a callback if it was cancelled. When an AsyncResultSet is cancelled, the contract is that the callback will be called one last time and that the tryNext() method will throw a SpannerException with code CANCELLED to ensure that the client application is notified and can handle it as an exception.

@thiagotnunes
Copy link
Contributor

@thiagotnunes thiagotnunes commented Sep 17, 2020

Thanks for the explanation @olavloite, I think I understand it a bit better now.

@thiagotnunes
Copy link
Contributor

@thiagotnunes thiagotnunes commented Sep 17, 2020

Maybe unrelated to this PR, but shouldn't we be updating the stop variable with the state.shouldStop inside a lock after this line as well:

@olavloite
Copy link
Contributor Author

@olavloite olavloite commented Sep 17, 2020

Maybe unrelated to this PR, but shouldn't we be updating the stop variable with the state.shouldStop inside a lock after this line as well:

No, I would say that that would be superfluous. It has always been updated almost directly before that:

  1. Here, and if one of the following if or while conditions are false, it falls right through to line 358.
  2. Here if it enters the while loop to wait for buffer capacity.

The worst that could happen is that one extra row is fetched and buffered. The startCallbackIfNecessary() will check whether it is allowed to do a callback (which is not equal to whether the further traversal and buffering results should continue, as the callback should also be called if the state is PAUSED, but it should continue to buffer results until the buffer is full).

@olavloite olavloite merged commit dce3ee7 into master Sep 17, 2020
18 checks passed
@olavloite olavloite deleted the async-result-dont-close-delegate-in-callback-runnable branch Sep 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants