Skip to content

Commit

Permalink
[SPARK-19499][SS] Add more notes in the comments of Sink.addBatch()
Browse files Browse the repository at this point in the history
## What changes were proposed in this pull request?

addBatch method in Sink trait is supposed to be a synchronous method to coordinate with the fault-tolerance design in StreamingExecution (being different with the compute() method in DStream)

We need to add more notes in the comments of this method to remind the developers

## How was this patch tested?

existing tests

Author: CodingCat <zhunansjtu@gmail.com>

Closes #16840 from CodingCat/SPARK-19499.
  • Loading branch information
CodingCat authored and zsxwing committed Feb 8, 2017
1 parent aeb8034 commit d4cd975
Showing 1 changed file with 4 additions and 1 deletion.
Expand Up @@ -31,8 +31,11 @@ trait Sink {
* this method is called more than once with the same batchId (which will happen in the case of
* failures), then `data` should only be added once.
*
* Note: You cannot apply any operators on `data` except consuming it (e.g., `collect/foreach`).
* Note 1: You cannot apply any operators on `data` except consuming it (e.g., `collect/foreach`).
* Otherwise, you may get a wrong result.
*
* Note 2: The method is supposed to be executed synchronously, i.e. the method should only return
* after data is consumed by sink successfully.
*/
def addBatch(batchId: Long, data: DataFrame): Unit
}

0 comments on commit d4cd975

Please sign in to comment.