[BEAM-6859] align teardown with setup calls also for empty streaming … #8443

adude3141 · 2019-04-30T22:23:45Z

…batches

This PR aligns teardown with setup calls. Currently, for empty batches in streaming for DoFns setup is called, but teardown isn't. [1]

This PR removes the call to setup. An alternative implementation could be to add

diff --git a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
index 830778fe95..97a4c288d9 100644
--- a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
+++ b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkProcessContext.java
@@ -58,6 +58,7 @@ class SparkProcessContext<FnInputT, FnOutputT, OutputT> {
 
     // skip if partition is empty.
     if (!partition.hasNext()) {
+      DoFnInvokers.invokerFor(doFn).invokeTeardown();
       return new ArrayList<>();
     }

This was implemented before, but the call to teardown was dropped and afterwards the call to setup was moved on some refactoring so that this issue was accidentally introduced.

[1] https://issues.apache.org/jira/browse/BEAM-6859

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.