Skip to content

Commit

Permalink
Added code examples for java and python
Browse files Browse the repository at this point in the history
  • Loading branch information
Ilya Ganelin committed Jan 8, 2015
1 parent 1fd59b2 commit 33b5a2d
Showing 1 changed file with 20 additions and 0 deletions.
20 changes: 20 additions & 0 deletions docs/programming-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -1318,6 +1318,8 @@ of that each task's update may be applied more than once if tasks or job stages

In addition, accumulators do not maintain lineage for the operations that use them. Consequently, accumulators updates are not guaranteed to be executed when made within a lazy transformation like `map()`. Unless something has triggered the evaluation of the lazy transformation that updates the value of the accumlator, subsequent operations will not themselves trigger that evaluation and the value of the accumulator will remain unchanged. The below code fragment demonstrates this issue:

<div class="codetabs">

<div data-lang="scala" markdown="1">
{% highlight scala %}
val acc = sc.accumulator(0)
Expand All @@ -1326,6 +1328,24 @@ In addition, accumulators do not maintain lineage for the operations that use th
{% endhighlight %}
</div>

<div data-lang="java" markdown="1">
{% highlight java %}
Accumulator<Integer> accum = sc.accumulator(0);
data.map(x -> accum.add(x); f(x););
// Here, acc is still 0 because no actions have cause the `map` to be computed.
{% endhighlight %}
</div>

<div data-lang="python" markdown="1">
{% highlight python %}
accum = sc.accumulator(0)
data.map(lambda x => acc.add(x); f(x))
# Here, acc is still 0 because no actions have cause the `map` to be computed.
{% endhighlight %}
</div>

</div>

# Deploying to a Cluster

The [application submission guide](submitting-applications.html) describes how to submit applications to a cluster.
Expand Down

0 comments on commit 33b5a2d

Please sign in to comment.