[SPARK-13165][STREAMING] Replace deprecated synchronizedBuffer in streaming #11067
Conversation
|
Test build #50706 has finished for PR 11067 at commit
|
|
jenkins, retest this please |
|
Test build #50708 has finished for PR 11067 at commit
|
|
Test build #50707 has finished for PR 11067 at commit
|
holdenk
changed the title
|
cc @srowen |
|
also @zsxwing |
| for (i <- 0 until output.size) { | ||
| assert(output(i) === expectedOutput(i)) | ||
| } | ||
| output.zipWithIndex.foreach{case (e, i) => assert(e == expectedOutput(i))} |
There was a problem hiding this comment.
Does this need spaces around braces to pass style checks? And === vs ==?
There was a problem hiding this comment.
I think in the latest scalatest == is compiled as ===
|
Minor questions; seems fine. It's actually the same simple changes in many files and mostly touches tests |
|
@srowen simplified those two places in the test code :) |
|
Test build #50836 has finished for PR 11067 at commit
|
|
@tdas want to take a look or since its mostly tests is that not needed? |
|
@tedyu I've cherry-picked in your DirectKafkaStreamSuite changes let me know if that looks ok to you. |
|
It's nice of you to put related changes under one PR. |
|
Test build #50905 has finished for PR 11067 at commit
|
|
Seems OK to me. This is all of the occurrences then? so we could add a scalastyle rule banning SynchronizedBuffer after this? |
|
Yah @tedyu has a follow up PR that adds the scalastyle rule ready to rebase once this gets merged in. |
|
merged to master |
…eaming Equivalent of apache#11067
Building with Scala 2.11 results in the warning trait SynchronizedBuffer in package mutable is deprecated: Synchronization via traits is deprecated as it is inherently unreliable. Consider java.util.concurrent.ConcurrentLinkedQueue as an alternative - we already use ConcurrentLinkedQueue elsewhere so lets replace it.
Some notes about how behaviour is different for reviewers:
The Seq from a SynchronizedBuffer that was implicitly converted would continue to receive updates - however when we do the same conversion explicitly on the ConcurrentLinkedQueue this isn't the case. Hence changing some of the (internal & test) APIs to pass an Iterable. toSeq is safe to use if there are no more updates.