[HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests. #1820

harishreedharan · 2014-08-07T00:19:37Z

No description provided.

…Sink in Flume Polling Suite

SparkQA · 2014-08-07T00:24:31Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18085/consoleFull

andrewor14 · 2014-08-07T00:30:01Z

core/src/main/scala/org/apache/spark/TestUtils.scala

Could you add a comment that describes the potential race condition that could happen? (i.e. right after you close the socket some other process can grab the port)

SparkQA · 2014-08-07T00:49:36Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18087/consoleFull

SparkQA · 2014-08-07T01:09:03Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18085/consoleFull

SparkQA · 2014-08-07T01:31:58Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18087/consoleFull

tdas · 2014-08-07T01:49:05Z

[info] - flume polling test multiple hosts *** FAILED ***
[info] 0 did not equal 1000 (FlumePollingStreamSuite.scala:184)

It might be that your two one-after-another calls to findFreePort will return the same port, and cause binding exception in one of them. So you have to use it in such a way where you findFreePort, use it, then call findFreePort another time.

SparkQA · 2014-08-07T02:34:46Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18095/consoleFull

SparkQA · 2014-08-07T03:19:55Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18095/consoleFull

harishreedharan · 2014-08-07T03:39:43Z

Actually, the Flume Polling Suite passed this time. The spark-submit tests seems to have failed.

…o. Leaving findFreePort as is, if other tests want to use it at some point.

SparkQA · 2014-08-07T18:24:32Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18136/consoleFull

vanzin · 2014-08-07T18:40:27Z

core/src/main/scala/org/apache/spark/TestUtils.scala

Not used anymore?

Yeah, it might be a good idea to remove this if we don't use it anymore, since this is somewhat hacky and is subject to a race condition. We can always add it back later if we find a need for it (my hopes is that we won't).

vanzin · 2014-08-07T18:41:12Z

Yay for letting netty do the work. Probably should update PR title / description.

andrewor14 · 2014-08-07T18:51:17Z

external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSink.scala

How about

serverOpt .map(_.getPort) .getOrElse(throw new RuntimeException(...))`

?

SparkQA · 2014-08-07T19:05:31Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18136/consoleFull

…e-ports

harishreedharan · 2014-08-07T19:12:17Z

The relevant tests passed. Looks like spark-submit failed again.

harishreedharan · 2014-08-07T19:12:59Z

[info] *** 3 TESTS FAILED ***
[error] Failed: Total 817, Failed 3, Errors 0, Passed 814, Ignored 6
[error] Failed tests:
[error] org.apache.spark.DriverSuite
[error] org.apache.spark.deploy.SparkSubmitSuite
error sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 2381 s, completed Aug 6, 2014 8:19:53 PM

SparkQA · 2014-08-07T19:14:44Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18138/consoleFull

SparkQA · 2014-08-07T20:05:51Z

QA results for PR 1820:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18138/consoleFull

andrewor14 · 2014-08-07T20:20:58Z

Yeah we really need to fix those 3 tests up soon.

SparkQA · 2014-08-07T23:39:36Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18156/consoleFull

SparkQA · 2014-08-08T00:23:21Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18156/consoleFull

harishreedharan · 2014-08-08T00:44:20Z

Last test failure was the DriverSuite again.

SparkQA · 2014-08-08T00:49:40Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18164/consoleFull

SparkQA · 2014-08-08T00:50:26Z

QA results for PR 1820:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18164/consoleFull

harishreedharan · 2014-08-08T00:51:54Z

Can a committer please re-run the tests. Failed due to
Failed to download pep8.py from "https://raw.githubusercontent.com/jcrocholl/pep8/1.5.7/pep8.py".

andrewor14 · 2014-08-08T02:15:53Z

@harishreedharan I think you can also say "test this please" to trigger the test.

test this please

andrewor14 · 2014-08-08T02:16:14Z

core/src/main/scala/org/apache/spark/TestUtils.scala

andrewor14 · 2014-08-08T02:16:53Z

1 minor comment and this LGTM - if the tests pass

harishreedharan · 2014-08-08T06:32:44Z

Jenkins, test this please

SparkQA · 2014-08-08T06:34:43Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18196/consoleFull

andrewor14 · 2014-08-08T20:56:52Z

test this please

harishreedharan · 2014-08-09T00:57:10Z

Jenkins, test this please

SparkQA · 2014-08-09T00:59:31Z

QA tests have started for PR 1820. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18235/consoleFull

SparkQA · 2014-08-09T01:48:44Z

QA results for PR 1820:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18235/consoleFull

harishreedharan · 2014-08-14T06:02:21Z

Can one of the committers please merge this? It would be nice to have this pulled in

tdas · 2014-08-18T02:50:04Z

Yeah, merging this. LGTM.

…to in Flume Polling Tests. Author: Hari Shreedharan <harishreedharan@gmail.com> Closes apache#1820 from harishreedharan/use-free-ports and squashes the following commits: b939067 [Hari Shreedharan] Remove unused import. 67856a8 [Hari Shreedharan] Remove findFreePort. 0ea51d1 [Hari Shreedharan] Make some changes to getPort to use map on the serverOpt. 1fb0283 [Hari Shreedharan] Merge branch 'master' of https://github.com/apache/spark into use-free-ports b351651 [Hari Shreedharan] Allow Netty to choose port, and query it to decide the port to bind to. Leaving findFreePort as is, if other tests want to use it at some point. e6c9620 [Hari Shreedharan] Making sure the second sink uses the correct port. 11c340d [Hari Shreedharan] Add info about race condition to scaladoc. e89d135 [Hari Shreedharan] Adding Scaladoc. 6013bb0 [Hari Shreedharan] [STREAMING] Find free ports to use before attempting to create Flume Sink in Flume Polling Suite

…to in Flume Polling Tests. Author: Hari Shreedharan <harishreedharan@gmail.com> Closes #1820 from harishreedharan/use-free-ports and squashes the following commits: b939067 [Hari Shreedharan] Remove unused import. 67856a8 [Hari Shreedharan] Remove findFreePort. 0ea51d1 [Hari Shreedharan] Make some changes to getPort to use map on the serverOpt. 1fb0283 [Hari Shreedharan] Merge branch 'master' of https://github.com/apache/spark into use-free-ports b351651 [Hari Shreedharan] Allow Netty to choose port, and query it to decide the port to bind to. Leaving findFreePort as is, if other tests want to use it at some point. e6c9620 [Hari Shreedharan] Making sure the second sink uses the correct port. 11c340d [Hari Shreedharan] Add info about race condition to scaladoc. e89d135 [Hari Shreedharan] Adding Scaladoc. 6013bb0 [Hari Shreedharan] [STREAMING] Find free ports to use before attempting to create Flume Sink in Flume Polling Suite

…to in Flume Polling Tests. Author: Hari Shreedharan <harishreedharan@gmail.com> Closes apache#1820 from harishreedharan/use-free-ports and squashes the following commits: b939067 [Hari Shreedharan] Remove unused import. 67856a8 [Hari Shreedharan] Remove findFreePort. 0ea51d1 [Hari Shreedharan] Make some changes to getPort to use map on the serverOpt. 1fb0283 [Hari Shreedharan] Merge branch 'master' of https://github.com/apache/spark into use-free-ports b351651 [Hari Shreedharan] Allow Netty to choose port, and query it to decide the port to bind to. Leaving findFreePort as is, if other tests want to use it at some point. e6c9620 [Hari Shreedharan] Making sure the second sink uses the correct port. 11c340d [Hari Shreedharan] Add info about race condition to scaladoc. e89d135 [Hari Shreedharan] Adding Scaladoc. 6013bb0 [Hari Shreedharan] [STREAMING] Find free ports to use before attempting to create Flume Sink in Flume Polling Suite

harishreedharan added 2 commits August 6, 2014 17:18

[STREAMING] Find free ports to use before attempting to create Flume …

6013bb0

…Sink in Flume Polling Suite

Adding Scaladoc.

e89d135

harishreedharan changed the title ~~[STREAMING] Find free ports to use before attempting to create Flume Sin...~~ [HOTFIX][STREAMING] Find free ports to use before attempting to create Flume Sin... Aug 7, 2014

andrewor14 reviewed Aug 7, 2014
View reviewed changes

Add info about race condition to scaladoc.

11c340d

Making sure the second sink uses the correct port.

e6c9620

Allow Netty to choose port, and query it to decide the port to bind t…

b351651

…o. Leaving findFreePort as is, if other tests want to use it at some point.

vanzin reviewed Aug 7, 2014
View reviewed changes

harishreedharan changed the title ~~[HOTFIX][STREAMING] Find free ports to use before attempting to create Flume Sin...~~ [HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests. Aug 7, 2014

andrewor14 reviewed Aug 7, 2014
View reviewed changes

Merge branch 'master' of https://github.com/apache/spark into use-fre…

1fb0283

…e-ports

Make some changes to getPort to use map on the serverOpt.

0ea51d1

Remove findFreePort.

67856a8

andrewor14 reviewed Aug 8, 2014
View reviewed changes

core/src/main/scala/org/apache/spark/TestUtils.scala Outdated

Copy link

Contributor

andrewor14 Aug 8, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not used

Remove unused import.

b939067

asfgit closed this in 95470a0 Aug 18, 2014

[HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests. #1820

[HOTFIX][STREAMING] Allow the JVM/Netty to decide which port to bind to in Flume Polling Tests. #1820

Uh oh!

Conversation

harishreedharan commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

andrewor14 Aug 7, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

tdas commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

harishreedharan commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

vanzin Aug 7, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 Aug 8, 2014

Choose a reason for hiding this comment

Uh oh!

vanzin commented Aug 7, 2014

Uh oh!

andrewor14 Aug 7, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

harishreedharan commented Aug 7, 2014

Uh oh!

harishreedharan commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

andrewor14 commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 7, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

harishreedharan commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

harishreedharan commented Aug 8, 2014

Uh oh!

andrewor14 commented Aug 8, 2014

Uh oh!

andrewor14 Aug 8, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Aug 8, 2014

Uh oh!

harishreedharan commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

andrewor14 commented Aug 8, 2014

Uh oh!

harishreedharan commented Aug 9, 2014

Uh oh!

SparkQA commented Aug 9, 2014

Uh oh!

SparkQA commented Aug 9, 2014