Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed while connecting to sparklyr to port (8880) #801

Closed
Dragonfly99 opened this issue Jun 29, 2017 · 35 comments
Closed

Failed while connecting to sparklyr to port (8880) #801

Dragonfly99 opened this issue Jun 29, 2017 · 35 comments
Assignees
Milestone

Comments

@Dragonfly99
Copy link

Hi all. I'm trying to connect a new spark cluster on my local machine but I keep getting a connection error. Here are my specs:

OS Windows 10
R Version 3.4.0
Rstudio 1.0.143
devtools installed

spark_installed_versions()
spark hadoop dir
1 2.1.0 2.7 spark-2.1.0-bin-hadoop2.7

> spark_installed_versions()
  spark hadoop                       dir
1 2.1.0    2.7 spark-2.1.0-bin-hadoop2.7

> library(sparklyr)
> library(dplyr)
> sc <- spark_connect(master = "local", version = "2.1.0")

I get this error:

Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (7076): Gateway in port (8880) did not respond.
Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 7076
Log: C:\Users\Kerry\AppData\Local\Temp\RtmpohIScV\file2268c6d7434_spark.log

---- Output Log ----
The system cannot find the path specified.

---- Error Log ---

Apologies is this isn't the correct format. This is my first time submitting an error.

@javierluraschi
Copy link
Collaborator

@Dragonfly99 this is a common error I've been trying to fix/reproduce for a while. If you have a few hours to help the community, find me on gitter to chat about this, I'm sure we can get it resolved but it's unfortunately hard without someone willing to put the time to debug this with us. You can find me online here: https://gitter.im/rstudio/sparklyr - Thanks for reporting this!

@javierluraschi javierluraschi added this to the CRAN 0.6 milestone Jun 30, 2017
@javierluraschi javierluraschi self-assigned this Jun 30, 2017
@javierluraschi
Copy link
Collaborator

Could you also share sessionInfo() and retry with the latest version od sparklyr? The issue is probably not resolved but we've added checks to help troubleshoot this. You can install the latest version with:

devtools::install_github("rstudio/sparklyr")

Once installed, retry connecting, share error log and please also include the output of spark_context_config(sc).

@javierluraschi javierluraschi changed the title Failed while connecting to sparklyr to port (8880) for sessionid (7076) Win 10 Failed while connecting to sparklyr to port (8880) Jun 30, 2017
@javierluraschi
Copy link
Collaborator

@Dragonfly99 any chance you could also run java -version from the console and share the output?

@Dragonfly99
Copy link
Author

Here's what I got:

sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] sparklyr_0.5.6-9010 bindrcpp_0.2        caret_6.0-76        lattice_0.20-35    
 [5] dplyr_0.7.1         purrr_0.2.2         readr_1.1.1         tidyr_0.6.3        
 [9] tibble_1.3.3        ggplot2_2.2.1       tidyverse_1.1.1    

loaded via a namespace (and not attached):
 [1] httr_1.2.1         jsonlite_1.5       splines_3.4.0      foreach_1.4.3     
 [5] modelr_0.1.0       shiny_1.0.3        assertthat_0.2.0   stats4_3.4.0      
 [9] cellranger_1.1.0   yaml_2.1.14        backports_1.1.0    quantreg_5.33     
[13] glue_1.1.1         digest_0.6.12      rvest_0.3.2        minqa_1.2.4       
[17] colorspace_1.3-2   httpuv_1.3.3       htmltools_0.3.6    Matrix_1.2-9      
[21] plyr_1.8.4         psych_1.7.5        pkgconfig_2.0.1    devtools_1.13.2   
[25] broom_0.4.2        SparseM_1.77       haven_1.0.0        config_0.2        
[29] xtable_1.8-2       scales_0.4.1       lme4_1.1-13        MatrixModels_0.4-1
[33] git2r_0.18.0       mgcv_1.8-17        car_2.1-4          withr_1.0.2       
[37] nnet_7.3-12        lazyeval_0.2.0     pbkrtest_0.4-7     mnormt_1.5-5      
[41] mime_0.5           magrittr_1.5       readxl_1.0.0       memoise_1.1.0     
[45] nlme_3.1-131       MASS_7.3-47        forcats_0.2.0      xml2_1.1.1        
[49] foreign_0.8-67     tools_3.4.0        hms_0.3            stringr_1.2.0     
[53] munsell_0.4.3      compiler_3.4.0     rlang_0.1.1        grid_3.4.0        
[57] nloptr_1.0.4       iterators_1.0.8    rstudioapi_0.6     rappdirs_0.3.1    
[61] base64enc_0.1-3    labeling_0.3       gtable_0.2.0       ModelMetrics_1.1.0
[65] codetools_0.2-15   DBI_0.7            curl_2.7           reshape2_1.4.2    
[69] R6_2.2.0           lubridate_1.6.0    knitr_1.16         bindr_0.1         
[73] rprojroot_1.2      stringi_1.1.5      parallel_3.4.0     Rcpp_0.12.11      
[77] dbplyr_1.1.0      
>sc <- spark_connect(master = "local", version = "2.1.0")
Error in force(code) : 
  Failed while connecting to sparklyr to port (8880) for sessionid (8113): Gateway in port (8880) did not respond.
    Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
    Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 8113
    Log: C:\Users\Kerry\AppData\Local\Temp\RtmpmwvQFd\file1a746c36c62_spark.log


---- Output Log ----
The system cannot find the path specified.

---- Error Log ----
> spark_context_config(sc)
Error in spark_context(sc) : object 'sc' not found
> java -version
Error: object 'java' not found

@javierluraschi
Copy link
Collaborator

@Dragonfly99 Sorry, I mean system("java -version") since you are running this from R.

@javierluraschi
Copy link
Collaborator

Does this file C:\Users\Kerry\AppData\Local\Temp\RtmpmwvQFd\file1a746c36c62_spark.log happen to be empty if you open it yourself?

@Dragonfly99
Copy link
Author

That path reads: "The system cannot find the path specified."
That command you gave me gives me this:

> system("java -version")
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 1, 2017

Pushed a fix to increase logging, could you try the following and share the output? Really appreciated @Dragonfly99!

devtools::install_github("rstudio/sparklyr")

# restart r session
sessionInfo()
sc <- spark_connect(master = "local", version = "2.1.0", config = list(sparklyr.log.console = TRUE))

@Dragonfly99
Copy link
Author

No problem. I like helping and I want too want to play with big data. Here's the latest results:

> sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] sparklyr_0.5.6-9011 bindrcpp_0.2        caret_6.0-76        lattice_0.20-35    
 [5] dplyr_0.7.1         purrr_0.2.2         readr_1.1.1         tidyr_0.6.3        
 [9] tibble_1.3.3        ggplot2_2.2.1       tidyverse_1.1.1    

loaded via a namespace (and not attached):
 [1] httr_1.2.1         jsonlite_1.5       splines_3.4.0      foreach_1.4.3     
 [5] modelr_0.1.0       shiny_1.0.3        assertthat_0.2.0   stats4_3.4.0      
 [9] cellranger_1.1.0   yaml_2.1.14        backports_1.1.0    quantreg_5.33     
[13] glue_1.1.1         digest_0.6.12      rvest_0.3.2        minqa_1.2.4       
[17] colorspace_1.3-2   httpuv_1.3.3       htmltools_0.3.6    Matrix_1.2-9      
[21] plyr_1.8.4         psych_1.7.5        pkgconfig_2.0.1    devtools_1.13.2   
[25] broom_0.4.2        SparseM_1.77       haven_1.0.0        config_0.2        
[29] xtable_1.8-2       scales_0.4.1       lme4_1.1-13        MatrixModels_0.4-1
[33] git2r_0.18.0       mgcv_1.8-17        car_2.1-4          withr_1.0.2       
[37] nnet_7.3-12        lazyeval_0.2.0     pbkrtest_0.4-7     mnormt_1.5-5      
[41] mime_0.5           magrittr_1.5       readxl_1.0.0       memoise_1.1.0     
[45] nlme_3.1-131       MASS_7.3-47        forcats_0.2.0      xml2_1.1.1        
[49] foreign_0.8-67     tools_3.4.0        hms_0.3            stringr_1.2.0     
[53] munsell_0.4.3      compiler_3.4.0     rlang_0.1.1        grid_3.4.0        
[57] nloptr_1.0.4       iterators_1.0.8    rstudioapi_0.6     rappdirs_0.3.1    
[61] base64enc_0.1-3    labeling_0.3       gtable_0.2.0       ModelMetrics_1.1.0
[65] codetools_0.2-15   DBI_0.7            curl_2.7           reshape2_1.4.2    
[69] R6_2.2.0           lubridate_1.6.0    knitr_1.16         bindr_0.1         
[73] rprojroot_1.2      stringi_1.1.5      parallel_3.4.0     Rcpp_0.12.11      
[77] dbplyr_1.1.0      
> sc <- spark_connect(master = "local", version = "2.1.0", config = list(sparklyr.log.console = TRUE))
Error in force(code) : 
  Failed while connecting to sparklyr to port (8880) for sessionid (941): Gateway in port (8880) did not respond.
    Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
    Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 941
    Log: C:\Users\Kerry\AppData\Local\Temp\RtmpmwvQFd\file1a74735f7e7a_spark.log


---- Output Log ----


---- Error Log ----

@javierluraschi
Copy link
Collaborator

@Dragonfly99 do you see any files named log4j.spark.log* could you please share its contents?

@Dragonfly99
Copy link
Author

I don't have that file.

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 3, 2017

@Dragonfly99 thank you, could you try running spark-submit directly from the Windows Command Prompt as follows?

C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd --class sparklyr.Shell "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar" 8880 7076

and share the results?

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 3, 2017

@Dragonfly99 also, please run the SparkPi basic example as follows from the Windows Command Prompt:

C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit.cmd --class org.apache.spark.examples.SparkPi C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\examples\jars\spark-examples_2.11-2.1.0.jar 10

Thank you!

@Dragonfly99
Copy link
Author

I'm being told that spark-submit2.cmd and spark-submit.cmd are invalid names for some reason. The farthest I can cd into is C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\

@javierluraschi
Copy link
Collaborator

@Dragonfly99 Can you share a screenshot and validate that spark-submit2.cmd has execute permissions?

For instance, in my Windows 10 installation, execution was enabled by default and looks as followed:

screen shot 2017-07-03 at 2 22 25 pm

@Dragonfly99
Copy link
Author

Dragonfly99 commented Jul 3, 2017

All command files have execute permissions.
sparkpermissions

@javierluraschi
Copy link
Collaborator

@Dragonfly99 can you share the screenshot or text from the output you are getting in the windows terminal? Should look something like this:

screen shot 2017-07-03 at 3 52 14 pm

screen shot 2017-07-03 at 3 52 24 pm

Both, for

C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd --class sparklyr.Shell "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar" 8880 7076

and

C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit.cmd --class org.apache.spark.examples.SparkPi C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\examples\jars\spark-examples_2.11-2.1.0.jar 10

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 3, 2017

@Dragonfly99 Also, how much memory and disk space do you have available on this machine?

Could you try:

spark_install(version = "2.1.0", reset = TRUE)

and reconnect? There was a fix in spark_install that might fix the issue you are experiencing.

@javierluraschi javierluraschi removed this from the CRAN 0.6 milestone Jul 6, 2017
@Dragonfly99
Copy link
Author

This is what I get for that code

spark_install(version = "2.1.0", reset = TRUE)
Spark 2.1.0 for Hadoop 2.7 or later already installed.
> sc <- spark_connect(master = "local")
* Using Spark: 2.1.0
Error in force(code) : 
  Failed while connecting to sparklyr to port (8880) for sessionid (2443): Gateway in port (8880) did not respond.
    Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
    Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 2443
    Log: C:\Users\Kerry\AppData\Local\Temp\RtmpEJsY3q\file378068cb3752_spark.log


---- Output Log ----
The system cannot find the path specified.

---- Error Log ----

I'm going to to enter those other commands in the terminal and send you the screenshots.

@javierluraschi
Copy link
Collaborator

Thanks for entering those commands in the terminal, that would be much appreciated. I'm hoping we can get more info as to why "The system cannot find the path specified.", it's obviously there, so hopefully, we can see a clear permissions error or something else...

@Dragonfly99
Copy link
Author

Here's a sample of the commands I try in the command prompt and the different failures I'm getting. Spark-submit.cmd and spark-submit2.cmd give me the exact same things despite being there.

spark-submit-fail

@javierluraschi
Copy link
Collaborator

@Dragonfly99:

  1. Could you run dir on the command prompt under the C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\ path? As in:
cd C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\
dir

and share the results? This is to list the files since I saw this post explaining that sometimes this error triggers when the file downloads incorrectly and has a size of zero.

  1. From this post, the problem could also be that the path is too long. Could you copy the entire contents of C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\, create a new directory say under C:\spark and copy everything into it? You should have then the bin dir under C:\spark\bin. Then could you try from the terminal the following and share the output?
cd C:\spark\bin\
spark-submit.cmd
  1. From this post it could also be the case that JAVA_HOME is incorrectly set, could you run the following form R to share the status of your environment variables?
Sys.getenv()

Looks like we are getting close to resolving this, thanks for persevering on this issue, the sparklyr community will appreciate this!

@Dragonfly99
Copy link
Author

Here's a screenshot of the directory results

directoryresults

I have 2 questions so I make sure I do this right.

  1. for step 2, would everything go inside of C:\spark\bin or C:\spark?
  2. For step 4 would it be ok if I just share up into Java Home's results? This is the result for sys.getenv
 > Sys.getenv()
ALLUSERSPROFILE            C:\ProgramData
APPDATA                    C:\Users\Kerry\AppData\Roaming
CommonProgramFiles         C:\Program Files\Common Files
CommonProgramFiles(x86)    C:\Program Files (x86)\Common Files
CommonProgramW6432         C:\Program Files\Common Files
COMPUTERNAME               DADRAGONFLY
ComSpec                    C:\WINDOWS\system32\cmd.exe
DISPLAY                    :0
FP_NO_HOST_CHECK           NO
FPS_BROWSER_APP_PROFILE_STRING
                           Internet Explorer
FPS_BROWSER_USER_PROFILE_STRING
                           Default
GFORTRAN_STDERR_UNIT       -1
GFORTRAN_STDOUT_UNIT       -1
HADOOP_HOME                C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-1.6.0-bin-hadoop2.6\tmp\hadoop
HOME                       C:/Users/Kerry/Documents
HOMEDRIVE                  C:
HOMEPATH                   \Users\Kerry
JAVA_HOME                  C:\Program Files\Java\jre1.8.0_77

I'm happy to help. Here's hoping the solution is close.

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 7, 2017

@Dragonfly99 thank you! We know the problem is not zero-size now, for your questions:

  1. Everything goes into C:\spark.
  2. That points to the correct JAVA_HOME, you don't need to share more info. Thank you!

Lets see if it's a problem with long paths by copying everything into C:\spark and retrying.

@Dragonfly99
Copy link
Author

No luck.
sparkbin

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 7, 2017

@Dragonfly99 Could you right-click on the saprk-submit2.cmd file, then properties and share the info, something like:

screen shot 2017-07-07 at 10 54 28 am

screen shot 2017-07-07 at 10 54 46 am

screen shot 2017-07-07 at 10 54 55 am

There must be something off about that file that prevents us from executing this...

@Dragonfly99
Copy link
Author

Here they are:
spark-submit-security

spark-submit-details

@javierluraschi
Copy link
Collaborator

javierluraschi commented Jul 7, 2017

@Dragonfly99 Could you run from the command prompt:

java

and

"C:\Program Files\Java\jre1.8.0_77\bin\java"

and

echo %JAVA_HOME%

The spark-submit scripts mostly call java so I wonder if that's somehow incorrectly installed or missing.

@Dragonfly99
Copy link
Author

So I looked into my program files and I don't think I have that jre version. Here's a pick of what's there.

jre_files

Here's a pic of Java results:
javatest2

javatest

@javierluraschi
Copy link
Collaborator

@Dragonfly99 Nice! So that's the problem, this was much-much harder to troubleshoot than what I would expect, I'll make a fix in sparklyr to check java exists in a more robust way and throw an error when that's not the case.

So I think what you want to do is to change the JAVA_HOME environment variable to the correct path, here is a pointer, let me know if you hit any issues and I'll be more specific: https://www.java.com/en/download/help/path.xml

@Dragonfly99
Copy link
Author

Ok I changed the path to point to jre.1.8.0_131. What do I do next?

@javierluraschi
Copy link
Collaborator

@Dragonfly99 Restart rstudio, retry spark_connect(master = "local") and 🤞!

@javierluraschi javierluraschi added this to the CRAN 0.6 milestone Jul 7, 2017
@Dragonfly99
Copy link
Author

I think it worked! Here's the new results:

> library(sparklyr)
> spark_connect(master = "local")
* Using Spark: 2.1.0
$master
[1] "local[4]"

$method
[1] "shell"

$app_name
[1] "sparklyr"

$config
$config$sparklyr.cores.local
[1] 4

$config$spark.sql.shuffle.partitions.local
[1] 4

$config$spark.env.SPARK_LOCAL_IP.local
[1] "127.0.0.1"

$config$sparklyr.csv.embedded
[1] "^1.*"

$config$`sparklyr.shell.driver-class-path`
[1] ""

attr(,"config")
[1] "default"
attr(,"file")
[1] "C:\\Users\\Kerry\\Documents\\R\\win-library\\3.4\\sparklyr\\conf\\config-template.yml"

$spark_home
[1] "C:\\Users\\Kerry\\AppData\\Local\\rstudio\\spark\\Cache\\spark-2.1.0-bin-hadoop2.7"

$backend
A connection with                               
description "->localhost:49584"
class       "sockconn"         
mode        "wb"               
text        "binary"           
opened      "opened"           
can read    "yes"              
can write   "yes"              

$monitor
A connection with                              
description "->localhost:8880"
class       "sockconn"        
mode        "rb"              
text        "binary"          
opened      "opened"          
can read    "yes"             
can write   "yes"             

$output_file
[1] "C:\\Users\\Kerry\\AppData\\Local\\Temp\\Rtmpc3olX1\\file1ea45e3e48e1_spark.log"

$spark_context
<jobj[6]>
  class org.apache.spark.SparkContext
  org.apache.spark.SparkContext@6c32c97c

$java_context
<jobj[7]>
  class org.apache.spark.api.java.JavaSparkContext
  org.apache.spark.api.java.JavaSparkContext@230b8ac4

attr(,"class")
[1] "spark_connection"       "spark_shell_connection" "DBIConnection"  

@javierluraschi
Copy link
Collaborator

@Dragonfly99 yup! you are good to go! Instead you want to do:

sc <- spark_connect(master = "local")

but yeah, you should be able to follow all http://spark.rstudio.com tutorials, etc. If you hit other issues, let open a new issue.

Thanks for sticking through several days, it will really make a difference to others hitting this issue.

@Dragonfly99
Copy link
Author

Thanks so much for helping. I'm glad it works now and I can play around with big data. I also hope I contribute like you do some day when I have more experience. This was my first issue open and first "contribution" ^__^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants