-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Failed while connecting to sparklyr to port (8880) #801
Comments
@Dragonfly99 this is a common error I've been trying to fix/reproduce for a while. If you have a few hours to help the community, find me on gitter to chat about this, I'm sure we can get it resolved but it's unfortunately hard without someone willing to put the time to debug this with us. You can find me online here: https://gitter.im/rstudio/sparklyr - Thanks for reporting this! |
Could you also share
Once installed, retry connecting, share error log and please also include the output of |
@Dragonfly99 any chance you could also run |
Here's what I got: sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] sparklyr_0.5.6-9010 bindrcpp_0.2 caret_6.0-76 lattice_0.20-35
[5] dplyr_0.7.1 purrr_0.2.2 readr_1.1.1 tidyr_0.6.3
[9] tibble_1.3.3 ggplot2_2.2.1 tidyverse_1.1.1
loaded via a namespace (and not attached):
[1] httr_1.2.1 jsonlite_1.5 splines_3.4.0 foreach_1.4.3
[5] modelr_0.1.0 shiny_1.0.3 assertthat_0.2.0 stats4_3.4.0
[9] cellranger_1.1.0 yaml_2.1.14 backports_1.1.0 quantreg_5.33
[13] glue_1.1.1 digest_0.6.12 rvest_0.3.2 minqa_1.2.4
[17] colorspace_1.3-2 httpuv_1.3.3 htmltools_0.3.6 Matrix_1.2-9
[21] plyr_1.8.4 psych_1.7.5 pkgconfig_2.0.1 devtools_1.13.2
[25] broom_0.4.2 SparseM_1.77 haven_1.0.0 config_0.2
[29] xtable_1.8-2 scales_0.4.1 lme4_1.1-13 MatrixModels_0.4-1
[33] git2r_0.18.0 mgcv_1.8-17 car_2.1-4 withr_1.0.2
[37] nnet_7.3-12 lazyeval_0.2.0 pbkrtest_0.4-7 mnormt_1.5-5
[41] mime_0.5 magrittr_1.5 readxl_1.0.0 memoise_1.1.0
[45] nlme_3.1-131 MASS_7.3-47 forcats_0.2.0 xml2_1.1.1
[49] foreign_0.8-67 tools_3.4.0 hms_0.3 stringr_1.2.0
[53] munsell_0.4.3 compiler_3.4.0 rlang_0.1.1 grid_3.4.0
[57] nloptr_1.0.4 iterators_1.0.8 rstudioapi_0.6 rappdirs_0.3.1
[61] base64enc_0.1-3 labeling_0.3 gtable_0.2.0 ModelMetrics_1.1.0
[65] codetools_0.2-15 DBI_0.7 curl_2.7 reshape2_1.4.2
[69] R6_2.2.0 lubridate_1.6.0 knitr_1.16 bindr_0.1
[73] rprojroot_1.2 stringi_1.1.5 parallel_3.4.0 Rcpp_0.12.11
[77] dbplyr_1.1.0 >sc <- spark_connect(master = "local", version = "2.1.0")
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (8113): Gateway in port (8880) did not respond.
Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 8113
Log: C:\Users\Kerry\AppData\Local\Temp\RtmpmwvQFd\file1a746c36c62_spark.log
---- Output Log ----
The system cannot find the path specified.
---- Error Log ---- > spark_context_config(sc)
Error in spark_context(sc) : object 'sc' not found
> java -version
Error: object 'java' not found |
@Dragonfly99 Sorry, I mean |
Does this file |
That path reads: "The system cannot find the path specified." > system("java -version")
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) |
Pushed a fix to increase logging, could you try the following and share the output? Really appreciated @Dragonfly99! devtools::install_github("rstudio/sparklyr")
# restart r session
sessionInfo()
sc <- spark_connect(master = "local", version = "2.1.0", config = list(sparklyr.log.console = TRUE)) |
No problem. I like helping and I want too want to play with big data. Here's the latest results: > sessionInfo()
R version 3.4.0 (2017-04-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] sparklyr_0.5.6-9011 bindrcpp_0.2 caret_6.0-76 lattice_0.20-35
[5] dplyr_0.7.1 purrr_0.2.2 readr_1.1.1 tidyr_0.6.3
[9] tibble_1.3.3 ggplot2_2.2.1 tidyverse_1.1.1
loaded via a namespace (and not attached):
[1] httr_1.2.1 jsonlite_1.5 splines_3.4.0 foreach_1.4.3
[5] modelr_0.1.0 shiny_1.0.3 assertthat_0.2.0 stats4_3.4.0
[9] cellranger_1.1.0 yaml_2.1.14 backports_1.1.0 quantreg_5.33
[13] glue_1.1.1 digest_0.6.12 rvest_0.3.2 minqa_1.2.4
[17] colorspace_1.3-2 httpuv_1.3.3 htmltools_0.3.6 Matrix_1.2-9
[21] plyr_1.8.4 psych_1.7.5 pkgconfig_2.0.1 devtools_1.13.2
[25] broom_0.4.2 SparseM_1.77 haven_1.0.0 config_0.2
[29] xtable_1.8-2 scales_0.4.1 lme4_1.1-13 MatrixModels_0.4-1
[33] git2r_0.18.0 mgcv_1.8-17 car_2.1-4 withr_1.0.2
[37] nnet_7.3-12 lazyeval_0.2.0 pbkrtest_0.4-7 mnormt_1.5-5
[41] mime_0.5 magrittr_1.5 readxl_1.0.0 memoise_1.1.0
[45] nlme_3.1-131 MASS_7.3-47 forcats_0.2.0 xml2_1.1.1
[49] foreign_0.8-67 tools_3.4.0 hms_0.3 stringr_1.2.0
[53] munsell_0.4.3 compiler_3.4.0 rlang_0.1.1 grid_3.4.0
[57] nloptr_1.0.4 iterators_1.0.8 rstudioapi_0.6 rappdirs_0.3.1
[61] base64enc_0.1-3 labeling_0.3 gtable_0.2.0 ModelMetrics_1.1.0
[65] codetools_0.2-15 DBI_0.7 curl_2.7 reshape2_1.4.2
[69] R6_2.2.0 lubridate_1.6.0 knitr_1.16 bindr_0.1
[73] rprojroot_1.2 stringi_1.1.5 parallel_3.4.0 Rcpp_0.12.11
[77] dbplyr_1.1.0 > sc <- spark_connect(master = "local", version = "2.1.0", config = list(sparklyr.log.console = TRUE))
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (941): Gateway in port (8880) did not respond.
Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 941
Log: C:\Users\Kerry\AppData\Local\Temp\RtmpmwvQFd\file1a74735f7e7a_spark.log
---- Output Log ----
---- Error Log ---- |
@Dragonfly99 do you see any files named |
I don't have that file. |
@Dragonfly99 thank you, could you try running
and share the results? |
@Dragonfly99 also, please run the
Thank you! |
I'm being told that spark-submit2.cmd and spark-submit.cmd are invalid names for some reason. The farthest I can cd into is |
@Dragonfly99 Can you share a screenshot and validate that For instance, in my Windows 10 installation, execution was enabled by default and looks as followed: |
@Dragonfly99 can you share the screenshot or text from the output you are getting in the windows terminal? Should look something like this: Both, for
and
|
@Dragonfly99 Also, how much memory and disk space do you have available on this machine? Could you try: spark_install(version = "2.1.0", reset = TRUE) and reconnect? There was a fix in |
This is what I get for that code spark_install(version = "2.1.0", reset = TRUE)
Spark 2.1.0 for Hadoop 2.7 or later already installed.
I'm going to to enter those other commands in the terminal and send you the screenshots. |
Thanks for entering those commands in the terminal, that would be much appreciated. I'm hoping we can get more info as to why "The system cannot find the path specified.", it's obviously there, so hopefully, we can see a clear permissions error or something else... |
and share the results? This is to list the files since I saw this post explaining that sometimes this error triggers when the file downloads incorrectly and has a size of zero.
Sys.getenv() Looks like we are getting close to resolving this, thanks for persevering on this issue, the |
Here's a screenshot of the directory results I have 2 questions so I make sure I do this right.
I'm happy to help. Here's hoping the solution is close. |
@Dragonfly99 thank you! We know the problem is not zero-size now, for your questions:
Lets see if it's a problem with long paths by copying everything into |
@Dragonfly99 Could you right-click on the There must be something off about that file that prevents us from executing this... |
@Dragonfly99 Could you run from the command prompt:
and
and
The |
@Dragonfly99 Nice! So that's the problem, this was much-much harder to troubleshoot than what I would expect, I'll make a fix in So I think what you want to do is to change the |
Ok I changed the path to point to jre.1.8.0_131. What do I do next? |
@Dragonfly99 Restart rstudio, retry |
I think it worked! Here's the new results:
|
@Dragonfly99 yup! you are good to go! Instead you want to do:
but yeah, you should be able to follow all http://spark.rstudio.com tutorials, etc. If you hit other issues, let open a new issue. Thanks for sticking through several days, it will really make a difference to others hitting this issue. |
Thanks so much for helping. I'm glad it works now and I can play around with big data. I also hope I contribute like you do some day when I have more experience. This was my first issue open and first "contribution" ^__^ |
Hi all. I'm trying to connect a new spark cluster on my local machine but I keep getting a connection error. Here are my specs:
OS Windows 10
R Version 3.4.0
Rstudio 1.0.143
devtools installed
I get this error:
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (7076): Gateway in port (8880) did not respond.
Path: C:\Users\Kerry\AppData\Local\rstudio\spark\Cache\spark-2.1.0-bin-hadoop2.7\bin\spark-submit2.cmd
Parameters: --class, sparklyr.Shell, "C:\Users\Kerry\Documents\R\win-library\3.4\sparklyr\java\sparklyr-2.1-2.11.jar", 8880, 7076
Log: C:\Users\Kerry\AppData\Local\Temp\RtmpohIScV\file2268c6d7434_spark.log
---- Output Log ----
The system cannot find the path specified.
---- Error Log ---
Apologies is this isn't the correct format. This is my first time submitting an error.
The text was updated successfully, but these errors were encountered: