Sparkling Water fails to create h2oContext in simple spark project #466
Comments
|
Hi @renegademonkey , thanks for the report. It seems that for some reason H2O couldn't find IP address on the local machine. It happened to me as well on some networks. Can we verify this in a little bit simple environment pls ? Could you pls download just standalone H2O from the following link - http://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/7/index.html and start h2o as instructed on the page ? Single node is just fine. I'm just interested if this is also reproducible from pure h2o without sparkling water. Thanks! Kuba |
|
Hi @jakubhava ! Thanks for the prompt response - this has been driving me mad for days. The strangest thing is that every few days sparkling water WILL work once with the same setup, but then doesn't work again :-/ . So running h2o in local standalone mode works fine and I can start up flow (see full log below). I can also run h2o from R in local mode without problems. Any ideas what could be causing this or a workaround? h2o connect log: |
|
Good to know that H2O works. The next step is to try Sparkling Water in local mode.
In Sparkling Water we use the IP address provided by Spark which in some environments is not what we want. We can override this by h2o client network & mask and h2o nodes network & mask. More info is here https://github.com/h2oai/sparkling-water/blob/master/doc/configuration/configuration_properties.rst . Look for Please let me know if you need more help configuring this or if the issue remains. |
|
trying it with the new conf settings leads to several repeated errors: |
|
running sparkling water locally with a local spark cluster seems to start up fine - but no h2o commands are available in the sparkling-water shell. |
|
I'm still having trouble with sparkling water on our cluster, though now not sure if this is still related to the Since I thought it might be a firewall issue, I switched my driver's network switch. My spark connection still terminates before starting an h20 context, but now the only non-INFO log message is Any ideas what might be going on here? |
|
Are you starting Sparkling Water in internal backend ( it's the default and you would definitely know if it would be the second - external backend ) This message says that there is collision in jar hashes - you may have several H2O clusters with the same name started or your cluster is using different sparkling-water jar artefacts. Can you please verify that you don't have extra clusters running and your spark driver and executors are using the exact same Sparkling Water jar ? |
|
OK, that was the issue with the jar mismatch - I was simultaneously testing sparkling water via rsparkling on the same cluster (which is having the same connection issues) - hence the clash. Now I'm back to my original error, |
|
It seems like it could be a communication issue the between the spark master / executors / driver when starting the h2o context. But this is strange as I can run regular spark jobs on this cluster without any communication issues. |
|
OK, after days of debugging, I've figured out that that
And it works in scala! However, I now need to set the same value for sparkling water in R - how do I set this property in the config file when connecting in R via rsparkling? |
|
Amazing @renegademonkey, thank you very much for the time you put into this and the debugging. We will make sure this is easily configurable in sparkling water. For now, you can use Kuba |
|
Hi @jakubhava - thanks once again for replying so quickly. I had discovered those settings a little while ago and have been trying different combinations of them for the past hour, but in R i still get My configs have been in a .yml file rather than in a spark config file created in R, i.e. with the following code: I have tried this with and without quotes around the -D settings. I imagine a yml file shouldn't make a difference - other settings specified in the yml always work. |
|
aaaaaand also just tried setting it in an R script config file - same error. |
|
@jakubhava, any further suggestions on what to try / how to troubleshoot this? |
|
@renegademonkey, sorry, didn’t have time to look at the problem so far in RSparkling. Good news is that it works using the Scala API. Have you figured it out so far in RSparkling ? Can you please have a look here ? sparklyr/sparklyr#895 It’s not the same, but very similar issue related to setting system property. Thanks, |
|
Hi @renegademonkey, I will close this issue for now, but feel free to comment/reopen . Please also have a look on the latest comment where they are setting up system property from SparklyR. |
|
so for some reason it is all working again in R after the christmas break, with no real config changes that I recall making. not sure I will ever get to the bottom of this one . . . thanks for your help again @jakubhava! |
I am setting up for the first time Sparkling Water on a standalone cluster running spark 2.2. I have run Sparkling Water on such a cluster before via R (using rsparkling + sparklyr + h2o), but am having issues setting this up as a spark application (in scala).
The app is built with Maven, so I have added the latest sparkling water dependancy:
Then the app code is as follows:
I then compile the fat jar to send to the cluster, however the h2oContext never gets created and the SparkContext gets shut down with exit code 255. The app exiting with no error codes before an h2o context is created - the only potentially useful message is IP address not found on this machine.
I've tried this with Sparkling Water version 2.2.0 and get the same issues, also tried adding dependencies for sparkling-water-ml and sparkling-water-repl, as well as adding all the h2o core dependencies (though assuming these are not needed as they are integrated into sparkling water?).
The strange thing is that I get almost the exact same issue when trying to connect via R (using rsparkling and sparklyr, see here) - and that method worked correctly up until a few weeks ago.
See log below.
The text was updated successfully, but these errors were encountered: