-
Notifications
You must be signed in to change notification settings - Fork 184
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
thunder-ec2 crashing during launch with spark 1.5.2 and spark 1.6.1 #272
Comments
Thunder version? Latest Spark locally? I've had some cluster startup issues recently which came down to having stale Spark with the current Thunder release on pypi. |
Local thunder (installed using pip) is 0.6.0 Have you tried booting a cluster today? Would you mind attempting? Thanks. On Thu, Apr 7, 2016 at 3:47 PM NP Young notifications@github.com wrote:
|
Similar error, getting |
@npyoung not sure if that is a related error or not. I consistently get the identical error message to what I pasted above. I may have tracked the problem down to the portion of spark-ec2/setup.sh that installs spark. The problem appears to be that the s3 hosting of the spark distribution is missing. Here is the relevant bit of the output from spark-ec2-setup.sh which is run by thunder ec2.py: (Note that before tracking this down I downgraded to spark 1.5.2 in hopes that might solve the problem)
|
The spark installation issue for 1.6.1 is similar but slightly different:
|
After installing spark version 1.6.0 everything works as expected. My assessment of the problem that sometime in the past few days the s3-hosted copy of spark 1.6.1 was corrupted such that the spark-ec2/setup.sh script could not unpack it. This caused the thunder-ec2 script to begin crashing. By pure coincidence my attempt to solve the problem by reverting to spark 1.5.2 also failed because someone forgot to host this version of spark on s3. By switching to spark 1.6.0, I can successfully launch clusters using thunder-ec2. Resolving this issue will require fixing the s3 hosting of spark variants 1.6.1 and 1.5.2. |
@nerduno @npyoung nice job tracking this down! I imagine the S3 version will get fixed soon, but you might want to post to the Spark mailing lists. This certainly sounds frustrating, and this kind of thing is one of the reasons that as of But moving forward, I'd rather work on a separate standalone utility that offers similar functionality, but is not so tightly coupled to All |
I believe that they just ran into this corrupt Spark package problem in I've been using The cluster it sets up has Spark and Hadoop but not much else, but a quick script that installs Thunder and a few key Python packages would make it easy for users to customize a vanilla flintrock cluster to their needs. Among other things, the default root volume size on flintrock clusters is greater than the 10GB limit of the |
Yes, I very much like the solution of having a script that installs anaconda, thunder, etc on top of a flintrock cluster. I haven't tried flintrock, but it is high on my to-do list primarily because of the default root volume size problem @broxtronix mentioned -- very frustrating. Related to this do know of way to force spark to clean up temporary files on the slaves? |
I'd just |
As of yesterday, thunder-ec2 crashes when launching a cluster with the error below. When logging into the master of the incompletely-launched cluster, /root/spark is largely empty with just an incomplete conf folder. I'm currently researching what is causing this problem, but I'm curious if others are having this problem?
The text was updated successfully, but these errors were encountered: