-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Look for Spark's make-distribution.sh script in its new location (plus its current one) #93
Conversation
Looks good to me. Have you tested this both against a recent Spark commit, like |
I tested the script itself but not as a part of flintrock, should I? |
While trying to test my change, I'm getting: paramiko.ssh_exception.SSHException: not a valid EC private key file despite having a properly formatted .pem file. Do you have any idea what could be causing this? |
Hmm, I've never seen that error before. It seems to be ultimately coming from EC2? Are you able to use that same private key file to log into EC2 instances outside of Flintrock? |
Found my problem: the user was misconfigured. I tested the change against today's commit: apache/spark@4eace4d The build is correctly started. However, the spark core project won't compile (it might be because I'm using t2.micro instances). |
Yeah, to build Spark in a reasonable amount of time you'd need at least Thanks for contributing this patch and testing it out! I'll merge this in. |
Hmm, actually I'm having trouble getting this to work against the latest commit of Spark. I get this error:
Do you get the same error? This may be a subtle change on Spark's side that we have to handle. |
Trying on |
after cding into the
|
Apparently the parallel build option ( The first maven instruction, which is:
Also, there is a warning regarding parallel execution which might be causing the failure:
Are you ok with removing it? |
Btw, I tested the script as a part of flintrock with the two previously mentioned commits and it worked in both cases (having removed |
I think something else is going on. If I clone Spark locally and run
it works fine against the latest commit. This smells like something related to the shell environment over SSH. Interestingly, it seems that the commit that moved I think a good next step would be to try to find the exact Spark commit that breaks this. I'll poke around more myself later this week to try to find it. Sorry this turned into more than a simple change @BenFradet! I'd really like to keep the |
For me apache/spark@4eace4d fails to build both locally and remotely with I'll investigate later commits. |
I found it. This is the commit that breaks Source PR: apache/spark#11178 |
Mmh interesting |
Revisiting the error message you posted above @BenFradet, it looks like some project changes are interfering with the parallel build option, as you pointed out. 😞 That PR I linked to is probably just where this change was introduced. So I now agree with your earlier suggestion: The simplest thing to do is to simply remove the |
ok, will do |
fixes #91