Bug 1335228 - Change owner of Spark log to spark instead of hadoop user.

[jezdez]

If Spark is writing to the file using the "spark" user as indicated in the output Sam posted in bug 1335228 then we should chown the logfile.

[vitillo]

LGTM. Has this been manually tested?

[jezdez]

@vitillo Can you elaborate what manually testing means in that case?

[vitillo]

As this project doesn't have any automated test suite, I would like to know how you verified (supposedly "manually") that the issue mentioned by Sam has been fixed.

[jezdez]

@vitillo Right, I haven't run the script, do you mean by running the ansible playbook?

[vitillo]

Unfortunately we don't have different environments yet so you can't just run the ansible playbook to test a change in the bootstrap script.

One way to do it would be to create a vanilla EMR cluster (i.e. not from within ATMO), run the bootstrap script manually and finally launch a Spark job and verify that the error Sam mentioned is gone.

An even better way would be to fix Bug 1312747 before so that testing these kind of changes will become much easier.

[jezdez]

@vitillo Okay, that's some viable yak you just offered to be shaved. I accept :)

[vitillo]

\o/

[acmiyaguchi]

@jezdez Bumping this PR since all the requisites are in order. Changing the owner of the spark log does not fix the original bug. One of the reasons for this might be because spark is writing its event log to hdfs. It would still be nice to obtain a copy of the spark application logs outside of the yarn application logs.

[acmiyaguchi]

This doesn't fix the problem described in the bug.