DSP-15832: Use home dir for Spark JobServer data and logs #18
Conversation
jacek-lewandowski
force-pushed
the
DSP-15832-dse
branch
from
9d7a269
to
183e3dc
Compare
| @@ -14,12 +14,12 @@ spark { | |||
|
|
|||
| jobserver { | |||
| port = 8090 | |||
| jar-store-rootdir = /tmp/jobserver/jars | |||
| jar-store-rootdir = ${HOME}/.spark-jobserver/jars | |||
There was a problem hiding this comment.
Does this config has variable expansion? Will Spark Jobserver eventually replace ${HOME} with actual path?
| @@ -22,13 +22,13 @@ spark { | |||
| # } | |||
|
|
|||
| filedao { | |||
| rootdir = /tmp/spark-jobserver/filedao/data | |||
| rootdir = ${HOME}/.spark-jobserver/filedao/data | |||
|
|
||
| jobdao = spark.jobserver.io.JobFileDAO | ||
|
|
||
| filedao { | ||
| rootdir = /tmp/spark-job-server/filedao/data | ||
| rootdir = ${HOME}/.spark-jobserver/filedao/data |
There was a problem hiding this comment.
This change will cause "historical data" loss after upgrade. Jars, info about past jobs will be missing after upgrade. Do we want to do it in patch release?
There was a problem hiding this comment.
I think that people can stick to old configuration. I don't know if configuration files are being changed when upgrading patch release at all (my guess would be no).
All in all, this is something we should ask @brianmhess
There was a problem hiding this comment.
On the other hand, the ticket is about not creating such directories with 777 permissions because they would not pass security audits. So I don't think we have other ways to fix that than notifying users to copy their files to respective home directories.
No description provided.