[SPARK-8372] Do not show applications that haven't recorded their app ID yet. #7097
Conversation
…on not started The history server may show an incorrect App ID for an incomplete application like <App ID>.inprogress. This app info will never disappear even after the app is completed.  The cause of the issue is that a log path name is used as the app id when app id cannot be got during replay. Author: Carson Wang <carson.wang@intel.com> Closes apache#6827 from carsonwang/SPARK-8372 and squashes the following commits: cdbb089 [Carson Wang] Fix code style 3e46b35 [Carson Wang] Update code style 90f5dde [Carson Wang] Add a unit test d8c9cd0 [Carson Wang] Replaying events only return information when app is started
These logs don't have app IDs, so they should not be filtered.
|
I made some small changes on top of @carsonwang original patch, to account for old Spark versions. Most of my added changes are in the test code. |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36032 has started for PR 7097 at commit |
|
Test build #36032 has finished for PR 7097 at commit
|
|
Merged build finished. Test FAILed. |
|
Just to add a link from the other PR: #6827 |
|
retest this please |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36058 has started for PR 7097 at commit |
|
Test build #36058 has finished for PR 7097 at commit
|
|
Merged build finished. Test FAILed. |
| ui.getSecurityManager.setViewAcls(attempt.sparkUser, | ||
| appListener.viewAcls.getOrElse("")) | ||
| ui | ||
| }.orNull |
There was a problem hiding this comment.
If we return null here, is it possible that we have a Some(null) up there in L143? Should we return an option here instead and do flatMap up there?
There was a problem hiding this comment.
NAVER - http://www.naver.com/
sujkh@naver.com 님께 보내신 메일 <Re: [spark] [SPARK-8372] Do not show applications that haven't recorded their app ID yet. (#7097)> 이 다음과 같은 이유로 전송 실패했습니다.
받는 사람이 회원님의 메일을 수신차단 하였습니다.
|
This looks great. I left a few comments but I think this is basically ready to be merged. |
|
Merged build triggered. |
|
Merged build started. |
|
Test build #36156 has started for PR 7097 at commit |
|
Test build #36156 has finished for PR 7097 at commit
|
|
Merged build finished. Test PASSed. |
|
Merging into master 1.4. |
… ID yet. Showing these applications may lead to weird behavior in the History Server. For old logs, if the app ID is recorded later, you may end up with a duplicate entry. For new logs, the app might be listed with a ".inprogress" suffix. So ignore those, but still allow old applications that don't record app IDs at all (1.0 and 1.1) to be shown. Author: Marcelo Vanzin <vanzin@cloudera.com> Author: Carson Wang <carson.wang@intel.com> Closes #7097 from vanzin/SPARK-8372 and squashes the following commits: a24eab2 [Marcelo Vanzin] Feedback. 112ae8f [Marcelo Vanzin] Merge branch 'master' into SPARK-8372 7b91b74 [Marcelo Vanzin] Handle logs generated by 1.0 and 1.1. 1eca3fe [Carson Wang] [SPARK-8372] History server shows incorrect information for application not started Conflicts: core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
Showing these applications may lead to weird behavior in the History Server. For old logs, if
the app ID is recorded later, you may end up with a duplicate entry. For new logs, the app might
be listed with a ".inprogress" suffix.
So ignore those, but still allow old applications that don't record app IDs at all (1.0 and 1.1) to be shown.