[SPARK-30914][CORE][DOC] Add version information to the configuration of UI #27806
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
beliefer
commented
Mar 5, 2020
| @@ -25,31 +25,37 @@ private[spark] object UI { | |||
|
|
|||
| val UI_SHOW_CONSOLE_PROGRESS = ConfigBuilder("spark.ui.showConsoleProgress") | |||
| .doc("When true, show the progress bar in the console.") | |||
| .version("1.2.1") | |||
There was a problem hiding this comment.
SPARK-4017, commit ID: 04b1bdb#diff-364713d7776956cb8b0a771e9b62f82d
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(false) | ||
|
|
||
| val UI_CONSOLE_PROGRESS_UPDATE_INTERVAL = | ||
| ConfigBuilder("spark.ui.consoleProgress.update.interval") | ||
| .version("2.1.0") |
There was a problem hiding this comment.
SPARK-16919， commit ID: e076fb0#diff-fbf4e388a66b6a37e984b91cd71a3e2c
beliefer
commented
Mar 5, 2020
| .timeConf(TimeUnit.MILLISECONDS) | ||
| .createWithDefault(200) | ||
|
|
||
| val UI_ENABLED = ConfigBuilder("spark.ui.enabled") | ||
| .doc("Whether to run the web UI for the Spark application.") | ||
| .version("1.1.1") |
There was a problem hiding this comment.
SPARK-3490， commit ID: 937de93#diff-364713d7776956cb8b0a771e9b62f82d
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(true) | ||
|
|
||
| val UI_PORT = ConfigBuilder("spark.ui.port") | ||
| .doc("Port for your application's dashboard, which shows memory and workload data.") | ||
| .version("0.7.0") |
There was a problem hiding this comment.
No JIRA ID, commit ID: f03d976#diff-ed8dbcebe16fda5ecd6df1a981dc6fee
beliefer
commented
Mar 5, 2020
| .intConf | ||
| .createWithDefault(4040) | ||
|
|
||
| val UI_FILTERS = ConfigBuilder("spark.ui.filters") | ||
| .doc("Comma separated list of filter class names to apply to the Spark Web UI.") | ||
| .version("1.0.0") |
There was a problem hiding this comment.
SPARK-1189, commit ID: 7edbea4#diff-f79a5ead735b3d0b34b6b94486918e1c
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val UI_ALLOW_FRAMING_FROM = ConfigBuilder("spark.ui.allowFramingFrom") | ||
| .version("1.6.0") |
There was a problem hiding this comment.
SPARK-10589, commit ID: 5dbaf3d#diff-f79a5ead735b3d0b34b6b94486918e1c
beliefer
commented
Mar 5, 2020
| @@ -61,6 +67,7 @@ private[spark] object UI { | |||
| "through spark master/proxy public URL. This setting affects all the workers and " + | |||
| "application UIs running in the cluster and must be set on all the workers, drivers " + | |||
| " and masters.") | |||
| .version("2.1.0") | |||
There was a problem hiding this comment.
SPARK-15487, commit ID: 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d
beliefer
commented
Mar 5, 2020
| @@ -69,89 +76,107 @@ private[spark] object UI { | |||
| "in front of Spark Master. This is useful when running proxy for authentication e.g. " + | |||
| "OAuth proxy. Make sure this is a complete URL including scheme (http/https) and port to " + | |||
| "reach your proxy.") | |||
| .version("2.1.0") | |||
There was a problem hiding this comment.
SPARK-15487, commit ID: 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val UI_KILL_ENABLED = ConfigBuilder("spark.ui.killEnabled") | ||
| .doc("Allows jobs and stages to be killed from the web UI.") | ||
| .version("1.0.0") |
There was a problem hiding this comment.
SPARK-1202, commit ID: 211f974#diff-a40023c80383451b6e29ee7a6e0593e9
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(true) | ||
|
|
||
| val UI_THREAD_DUMPS_ENABLED = ConfigBuilder("spark.ui.threadDumpsEnabled") | ||
| .version("1.2.0") |
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(true) | ||
|
|
||
| val UI_PROMETHEUS_ENABLED = ConfigBuilder("spark.ui.prometheus.enabled") | ||
| .internal() | ||
| .doc("Expose executor metrics at /metrics/executors/prometheus. " + | ||
| "For master/worker/driver metrics, you need to configure `conf/metrics.properties`.") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-29064, commit ID: bbfaadb#diff-f70174ad0759db1fb4cb36a7ff9324a7
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(false) | ||
|
|
||
| val UI_X_XSS_PROTECTION = ConfigBuilder("spark.ui.xXssProtection") | ||
| .doc("Value for HTTP X-XSS-Protection response header") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22188, commit ID: 5a07aca#diff-6bdad48cfc34314e89599655442ff210
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .createWithDefaultString("1; mode=block") | ||
|
|
||
| val UI_X_CONTENT_TYPE_OPTIONS = ConfigBuilder("spark.ui.xContentTypeOptions.enabled") | ||
| .doc("Set to 'true' for setting X-Content-Type-Options HTTP response header to 'nosniff'") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22188, commit ID: 5a07aca#diff-6bdad48cfc34314e89599655442ff210
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(true) | ||
|
|
||
| val UI_STRICT_TRANSPORT_SECURITY = ConfigBuilder("spark.ui.strictTransportSecurity") | ||
| .doc("Value for HTTP Strict Transport Security Response Header") | ||
| .version("2.3.0") |
There was a problem hiding this comment.
SPARK-22188, commit ID: 5a07aca#diff-6bdad48cfc34314e89599655442ff210
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .createOptional | ||
|
|
||
| val UI_REQUEST_HEADER_SIZE = ConfigBuilder("spark.ui.requestHeaderSize") | ||
| .doc("Value for HTTP request header size in bytes.") | ||
| .version("2.2.3") |
There was a problem hiding this comment.
SPARK-26118, commit ID: 9ceee6f#diff-6bdad48cfc34314e89599655442ff210
beliefer
commented
Mar 5, 2020
| .bytesConf(ByteUnit.BYTE) | ||
| .createWithDefaultString("8k") | ||
|
|
||
| val UI_TIMELINE_TASKS_MAXIMUM = ConfigBuilder("spark.ui.timeline.tasks.maximum") | ||
| .version("1.4.0") |
There was a problem hiding this comment.
SPARK-7296, commit ID: a5f7b3b#diff-fa4cfb2cce1b925f55f41f2dfa8c8501
beliefer
commented
Mar 5, 2020
| .intConf | ||
| .createWithDefault(1000) | ||
|
|
||
| val ACLS_ENABLE = ConfigBuilder("spark.acls.enable") | ||
| .version("1.1.0") |
There was a problem hiding this comment.
SPARK-1890 and SPARK-1891， commit ID: e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .booleanConf | ||
| .createWithDefault(false) | ||
|
|
||
| val UI_VIEW_ACLS = ConfigBuilder("spark.ui.view.acls") | ||
| .version("1.0.0") |
There was a problem hiding this comment.
SPARK-1189, commit ID: 7edbea4#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val UI_VIEW_ACLS_GROUPS = ConfigBuilder("spark.ui.view.acls.groups") | ||
| .version("2.0.0") |
There was a problem hiding this comment.
SPARK-4224, commit ID: ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val ADMIN_ACLS = ConfigBuilder("spark.admin.acls") | ||
| .version("1.1.0") |
There was a problem hiding this comment.
SPARK-1890 and SPARK-1891, commit ID: e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val ADMIN_ACLS_GROUPS = ConfigBuilder("spark.admin.acls.groups") | ||
| .version("2.0.0") |
There was a problem hiding this comment.
SPARK-4224, commit ID: ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val MODIFY_ACLS = ConfigBuilder("spark.modify.acls") | ||
| .version("1.1.0") |
There was a problem hiding this comment.
SPARK-1890 and SPARK-1891, commit ID: e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val MODIFY_ACLS_GROUPS = ConfigBuilder("spark.modify.acls.groups") | ||
| .version("2.0.0") |
There was a problem hiding this comment.
SPARK-4224, commit ID: ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .toSequence | ||
| .createWithDefault(Nil) | ||
|
|
||
| val USER_GROUPS_MAPPING = ConfigBuilder("spark.user.groups.mapping") | ||
| .version("2.0.0") |
There was a problem hiding this comment.
SPARK-4224, commit ID: ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98
beliefer
commented
Mar 5, 2020
| .stringConf | ||
| .createWithDefault("org.apache.spark.security.ShellBasedGroupsMappingProvider") | ||
|
|
||
| val PROXY_REDIRECT_URI = ConfigBuilder("spark.ui.proxyRedirectUri") | ||
| .doc("Proxy address to use when responding with HTTP redirects.") | ||
| .version("3.0.0") |
There was a problem hiding this comment.
SPARK-30240, commit ID: a9fbd31#diff-f70174ad0759db1fb4cb36a7ff9324a7
beliefer
commented
Mar 5, 2020
| @@ -163,6 +188,7 @@ private[spark] object UI { | |||
| "This configuration replaces original log urls in event log, which will be also effective " + | |||
| "when accessing the application on history server. The new log urls must be permanent, " + | |||
| "otherwise you might have dead link for executor log urls.") | |||
| .version("3.0.0") | |||
There was a problem hiding this comment.
SPARK-26792, commit ID: d5bda2c#diff-f70174ad0759db1fb4cb36a7ff9324a7
|
Test build #119383 has finished for PR 27806 at commit
|
|
retest this please |
|
Test build #119384 has finished for PR 27806 at commit
|
|
Test build #119396 has finished for PR 27806 at commit
|
|
retest this please |
|
Merged to master. |
|
Test build #119430 has finished for PR 27806 at commit
|
HyukjinKwon
pushed a commit
that referenced
this pull request
Apr 5, 2020
… of UI ### What changes were proposed in this pull request? 1.Add version information to the configuration of `UI`. 2.Update the docs of `UI`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.ui.showConsoleProgress | 1.2.1 | SPARK-4017 | 04b1bdb#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.consoleProgress.update.interval | 2.1.0 | SPARK-16919 | e076fb0#diff-fbf4e388a66b6a37e984b91cd71a3e2c | spark.ui.enabled | 1.1.1 | SPARK-3490 | 937de93#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.port | 0.7.0 | None | f03d976#diff-ed8dbcebe16fda5ecd6df1a981dc6fee | spark.ui.filters | 1.0.0 | SPARK-1189 | 7edbea4#diff-f79a5ead735b3d0b34b6b94486918e1c | spark.ui.allowFramingFrom | 1.6.0 | SPARK-10589 | 5dbaf3d#diff-f79a5ead735b3d0b34b6b94486918e1c | spark.ui.reverseProxy | 2.1.0 | SPARK-15487 | 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.reverseProxyUrl | 2.1.0 | SPARK-15487 | 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.killEnabled | 1.0.0 | SPARK-1202 | 211f974#diff-a40023c80383451b6e29ee7a6e0593e9 | spark.ui.threadDumpsEnabled | 1.2.0 | SPARK-611 | 866c7bb#diff-5d18fb70c572369a0fff0b97de94f265 | spark.ui.prometheus.enabled | 3.0.0 | SPARK-29064 | bbfaadb#diff-f70174ad0759db1fb4cb36a7ff9324a7 | spark.ui.xXssProtection | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.xContentTypeOptions.enabled | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.strictTransportSecurity | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.requestHeaderSize | 2.2.3 | SPARK-26118 | 9ceee6f#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.timeline.tasks.maximum | 1.4.0 | SPARK-7296 | a5f7b3b#diff-fa4cfb2cce1b925f55f41f2dfa8c8501 | spark.acls.enable | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.view.acls | 1.0.0 | SPARK-1189 | 7edbea4#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.view.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.admin.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.admin.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.modify.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.modify.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.user.groups.mapping | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.proxyRedirectUri | 3.0.0 | SPARK-30240 | a9fbd31#diff-f70174ad0759db1fb4cb36a7ff9324a7 | spark.ui.custom.executor.log.url | 3.0.0 | SPARK-26792 | d5bda2c#diff-f70174ad0759db1fb4cb36a7ff9324a7 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Exists UT Closes #27806 from beliefer/add-version-to-UI-config. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
|
Merged to branch-3.0 too |
sjincho
pushed a commit
to sjincho/spark
that referenced
this pull request
Apr 15, 2020
… of UI ### What changes were proposed in this pull request? 1.Add version information to the configuration of `UI`. 2.Update the docs of `UI`. I sorted out some information show below. Item name | Since version | JIRA ID | Commit ID | Note -- | -- | -- | -- | -- spark.ui.showConsoleProgress | 1.2.1 | SPARK-4017 | 04b1bdb#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.consoleProgress.update.interval | 2.1.0 | SPARK-16919 | e076fb0#diff-fbf4e388a66b6a37e984b91cd71a3e2c | spark.ui.enabled | 1.1.1 | SPARK-3490 | 937de93#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.port | 0.7.0 | None | f03d976#diff-ed8dbcebe16fda5ecd6df1a981dc6fee | spark.ui.filters | 1.0.0 | SPARK-1189 | 7edbea4#diff-f79a5ead735b3d0b34b6b94486918e1c | spark.ui.allowFramingFrom | 1.6.0 | SPARK-10589 | 5dbaf3d#diff-f79a5ead735b3d0b34b6b94486918e1c | spark.ui.reverseProxy | 2.1.0 | SPARK-15487 | 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.reverseProxyUrl | 2.1.0 | SPARK-15487 | 92ce8d4#diff-364713d7776956cb8b0a771e9b62f82d | spark.ui.killEnabled | 1.0.0 | SPARK-1202 | 211f974#diff-a40023c80383451b6e29ee7a6e0593e9 | spark.ui.threadDumpsEnabled | 1.2.0 | SPARK-611 | 866c7bb#diff-5d18fb70c572369a0fff0b97de94f265 | spark.ui.prometheus.enabled | 3.0.0 | SPARK-29064 | bbfaadb#diff-f70174ad0759db1fb4cb36a7ff9324a7 | spark.ui.xXssProtection | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.xContentTypeOptions.enabled | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.strictTransportSecurity | 2.3.0 | SPARK-22188 | 5a07aca#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.requestHeaderSize | 2.2.3 | SPARK-26118 | 9ceee6f#diff-6bdad48cfc34314e89599655442ff210 | spark.ui.timeline.tasks.maximum | 1.4.0 | SPARK-7296 | a5f7b3b#diff-fa4cfb2cce1b925f55f41f2dfa8c8501 | spark.acls.enable | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.view.acls | 1.0.0 | SPARK-1189 | 7edbea4#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.view.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.admin.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.admin.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.modify.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe657#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.modify.acls.groups | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.user.groups.mapping | 2.0.0 | SPARK-4224 | ae79032#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 | spark.ui.proxyRedirectUri | 3.0.0 | SPARK-30240 | a9fbd31#diff-f70174ad0759db1fb4cb36a7ff9324a7 | spark.ui.custom.executor.log.url | 3.0.0 | SPARK-26792 | d5bda2c#diff-f70174ad0759db1fb4cb36a7ff9324a7 | ### Why are the changes needed? Supplemental configuration version information. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Exists UT Closes apache#27806 from beliefer/add-version-to-UI-config. Authored-by: beliefer <beliefer@163.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
1.Add version information to the configuration of
UI.2.Update the docs of
UI.I sorted out some information show below.
Why are the changes needed?
Supplemental configuration version information.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Exists UT