Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-27869][CORE] Redact sensitive information in System Properties from UI #24733

Closed
wants to merge 1 commit into from
Closed

[SPARK-27869][CORE] Redact sensitive information in System Properties from UI #24733

wants to merge 1 commit into from

Conversation

aaruna
Copy link
Contributor

@aaruna aaruna commented May 28, 2019 •
edited by dongjoon-hyun
Loading

What changes were proposed in this pull request?

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

How was this patch tested?

Manual test. Run the following and see the UI.

bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'

srowen
srowen approved these changes May 29, 2019
View reviewed changes
Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me. Does it happen to match any JVM properties, or really will only affect user-supplied secret, token, password keys?

@SparkQA
Copy link

SparkQA commented May 29, 2019

Test build #4787 has finished for PR 24733 at commit 4498c34.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 29, 2019

Test build #4789 has finished for PR 24733 at commit 4498c34.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-27869] Redact sensitive information in System Properties from UI [SPARK-27869][CORE] Redact sensitive information in System Properties from UI May 29, 2019
dongjoon-hyun
dongjoon-hyun approved these changes May 29, 2019
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you for your first contribution, @aaruna ! Welcome to the Apache Spark community.

I updated the PR description and tested manually.

Merged to master/branch-2.4/branch-2.3.

Also, thank you, @srowen .

dongjoon-hyun pushed a commit that referenced this pull request May 29, 2019
@aaruna @dongjoon-hyun
[SPARK-27869][CORE] Redact sensitive information in System Properties…
b876c14 
… from UI

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

Manual test. Run the following and see the UI.
```
bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'
```

Closes #24733 from aaruna/27869.

Authored-by: Aaruna <aaruna.godthi@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit bfa7f11)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
dongjoon-hyun pushed a commit that referenced this pull request May 29, 2019
@aaruna @dongjoon-hyun
[SPARK-27869][CORE] Redact sensitive information in System Properties…
1d35fff 
… from UI

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

Manual test. Run the following and see the UI.
```
bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'
```

Closes #24733 from aaruna/27869.

Authored-by: Aaruna <aaruna.godthi@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit bfa7f11)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@aaruna aaruna deleted the 27869 branch May 29, 2019 23:12
@aaruna
Copy link
Contributor Author

aaruna commented May 29, 2019

@dongjoon-hyun @srowen Thank you for reviewing and merging this.

Does it happen to match any JVM properties, or really will only affect user-supplied secret, token, password keys?

This PR will only redact the System Properties wherein the property key/value matches the spark.redaction.regex config value (default of (?i)secret|password).

@srowen
Copy link
Member

srowen commented May 29, 2019

@aaruna Yes, of course. My question was: does that regex match any standard JVM system properties? I don't think so, and maybe OK to redact anyway, but I was trying to understand whether it would ever affect anything but user-supplied system properties.

rluta pushed a commit to rluta/spark that referenced this pull request Sep 17, 2019
@aaruna
[SPARK-27869][CORE] Redact sensitive information in System Properties…
0ec0df2 
… from UI

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

Manual test. Run the following and see the UI.
```
bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'
```

Closes apache#24733 from aaruna/27869.

Authored-by: Aaruna <aaruna.godthi@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit bfa7f11)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
kai-chi pushed a commit to kai-chi/spark that referenced this pull request Sep 26, 2019
@aaruna @kai-chi
[SPARK-27869][CORE] Redact sensitive information in System Properties…
6212005 
… from UI

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

Manual test. Run the following and see the UI.
```
bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'
```

Closes apache#24733 from aaruna/27869.

Authored-by: Aaruna <aaruna.godthi@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit bfa7f11)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
igreenfield pushed a commit to axiomsl/spark that referenced this pull request Nov 4, 2019
@aaruna
[SPARK-27869][CORE] Redact sensitive information in System Properties…
a8b9f70 
… from UI

Currently system properties are not redacted. This PR fixes that, so that any credentials passed as System properties are redacted as well.

Manual test. Run the following and see the UI.
```
bin/spark-shell --conf 'spark.driver.extraJavaOptions=-DMYSECRET=app'
```

Closes apache#24733 from aaruna/27869.

Authored-by: Aaruna <aaruna.godthi@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit bfa7f11)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srowen srowen srowen approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aaruna @SparkQA @srowen @dongjoon-hyun