New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Masked Passwords are exposed if a job is run in debug mode #1780

Closed
anelson425 opened this Issue Apr 4, 2016 · 16 comments

Comments

Projects
None yet
9 participants
@anelson425

anelson425 commented Apr 4, 2016

It looks like if you run a job in debug mode, masked passwords can be exposed on the console. This seems like somewhat of a security issue. I am curious as to whether this was intended functionality or not though.

If it wasn't, I would be more than happy to work on a code fix and issue a PR for it.

@atheiman

This comment has been minimized.

Show comment
Hide comment
@atheiman

atheiman Apr 4, 2016

Example:

I set the option secret to the value MySecret and use that password in a command workflow step:

echo $RD_OPTION_SECRET >> /tmp/secrets.txt

If I run this with normal logging, no problem. But if I run in debug mode, my secret value is stored in the job log output for others to see:

Setting environment variable: RD_OPTION_SECRET=MySecret
Setting environment variable: RD_SECUREOPTION_SECRET=MySecret

Note that the secure option is masked elsewhere in the debug log:

secureOption={secret=****}
option={secret=****}

atheiman commented Apr 4, 2016

Example:

I set the option secret to the value MySecret and use that password in a command workflow step:

echo $RD_OPTION_SECRET >> /tmp/secrets.txt

If I run this with normal logging, no problem. But if I run in debug mode, my secret value is stored in the job log output for others to see:

Setting environment variable: RD_OPTION_SECRET=MySecret
Setting environment variable: RD_SECUREOPTION_SECRET=MySecret

Note that the secure option is masked elsewhere in the debug log:

secureOption={secret=****}
option={secret=****}
@cchesser

This comment has been minimized.

Show comment
Hide comment
@cchesser

cchesser Aug 4, 2016

We are running into this issue as well. As mentioned by @anelson425 , I would be happy to issue the enhancement if there can be some general guidance on expected scope of the change (or design). I would expect to take a similar approach of building a masking OutputStream which the logger would utilize that can get context of the secrets to match and then mask. Similar to how the Mask Password Plugin via the MaskPasswordsOutputStream works with Jenkins.

cchesser commented Aug 4, 2016

We are running into this issue as well. As mentioned by @anelson425 , I would be happy to issue the enhancement if there can be some general guidance on expected scope of the change (or design). I would expect to take a similar approach of building a masking OutputStream which the logger would utilize that can get context of the secrets to match and then mask. Similar to how the Mask Password Plugin via the MaskPasswordsOutputStream works with Jenkins.

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Aug 4, 2016

Member

a pr for this would be great.

some pointers for an implementation:

We create a "loghandler" to write the log output here: https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/ExecutionService.groovy#L813

We capture the System.out/err and redirect to the loghandler here: https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/ExecutionService.groovy#L886

I think a "MaskPasswords" loghandler would be the way to do this, as System.out/err feeds into that, but plugins/steps can also write to the loghandler directly. (A MaskPasswordsOutputStream would probably be ok, but would not cover all possible ways to write log data.)

There are two sets of "password" values. One is meant to never be exposed in scripts/commands (the Secure Authentication Option data), and one is meant to be used in scripts (Secure Option data). In that linked method they are misnamed as "extraParams" and "extraParamsExposed". Probably all of the values for those two datasets should be masked, even tho the Secure Authentication Option data would never be available in scripts.

Member

gschueler commented Aug 4, 2016

a pr for this would be great.

some pointers for an implementation:

We create a "loghandler" to write the log output here: https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/ExecutionService.groovy#L813

We capture the System.out/err and redirect to the loghandler here: https://github.com/rundeck/rundeck/blob/master/rundeckapp/grails-app/services/rundeck/services/ExecutionService.groovy#L886

I think a "MaskPasswords" loghandler would be the way to do this, as System.out/err feeds into that, but plugins/steps can also write to the loghandler directly. (A MaskPasswordsOutputStream would probably be ok, but would not cover all possible ways to write log data.)

There are two sets of "password" values. One is meant to never be exposed in scripts/commands (the Secure Authentication Option data), and one is meant to be used in scripts (Secure Option data). In that linked method they are misnamed as "extraParams" and "extraParamsExposed". Probably all of the values for those two datasets should be masked, even tho the Secure Authentication Option data would never be available in scripts.

@cchesser

This comment has been minimized.

Show comment
Hide comment
@cchesser

cchesser Aug 4, 2016

Sounds good. I will start by looking at the construction pattern of the loghandler via the LoggingService which can get a set of strings to mask.

cchesser commented Aug 4, 2016

Sounds good. I will start by looking at the construction pattern of the loghandler via the LoggingService which can get a set of strings to mask.

@Kronnork

This comment has been minimized.

Show comment
Hide comment
@Kronnork

Kronnork Sep 10, 2016

This is issue is occurring in 2.6.8 Version of Rundeck

Kronnork commented Sep 10, 2016

This is issue is occurring in 2.6.8 Version of Rundeck

@Payback159

This comment has been minimized.

Show comment
Hide comment
@Payback159

Payback159 Apr 7, 2017

Hello,

is there a way to mask the password in the 2.7.2 Version of Rundeck?

Payback159 commented Apr 7, 2017

Hello,

is there a way to mask the password in the 2.7.2 Version of Rundeck?

@majkinetor

This comment has been minimized.

Show comment
Hide comment
@majkinetor

majkinetor Jun 15, 2017

+1

Serious security issue.

majkinetor commented Jun 15, 2017

+1

Serious security issue.

@gschueler gschueler added this to the 2.9.0 milestone Jun 15, 2017

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Jun 15, 2017

Member

fixed by #2482 adds a Mask passwords plugin

Member

gschueler commented Jun 15, 2017

fixed by #2482 adds a Mask passwords plugin

@gschueler gschueler closed this Jun 20, 2017

@ioannisas

This comment has been minimized.

Show comment
Hide comment
@ioannisas

ioannisas Aug 3, 2017

Tested that on beta 2.9.0-1 version of Rundeck.
It seems that secure options are still exposed in the debug level. The plugin is working but not completely. There are also some output fields where the value of secure option is exposed.

Following is a portion of the debug log where we can see masked and unmasked logs of the same option:

[admin@localhost _][VERBOSE] [workflow] Begin execution: node-first
...
[admin@localhost _][VERBOSE] Create workflow engine with state: StateLogger{state=DataState{state={option.password=test-pass, .... job.name=test, job.successOnEmptyNodeFilter=false, job.executionType=user, secureOption.password=test-pass
[admin@localhost _][VERBOSE] DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={option.password=test-pass,.....job.executionType=user, secureOption.password=test-pass
....
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_JOB_SERVERURL=http://localhost:4440/
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_OPTION_PASSWORD=[SECURE]
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_NODE_NAME=localhost
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_SECUREOPTION_PASSWORD=[SECURE]
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_JOB_EXECUTIONTYPE=user
....
[admin@localhost _][VERBOSE] DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={option.password=test-pass,....job.successOnEmptyNodeFilter=false, job.executionType=user, secureOption.password=test-pass,

ioannisas commented Aug 3, 2017

Tested that on beta 2.9.0-1 version of Rundeck.
It seems that secure options are still exposed in the debug level. The plugin is working but not completely. There are also some output fields where the value of secure option is exposed.

Following is a portion of the debug log where we can see masked and unmasked logs of the same option:

[admin@localhost _][VERBOSE] [workflow] Begin execution: node-first
...
[admin@localhost _][VERBOSE] Create workflow engine with state: StateLogger{state=DataState{state={option.password=test-pass, .... job.name=test, job.successOnEmptyNodeFilter=false, job.executionType=user, secureOption.password=test-pass
[admin@localhost _][VERBOSE] DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={option.password=test-pass,.....job.executionType=user, secureOption.password=test-pass
....
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_JOB_SERVERURL=http://localhost:4440/
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_OPTION_PASSWORD=[SECURE]
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_NODE_NAME=localhost
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_SECUREOPTION_PASSWORD=[SECURE]
20:36:27 [rundeck@localhost 1][VERBOSE] Setting environment variable: RD_JOB_EXECUTIONTYPE=user
....
[admin@localhost _][VERBOSE] DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={option.password=test-pass,....job.successOnEmptyNodeFilter=false, job.executionType=user, secureOption.password=test-pass,

@niphlod

This comment has been minimized.

Show comment
Hide comment
@niphlod

niphlod Sep 6, 2017

in 2.9.2 the problem is still present for the debug output
/cc @gschueler

niphlod commented Sep 6, 2017

in 2.9.2 the problem is still present for the debug output
/cc @gschueler

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Oct 13, 2017

Member

@niphlod i am not able to reproduce, do you have an example?

Member

gschueler commented Oct 13, 2017

@niphlod i am not able to reproduce, do you have an example?

@niphlod

This comment has been minimized.

Show comment
Hide comment
@niphlod

niphlod Oct 14, 2017

I'm using ScriptPlugin mostly and if you enable the debug, you can see secure/masked bits in the internals logging of rundeck, as in a RemoteScriptNodeStep "launching path/to_script with args [-var MASKED ...]" . This was in 2.9.2 for sure, I'll update to 2.9.4 next week and I'll be able to provide a screenshot if needed.

niphlod commented Oct 14, 2017

I'm using ScriptPlugin mostly and if you enable the debug, you can see secure/masked bits in the internals logging of rundeck, as in a RemoteScriptNodeStep "launching path/to_script with args [-var MASKED ...]" . This was in 2.9.2 for sure, I'll update to 2.9.4 next week and I'll be able to provide a screenshot if needed.

@ioannisas

This comment has been minimized.

Show comment
Hide comment
@ioannisas

ioannisas Oct 14, 2017

@niphlod did you enable the Mask Password Plugin?

ioannisas commented Oct 14, 2017

@niphlod did you enable the Mask Password Plugin?

@niphlod

This comment has been minimized.

Show comment
Hide comment
@niphlod

niphlod Oct 15, 2017

niphlod commented Oct 15, 2017

@gschueler

This comment has been minimized.

Show comment
Hide comment
@gschueler

gschueler Oct 16, 2017

Member

@niphlod it is supposed to mask all output, including debug level and any internal logging. if it is not doing it, that is a bug for sure.

Member

gschueler commented Oct 16, 2017

@niphlod it is supposed to mask all output, including debug level and any internal logging. if it is not doing it, that is a bug for sure.

@niphlod

This comment has been minimized.

Show comment
Hide comment
@niphlod

niphlod Nov 7, 2017

@gschueler : I'm so sorry it took so long but I confirm it's fixed. The key point is that internal logging has - in some cases, like RemoteScriptNodeStep - pre and post logging that are not covered by applying the log filter to the step only. It has to be applied on the whole job to be sure secure options are masked properly.

See [SECURE] vs VALUE_SHOULD_NOT_BE_DISPLAYED in the following lines: first batch is from step execution itself, second batch is from internals logging.

[workflow] beginExecuteNodeStep(nodename): NodeDispatch: StepExecutionItem{type='NodeDispatch', keepgoingOnSuccess=false, hasFailureHandler=false}
[psn-winrm-fastcopy] executing: [powershell.exe, -NoProfile, -File, C:\RUNDECK\libext\cache\psn-rundeck-winrm-fastcopy-plugin\ps-file-copier.ps1, -hostname, ${node.hostname}, -protocol, ${node.winrm-protocol}, -username, ${node.username}, -password, ${config.winrmPassword}, -include_port_in_spn, ${node.winrm-spn-add-port}, -src, C:\RUNDECK\libext\cache\psn-rundeck-wdeploy-plugin\psn-wdeploy-pull.ps1, -dst, C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1]
[psn-winrm-fastcopy]: result code: 0
[psn-winrm-fastcopy]: result filepath: C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1
[psn-winrm-exec-combined2] execCommand started, command: C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1 -SourcePath '"c:\a_path"' -TargetPath '"d:\anotherpath"' -DoNotDelete "false" -Skip_files "^*app_offline.htm$" -Skip_dirs "\\App_data" -ComputerName "some.computer.domain" -RemoteUser "0" -ReportProgress "false" -SecurePass [SECURE]
[psn-winrm-exec-combined2] executing: [[powershell.exe, -NoProfile, -File, C:\RUNDECK\libext\cache\psn-rundeck-winrm-exec-combined2-plugin\ps-command-executor.ps1, -hostname, ${node.hostname}, -protocol, ${node.winrm-protocol}, -username, ${node.username}, -password, ${config.winrmPassword}, -include_port_in_spn, ${node.winrm-spn-add-port}, -remotecomputerpass, ${node.remotecomputerpass}, -runaspass, ${node.runaspass}]]


2: Workflow step finished, result: Dispatch successful (1 nodes)
[workflow] Finish step: 2,NodeDispatch
OperationSuccess: operation succeeded: StepSuccess{stepNum=2, stepResultCapture=StepResultCapture{stepResult=Dispatch successful (1 nodes), stepSuccess=true, statusString='null', controlBehavior=Dispatch successful (1 nodes), resultData=MultiDataContextImpl(map={ContextView(node:nodename)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:2, node:nodename)=BaseDataContext{{exec={exitCode=0}}}}, base=null)}, newState=DataState{state={after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}}}
WillProcessStateChange: saw state changes: {after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}
Update conditional state: {after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}
Update conditional state: {step.1.start=true, step.2.start=true}
DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={job.url=https://fiercely.domain/rundeck/project/HCCDIAG/execution/follow/473264, job.loglevel=DEBUG, step.any.state.success=true, node.os-name=Windows Server 2012 R2, node.wdeploy:TargetPath=${globals.DEV_DEPLOYFOLDER}, node.hostname=some.node.domain, node.wdeploy:TemplateOffline=c:\lots_of_things\here.html, globals.RELEASEFOLDER=D:\dumpme\, option.SecurePass=VALUE_SHOULD_NOT_BE_DISPLAYED, globals.TST_DEPLOYFOLDER=d:\anotherpath, node.tags=DEV, option.TARGET_NODE=nodename, job.name=Pull, step.1.state=success, node.wdeploy:RemotePass=0, step.2.completed=true, globals.DEV_BACKUPFOLDER=D:\TOBackup, job.successOnEmptyNodeFilter=true, job.serverUUID=B064F003-7E27-4365-9E43-9614909CE249, step.1.completed=true, job.wasRetry=false, job.project=HCCDIAG, step.1.start=true, job.username=niphlod@domain.com, node.os-version=, workflow.keepgoing=false, node.wdeploy:SourcePath=${globals.RELEASEFOLDER}\Dev\WebApp, option.winrmElevated=1, option.DoNotDelete=false, job.id=a2e57820-7b62-4432-9720-ad0348bd6537, after.step.2=true, after.step.1=true, step.2.start=true, workflow.state=started, node.os-family=windows, job.user.name=niphlod@domain.com, option.Skip_dirs=\\App_data, node.description=Dev Webapp, option.ReportProgress=false, globals.VALIDATION=ciao,miao, node.username=an_user, node.wdeploy:RemoteUser=0, node.winrm-protocol=http, job.executionType=user, node.wdeploy:ArchivePath=${globals.DEV_BACKUPFOLDER}, secureOption.SecurePass=VALUE_SHOULD_NOT_BE_DISPLAYED, node.winrmPassword=keys/win/aduser/${node.username}, before.step.1=false, before.step.2=false, node.wdeploy:ComputerName=some.computer.domain, node.os-arch=, job.retryAttempt=0, globals.TST_BACKUPFOLDER=D:\TOBackup, globals.DEV_DEPLOYFOLDER=d:\anotherpath, option.Skip_files=^*app_offline.htm$, job.group=WebApp, job.execid=473264, node.name=nodename, job.serverUrl=https://fiercely.domain/rundeck/, step.2.state=success, node.node-executor=psn-winrm-exec-combined2}}}

niphlod commented Nov 7, 2017

@gschueler : I'm so sorry it took so long but I confirm it's fixed. The key point is that internal logging has - in some cases, like RemoteScriptNodeStep - pre and post logging that are not covered by applying the log filter to the step only. It has to be applied on the whole job to be sure secure options are masked properly.

See [SECURE] vs VALUE_SHOULD_NOT_BE_DISPLAYED in the following lines: first batch is from step execution itself, second batch is from internals logging.

[workflow] beginExecuteNodeStep(nodename): NodeDispatch: StepExecutionItem{type='NodeDispatch', keepgoingOnSuccess=false, hasFailureHandler=false}
[psn-winrm-fastcopy] executing: [powershell.exe, -NoProfile, -File, C:\RUNDECK\libext\cache\psn-rundeck-winrm-fastcopy-plugin\ps-file-copier.ps1, -hostname, ${node.hostname}, -protocol, ${node.winrm-protocol}, -username, ${node.username}, -password, ${config.winrmPassword}, -include_port_in_spn, ${node.winrm-spn-add-port}, -src, C:\RUNDECK\libext\cache\psn-rundeck-wdeploy-plugin\psn-wdeploy-pull.ps1, -dst, C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1]
[psn-winrm-fastcopy]: result code: 0
[psn-winrm-fastcopy]: result filepath: C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1
[psn-winrm-exec-combined2] execCommand started, command: C:\WINDOWS\TEMP\45460-473264-some.computer.domain_HCCDiag-psn-wdeploy-pull.ps1 -SourcePath '"c:\a_path"' -TargetPath '"d:\anotherpath"' -DoNotDelete "false" -Skip_files "^*app_offline.htm$" -Skip_dirs "\\App_data" -ComputerName "some.computer.domain" -RemoteUser "0" -ReportProgress "false" -SecurePass [SECURE]
[psn-winrm-exec-combined2] executing: [[powershell.exe, -NoProfile, -File, C:\RUNDECK\libext\cache\psn-rundeck-winrm-exec-combined2-plugin\ps-command-executor.ps1, -hostname, ${node.hostname}, -protocol, ${node.winrm-protocol}, -username, ${node.username}, -password, ${config.winrmPassword}, -include_port_in_spn, ${node.winrm-spn-add-port}, -remotecomputerpass, ${node.remotecomputerpass}, -runaspass, ${node.runaspass}]]


2: Workflow step finished, result: Dispatch successful (1 nodes)
[workflow] Finish step: 2,NodeDispatch
OperationSuccess: operation succeeded: StepSuccess{stepNum=2, stepResultCapture=StepResultCapture{stepResult=Dispatch successful (1 nodes), stepSuccess=true, statusString='null', controlBehavior=Dispatch successful (1 nodes), resultData=MultiDataContextImpl(map={ContextView(node:nodename)=BaseDataContext{{exec={exitCode=0}}}, ContextView(step:2, node:nodename)=BaseDataContext{{exec={exitCode=0}}}}, base=null)}, newState=DataState{state={after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}}}
WillProcessStateChange: saw state changes: {after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}
Update conditional state: {after.step.2=true, step.any.state.success=true, before.step.2=false, step.2.state=success, step.2.completed=true}
Update conditional state: {step.1.start=true, step.2.start=true}
DidProcessStateChange: applied state changes and rules (changed? true): StateLogger{state=DataState{state={job.url=https://fiercely.domain/rundeck/project/HCCDIAG/execution/follow/473264, job.loglevel=DEBUG, step.any.state.success=true, node.os-name=Windows Server 2012 R2, node.wdeploy:TargetPath=${globals.DEV_DEPLOYFOLDER}, node.hostname=some.node.domain, node.wdeploy:TemplateOffline=c:\lots_of_things\here.html, globals.RELEASEFOLDER=D:\dumpme\, option.SecurePass=VALUE_SHOULD_NOT_BE_DISPLAYED, globals.TST_DEPLOYFOLDER=d:\anotherpath, node.tags=DEV, option.TARGET_NODE=nodename, job.name=Pull, step.1.state=success, node.wdeploy:RemotePass=0, step.2.completed=true, globals.DEV_BACKUPFOLDER=D:\TOBackup, job.successOnEmptyNodeFilter=true, job.serverUUID=B064F003-7E27-4365-9E43-9614909CE249, step.1.completed=true, job.wasRetry=false, job.project=HCCDIAG, step.1.start=true, job.username=niphlod@domain.com, node.os-version=, workflow.keepgoing=false, node.wdeploy:SourcePath=${globals.RELEASEFOLDER}\Dev\WebApp, option.winrmElevated=1, option.DoNotDelete=false, job.id=a2e57820-7b62-4432-9720-ad0348bd6537, after.step.2=true, after.step.1=true, step.2.start=true, workflow.state=started, node.os-family=windows, job.user.name=niphlod@domain.com, option.Skip_dirs=\\App_data, node.description=Dev Webapp, option.ReportProgress=false, globals.VALIDATION=ciao,miao, node.username=an_user, node.wdeploy:RemoteUser=0, node.winrm-protocol=http, job.executionType=user, node.wdeploy:ArchivePath=${globals.DEV_BACKUPFOLDER}, secureOption.SecurePass=VALUE_SHOULD_NOT_BE_DISPLAYED, node.winrmPassword=keys/win/aduser/${node.username}, before.step.1=false, before.step.2=false, node.wdeploy:ComputerName=some.computer.domain, node.os-arch=, job.retryAttempt=0, globals.TST_BACKUPFOLDER=D:\TOBackup, globals.DEV_DEPLOYFOLDER=d:\anotherpath, option.Skip_files=^*app_offline.htm$, job.group=WebApp, job.execid=473264, node.name=nodename, job.serverUrl=https://fiercely.domain/rundeck/, step.2.state=success, node.node-executor=psn-winrm-exec-combined2}}}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment