New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secret environment variables not accessible in task #145

Closed
robinweston opened this Issue May 10, 2016 · 14 comments

Comments

Projects
None yet
6 participants
@robinweston

robinweston commented May 10, 2016

We have a simple build task that takes in 2 variables. One is secret, the other not. The non-secret variable can be accessed fine by the task (a shell script that calls through to npm), but the secret variable is not accessible.

A printenv command executed before the npm call in the shell script does not show the secret variable (on the old agent it would show up but be starred out).

@bryanmacfarlane

This comment has been minimized.

Show comment
Hide comment
@bryanmacfarlane

bryanmacfarlane May 10, 2016

Member

Nit: Tasks don't take variables - they work on inputs and users can map variables into input values.

Then there's ad-hoc scripts. That's the powershell, schellscript, cmd task running your script.

Can you clarify - are you creating a task or are you running an ad-hoc with one of our tasks?

Member

bryanmacfarlane commented May 10, 2016

Nit: Tasks don't take variables - they work on inputs and users can map variables into input values.

Then there's ad-hoc scripts. That's the powershell, schellscript, cmd task running your script.

Can you clarify - are you creating a task or are you running an ad-hoc with one of our tasks?

@robinweston

This comment has been minimized.

Show comment
Hide comment
@robinweston

robinweston May 10, 2016

We're running a Shell Script task from the pre-defined list of available task types. This shell script is failing because it can't find a secret environment variable that it is expecting. The secret variable is set in the variables section of the build definition. To verify the variable is missing we added a printenv command to the shell script and the variable indeed does not show up.

The same Shell Script works fine when run on the old node build agent (https://github.com/Microsoft/vso-agent).

robinweston commented May 10, 2016

We're running a Shell Script task from the pre-defined list of available task types. This shell script is failing because it can't find a secret environment variable that it is expecting. The secret variable is set in the variables section of the build definition. To verify the variable is missing we added a printenv command to the shell script and the variable indeed does not show up.

The same Shell Script works fine when run on the old node build agent (https://github.com/Microsoft/vso-agent).

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple May 10, 2016

Member

For ad-hoc scripts, secret variables need to be passed in via inputs. That's by design.

Member

ericsciple commented May 10, 2016

For ad-hoc scripts, secret variables need to be passed in via inputs. That's by design.

@robinweston

This comment has been minimized.

Show comment
Hide comment
@robinweston

robinweston May 10, 2016

Ah ok thanks, that's good to know. So I have to explicitly use the $(variable_name) syntax and pass the secret variable as an argument to my custom shell script?

Is this behaviour documented anywhere, as it's a functionality change from the node build agent? I can absolutely see the value in the change from a security perspective.

robinweston commented May 10, 2016

Ah ok thanks, that's good to know. So I have to explicitly use the $(variable_name) syntax and pass the secret variable as an argument to my custom shell script?

Is this behaviour documented anywhere, as it's a functionality change from the node build agent? I can absolutely see the value in the change from a security perspective.

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple May 10, 2016

Member

It's documented here.

And I recently added it to the ##setvariable documentation here.

I'm unclear whether master from vso-agent has the behavior, or if only older versions didn't do it.

Member

ericsciple commented May 10, 2016

It's documented here.

And I recently added it to the ##setvariable documentation here.

I'm unclear whether master from vso-agent has the behavior, or if only older versions didn't do it.

@robinweston

This comment has been minimized.

Show comment
Hide comment
@robinweston

robinweston May 10, 2016

That's my bad for not hunting in the documentation. I wrongly assumed the behaviour would be the same across the new and old agents.

I'm pretty sure that secret variables do get set as environment variables in the master of vso-agent. We just set up a new agent yesterday and could see the behaviour.

On 10 May 2016, at 19:51, ericsciple notifications@github.com wrote:

It's documented here.

And I recently added it to the ##setvariable documentation here.

I'm unclear whether master from vso-agent has the behavior, or if only older versions didn't do it.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

robinweston commented May 10, 2016

That's my bad for not hunting in the documentation. I wrongly assumed the behaviour would be the same across the new and old agents.

I'm pretty sure that secret variables do get set as environment variables in the master of vso-agent. We just set up a new agent yesterday and could see the behaviour.

On 10 May 2016, at 19:51, ericsciple notifications@github.com wrote:

It's documented here.

And I recently added it to the ##setvariable documentation here.

I'm unclear whether master from vso-agent has the behavior, or if only older versions didn't do it.


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple May 10, 2016

Member

The problem that we're trying to solve is merging the code bases from the OSX/Linux agent and Windows agent :)

I wasn't aware of this difference. It's good that you pointed it out. @bryanmacfarlane is there any place in the new documentation we can put this kind of information?

Member

ericsciple commented May 10, 2016

The problem that we're trying to solve is merging the code bases from the OSX/Linux agent and Windows agent :)

I wasn't aware of this difference. It's good that you pointed it out. @bryanmacfarlane is there any place in the new documentation we can put this kind of information?

@mcm-ham

This comment has been minimized.

Show comment
Hide comment
@mcm-ham

mcm-ham May 18, 2016

Thanks for letting me know as well, just upgraded build agent. The https://www.visualstudio.com/docs/build/scripts/variables#SecretVariables link is currently 404.

mcm-ham commented May 18, 2016

Thanks for letting me know as well, just upgraded build agent. The https://www.visualstudio.com/docs/build/scripts/variables#SecretVariables link is currently 404.

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@bryanmacfarlane

This comment has been minimized.

Show comment
Hide comment
@bryanmacfarlane

bryanmacfarlane May 24, 2016

Member

Yeah, technically it was a bug in the node agent that we let it through. Should map secret variables into inputs. I'll create a coming from node agent doc.

Member

bryanmacfarlane commented May 24, 2016

Yeah, technically it was a bug in the node agent that we let it through. Should map secret variables into inputs. I'll create a coming from node agent doc.

@ajbeaven

This comment has been minimized.

Show comment
Hide comment
@ajbeaven

ajbeaven May 18, 2017

So is there any way to use a secret variable for a task's environment variable?

Could you please expand on what you mean by "map secret variables into inputs"? This is what I get when I true to use a secret variable in a build task:

image

image

I can understand why it wouldn't appear in plain text in the logs (this is good), but it also appears that the variable doesn't get mapped in the task arguments either. After all, ECHO is on. is what is displayed when no value is given to the echo command.

ajbeaven commented May 18, 2017

So is there any way to use a secret variable for a task's environment variable?

Could you please expand on what you mean by "map secret variables into inputs"? This is what I get when I true to use a secret variable in a build task:

image

image

I can understand why it wouldn't appear in plain text in the logs (this is good), but it also appears that the variable doesn't get mapped in the task arguments either. After all, ECHO is on. is what is displayed when no value is given to the echo command.

@ericsciple

This comment has been minimized.

Show comment
Hide comment
@ericsciple

ericsciple May 18, 2017

Member

It should work. Does CSC_KEY_PASSWORD have a value? Or does it contain a special leading character? E.g. & or " or something?

Member

ericsciple commented May 18, 2017

It should work. Does CSC_KEY_PASSWORD have a value? Or does it contain a special leading character? E.g. & or " or something?

@ajbeaven

This comment has been minimized.

Show comment
Hide comment
@ajbeaven

ajbeaven May 18, 2017

Urgh, ain't that the way. I re-added my variable just to be sure and after doing that, it started working.

I think I must have inadvertently toggled the "lock" icon at some stage, made some changes to one of the other variables and saved the page. It looks like this clears the value for the variable but you don't know as it appears as •••••••••••. I'd suggest that you shouldn't be able to toggle the "lock" icon on previously saved secure variables to stop this from happening in the future.

Thanks for your help anyway!

ajbeaven commented May 18, 2017

Urgh, ain't that the way. I re-added my variable just to be sure and after doing that, it started working.

I think I must have inadvertently toggled the "lock" icon at some stage, made some changes to one of the other variables and saved the page. It looks like this clears the value for the variable but you don't know as it appears as •••••••••••. I'd suggest that you shouldn't be able to toggle the "lock" icon on previously saved secure variables to stop this from happening in the future.

Thanks for your help anyway!

@andreacassioli

This comment has been minimized.

Show comment
Hide comment
@andreacassioli

andreacassioli Apr 4, 2018

Hi all,
I am smashing similar issues.
My deploy pipeline uses a key vault task to import a number of secrets. Then I have a shell task to perform substitutions to inject those secrets in a kubernetes deploy script.

All works well if I expliticly use the secret. Say a key x is accessible using $(x). But what I would like to do is the following:

  • get the list of keys from the kubernetes script which is a kind of template
  • for each key look up the corresponding secret and make the substitution

The first step is an easy one liner, but then it seems there is no way to access the imported secret in any indirect way. For instance, assuming y=x, in bash one should be able to say ${!y} to get the content of x. Any clue why? Is the shell task running on some kind of limited bash?

andreacassioli commented Apr 4, 2018

Hi all,
I am smashing similar issues.
My deploy pipeline uses a key vault task to import a number of secrets. Then I have a shell task to perform substitutions to inject those secrets in a kubernetes deploy script.

All works well if I expliticly use the secret. Say a key x is accessible using $(x). But what I would like to do is the following:

  • get the list of keys from the kubernetes script which is a kind of template
  • for each key look up the corresponding secret and make the substitution

The first step is an easy one liner, but then it seems there is no way to access the imported secret in any indirect way. For instance, assuming y=x, in bash one should be able to say ${!y} to get the content of x. Any clue why? Is the shell task running on some kind of limited bash?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment