installers that 'SetEnvironmentVariable'(s) from a service mess up corresponding service variable context

[mwallner]

what did I see?

after installing a package (Boxstarter 2.12.0) from a jenkins job, all other jobs that followed failed because powershell.exe was "broken" in that service session.

• a restart of the service (jenkinsslave) as well as a restart of the host (obviously) solved the problem.

why am I seeing this?

as a part of it's install behavior, Boxstarter modified $env:PSModulePath - for both 'User' and 'Machine' level. (see Boxstarter setup.ps1)
this is totally fine when run from a user session, but according to microsoft support, environmental variable changes will not be affected until system reboot. (which is weird, because in my case a service-restart seemed to do the trick also).

WHAT?

As a result of changing $PSModulePath, whenever a jenkins-job executed powershell.exe it quickly failed because "basic features" - such as "Split-String", "Join-Path" etc. are missing.
you'll see messages like
The term 'Split-String' is not recognized as the name of a cmdlet, function, script file, or operable program...

And that's where I got completely confused: if setting environmental variables from a service process only get re-read, it should at least not influence other processes that are started after that initial process finished.
In my case the $PSModulePath was neither the old, nor the new value it should be - but rather something else.

backing log data

I've added a bunch of debug-printlns right before and after installing that upgrade + before starting my failing jobs, here are the results:

before upgrading Boxstarter:

--- printDebugVar ---
PSModulePath -    User: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter
PSModulePath - Machine: C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PSModulePath -     env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
---------------------

after upgrading Boxstarter: (same Job/Process as above)

--- printDebugVar ---
PSModulePath -    User: C:\Users\builduser\Documents\WindowsPowerShell\Modules
PSModulePath - Machine: C:\ProgramData\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PSModulePath -     env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
---------------------

• ok, so ;C:\Users\builduser\AppData\Roaming\Boxstarter has been removed from the 'User' env-var 'PSModulePath', C:\ProgramData\Boxstarter; has been added to the 'Machine' env-var 'PSModulePath'
• that's exacly what should happen when you take a look at Boxstarter setup.ps1
  (and the same thing that's also happening when you run it manually in a user-session)

after upgrading Boxstarter: (new Job/Process, Jenkins-Service NOT restarted)

--- printDebugVar ---
PSModulePath -    User: C:\Users\builduser\Documents\WindowsPowerShell\Modules
PSModulePath - Machine: C:\ProgramData\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PSModulePath -     env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter
---------------------

now here is the fun part: where does $env:PSModulePath come from? - it's C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter

in my tests, $env:PSModulePath has always been a concat of [Environment]::GetEnvironmentVariable('PSModulePath', 'Machine') and [Environment]::GetEnvironmentVariable('PSModulePath', 'User')

also, when querying from a user-session on the very same machine, $env:PSModulePath still looks ok.

after upgrading Boxstarter: (new Job/Process, Jenkins-Service was restarted)

--- printDebugVar ---
PSModulePath -    User: C:\Users\builduser\Documents\WindowsPowerShell\Modules
PSModulePath - Machine: C:\ProgramData\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PSModulePath -     env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\ProgramData\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3\;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
---------------------

• everything working again as expected

my question(s)

• why is $env:PSModulePath neither the old, nor the new value after setting it from inside a service - for all child processes of that service started afterwards
• why does a restart of the service alone solve the problem when microsoft support states you have to reboot the machine
• what can be done to overcome this "limitation", other than not running anything that may change environmental variables from a service?

[ferventcoder]

Can you also put in a full repo? Steps are helpful.

[ferventcoder]

why does a restart of the service alone solve the problem when microsoft support states you have to reboot the machine

This actually makes sense to me - I was hoping it was just a restart of the service and not the machine.

[ferventcoder]

after upgrading Boxstarter: (new Job/Process, Jenkins-Service NOT restarted)

It looks like it is carrying the old USER only environment variables into process and ignoring machine variables.

Before upgrade (emphasis mine):

• PSModulePath - User: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter
• PSModulePath - Machine: C:\Program Files (x86)\PowerShell Community Extensions\Pscx3;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
• PSModulePath - [Process] env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules

After upgrade (new job/process without restart):

• PSModulePath - User: C:\Users\builduser\Documents\WindowsPowerShell\Modules
• PSModulePath - Machine: C:\ProgramData\Boxstarter;C:\Program Files (x86)\PowerShell Community Extensions\Pscx3;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
• PSModulePath - env: C:\Users\builduser\Documents\WindowsPowerShell\Modules;C:\Users\builduser\AppData\Roaming\Boxstarter

[ferventcoder]

It seems like there is a bug somewhere in Update-SessionEnvironmentVariables that is triggered by a service call. We also have the element of subprocesses (child processes being called) that never persist changes to environment data back up the stream - this is a known limitation of Windows. You can't add something to the PATH in a subprocess and have the parent process see it without something that helps it see that - but refreshenv.cmd / Update-SessionEnvironment are the hacks that allow for this behavior to occur.

Can a simple repo be a powershell script that does the following:

• Check environment variables
• Add something to environment variables to make a change to environment variables
• Check environment variables
• Start cmd.exe, then run powershell
• From the subprocess, call the check to environment variables - state you are in the subprocess
• From the subprocess, add something to the environment variables to make a change to this variable - persist it
• From the subprocess, call the check to environment variables - state you are in the subprocess
• Close that subprocess
• In the parent call Update-SessionEnvironment
• Open another subprocess
• From the subprocess, call the check to environment variables - state you are in the subprocess
• Close the subprocess
• Check environment variables

Then run this through a service process to see the behavior.

[mwallner]

this bug seemingly doesn't exist or behaves differently from a Docker (microsoft/windowsservercore:ltsc2016) instance..

I've create a little demo PS-script that should be able to cause the effect on Win 10 LTSB 1706 with Jenkins running as a Service.
edit: link to ps-gist: https://gist.github.com/mwallner/5736ad119398decbe538bbbcbd1b8978

[mwallner]

I've run this script twice on the same node and the error occured again.
as @ferventcoder suggested, it's really the case, that the old USER environent is set to env:PSModulePath, the Machine environment only gets reload/updated once the service is restarted.

see https://gist.github.com/mwallner/5736ad119398decbe538bbbcbd1b8978#gistcomment-2750787
vs https://gist.github.com/mwallner/5736ad119398decbe538bbbcbd1b8978#gistcomment-2750788

[mwallner]

Adding Update-SessionEnvironment did not change anything, $env:PSModulePath still points to the old USER environment

see https://gist.github.com/mwallner/5736ad119398decbe538bbbcbd1b8978#gistcomment-2750802

[mwallner]

I think $env:PSModulePath is really treated separately from other env vars.
maybe it'd be a temporary workaround to have Chocolatey (actually Update-SessionEnvironment ) check if it's value is "valid" and "fix" it - if it doesn't contain the machine environemnt.

[mwallner]

seems to be a "known" problem: pnp/PnP-PowerShell#37 (comment)