During deployment, update, or add-node you may encounter 'Access Denied'. Some examples include:
During deployment you might see the following error message(s)
Exception: Connecting to remote server <hostname/IP> failed with the following error message: Access is denied
Exception : Type 'ValidateConnectivity' of Role 'EnvironmentValidator' raised an exception: Unable to create a valid session to <ip>: [<ip>]
Connecting to remote server <ip> failed with the following error message : Access is denied.
During update download stage, you may see:
update package download failure with details similar to "Action plan GetCauDeviceInfo ID xxx failed with state: Failed".
During any action plan, if you see the following, go to "Known Causes" #6
Connecting to remote server <ClusterName> failed with the following error message : An unknown security error occured
- The LCM (deployment user) credentials were not updated properly using the Set-AzureStackLCMUserPassword cmdlet. This cmdlet is responsible for updating the password in Active Directory as well as updating it in the ECE Store. Both of these should be in sync, otherwise there will be access denied issues.
- The LCM user does not have sufficient permissions on the Organization Unit (OU) in Active Directory (AD).
- The NTLM policy, configured via Group Policy, may be blocking remote operations such as Invoke-Command. This can occur if NTLM is restricted either at the OU level or on the individual nodes or the domain controller via applied Group Policy Objects (GPOs).
- The WinRM trusted hosts configuration is set up incorrectly.
- The LCM user is part of the "Protected Users" group (See Protected Users heading below).
- Cluster's SPN is misconfigured (See Unknown Security Error heading below)
Before running the following steps, please log into a node with the LCM user credential to ensure the credential works. If you do not know your LCM username, please view the Retrieving your LCM (user deployment) username section. If you do not know your LCM user credentials, then you will need to recreate the LCM user and follow this document to ensure the LCM user is set up properly.
Run the script below on all cluster nodes. Replace the $lcmUser variable with your LCM (deployment) user, and fill in $groupsContainingLCMUser if not logged in as lcmUser.
# Set your LCM user
$lcmUser = 'DOMAIN\LCMUser'
# If logged in as LCMUser, run script at this point. Otherwise go to next comment
$groupsContainingLCMUser = (whoami /groups) |
Select-Object -Skip 6 |
ForEach-Object {
($_ -split '\s{2,}')[0]
} |
Where-Object { $_ -and $_ -ne "===" }
# If not logged in as lcmUser, manually populate groups lcmUser belongs to, and uncomment below
<#$groupsContainingLCMUser = @(
'DOMAIN\Group1',
'DOMAIN\Group2',
'DOMAIN\Group3'
)#>
Write-Output "Checking if the LCM user or any of the specified groups are in the local Administrators group..."
try {
$adminGroupMembers = Get-LocalGroupMember -Group 'Administrators'
} catch {
Write-Error "Failed to retrieve Administrators group members. Error: $_"
return
}
$found = $false
# Check if any of the groups are in the Administrators group
foreach ($group in $groupsContainingLCMUser) {
if ($adminGroupMembers.Name -contains $group) {
Write-Output "Group '$group' is in the Administrators group."
$found = $true
}
}
# Check if LCM user is directly in the Administrators group
if ($adminGroupMembers.Name -contains $lcmUser) {
Write-Output "User '$lcmUser' is directly in the Administrators group."
$found = $true
}
if ($found) {
Write-Output "$lcmUser has administrative rights either directly or via one of the specified groups."
} else {
Write-Output "$lcmUser does NOT have administrative rights via the specified groups or directly."
}If the script shows that the lcmUser is not in the local Administrators group, then run the mitigation below:
# Replace 'DOMAIN\LCMUser' with your actual LCM username
$lcmUser = 'DOMAIN\LCMUser'
Write-Output "Attempting to add $lcmUser to the local Administrators group..."
try {
Add-LocalGroupMember -Group 'Administrators' -Member $lcmUser
Write-Output "$lcmUser was successfully added to the local Administrators group."
} catch {
Write-Error "Failed to add $lcmUser to the Administrators group. Error: $_"
}Verify the LCM user credentials work with all iterations of Invoke-Command with and without CredSSP and using the hostname or the IP as the target:
$credential = Get-Credential # Input your LCM (deployment user) credentials
$targetHostname = "<target_hostname>" # Replace with your target host
$targetIp = "<target_ip>" # Replace with your target IP address
Invoke-Command -computername $targetHostname -credential $credential -scriptblock {hostname}
Invoke-Command -ComputerName $targetHostname -Credential $credential -Authentication Credssp -ScriptBlock {whoami}
Invoke-Command -computername $targetIp -credential $credential -scriptblock {hostname}
Invoke-Command -ComputerName $targetIp -Credential $credential -Authentication Credssp -ScriptBlock {whoami}If there are no issues running the previous script, then proceed to Step 3, otherwise:
- If no Invoke-Command works (with or without CredSSP), double check you have entered the right LCM user credentials. If you are certain they are correct, then check the WinRM Trusted Hosts configuration is set up properly to include all hostnames, FQDNs and IPs of all nodes in your cluster.
- If Invoke-Command without CredSSP works, but CredSSP does not work, please review the CredSsp-Authentication-Issues Doc.
- If Invoke-Command with hostname works, but using the IP does not work, ensure the WinRM Trusted Hosts configuration is set up properly to include all IPs of all nodes in your cluster and that a Group Policy Object (GPO) is not blocking NTLM.
If Invoke-Command works, then validate the credential exists in ECE store by running the script in the Validating LCM (user deployment) credentials match the ECE Store section. If this script returns that they do not match, run the mitigation script from the same section.
Verify the Active Directory permissions are set properly for the LCM user. Log into the domain controller node using the LCM user credentials and run the following to get the Access Control List (ACL) for the user on the OU. Replace the LCM username in the script with your LCM username and the sample OU with your environment OU. You can get the OU by running Get-ADOrganizationalUnit -Filter * on the domain controller (DC) node. You should also check the permissions of any groups that the LCM user belongs to, since they are inherited by the LCM user.
# Define the target OU (update to match your environment)
$ou = "OU=MyOU,DC=domain,DC=com"
# Get the ACL of the OU
$acl = Get-Acl "AD:$ou"
# Define the LCM username (without domain prefix)
$lcmUsername = "<lcm_username>"
# Get the domain name in short form
$domain = (Get-WmiObject -Class Win32_ComputerSystem).Domain
$shortDomain = $domain.Split('.')[0]
$user = "$shortDomain\$lcmUsername"
$userAcl = $acl.Access | Where-Object { $_.IdentityReference -like "*$user*" }
$userAcl | Select-Object IdentityReference, ActiveDirectoryRights, AccessControlTypeYou should see output like the following:
IdentityReference ActiveDirectoryRights AccessControlType
----------------- --------------------- -----------------
<domain>\<LCM Username> ReadProperty Allow
<domain>\<LCM Username> GenericAll Allow
<domain>\<LCM Username> CreateChild, DeleteChild Allow
The user should have the following permissions on the OU:
- ReadProperty
- GenericAll
- CreateChild
- DeleteChild
If you do not see all of these ActiveDirectoryRights listed for this user, please follow instructions to prepare active directory, including running the AsHciADArtifactsPreCreationTool listed in the wiki to ensure all permissions are set appropriately for the user. Ensure these same permissions are applied to all OU objects and their descendents.
TrustedHosts should be checked on each node for correct configuration.
Get-Item WSMan:\localhost\Client\TrustedHostsTrustedHosts should include the following:
- Each cluster node's hostname
- Each cluster node's IP address
- Each cluster node's FQDN
To set TrustedHosts, use one of the commands below and replace the placeholder values with your cluster node hostnames, IPs, and FQDNs.
# Setting and overwriting all values
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "hostname1,hostname2,192.168.0.1,192.168.0.2,hostname1.contoso.local,hostname2.contoso.local" -Force# Adding missing values to existing TrustedHosts
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "hostname1.contoso.local,hostname2.contoso.local" -Concatenate -ForceNOTE: Please note that the LCM Username should be provided without domain and not contain any special characters. To Validate that the username provided is of the correct format, run:
if ($credential.UserName -match '^[^\\]+(?=\\)|(?<=@).+$') {
throw "Please provide user name without domain."
}Proceed with credential validation:
$owner_node = (Get-ClusterResource *Orc*).OwnerNode.Name
$owner_nodefunction CredentialsHelper
{
[CmdletBinding(DefaultParameterSetName = 'VerifyCredential')]
param (
[Parameter(Mandatory, ParameterSetName = 'WithCredential')]
[pscredential]
$Credential,
[Parameter(Mandatory, ParameterSetName = 'FetchLCMUserName')]
[switch]
$FetchUsername
)
try
{
# Retrieve the latest ECEWinService NuGet Version
$eceWinService = Get-ChildItem "C:\Agents" -Directory |
Where-Object { $_.Name -match 'Microsoft\.AzureStack\.Solution\.ECEWinService\.(\d+\.\d+\.\d+\.\d+)' } |
ForEach-Object {
[PSCustomObject]@{
Path = $_.FullName
Version = [version]$matches[1]
}
} |
Sort-Object Version -Descending |
Select-Object -First 1
$eceWinServiceVersion = $eceWinService.Version.ToString()
$eceWinServicePath = $eceWinService.Path
# Load Assemblies
[System.Reflection.Assembly]::LoadFile("$eceWinServicePath\content\ECEWinService\CloudEngine.dll") | Out-Null
[System.Reflection.Assembly]::LoadFile("$eceWinServicePath\content\ECEWinService\Microsoft.AzureStack.Orchestration.Common.Packaging.Contract.dll") | Out-Null
[System.Reflection.Assembly]::LoadFile("$eceWinServicePath\content\ECEWinService\Microsoft.AzureStack.Orchestration.Common.Packaging.dll") | Out-Null
[System.Reflection.Assembly]::LoadFile("$eceWinServicePath\content\ECEWinService\Microsoft.Diagnostics.Tracing.EventSource.dll") | Out-Null
# Load MetricTelemetry.dll if it exists, otherwise fallback to Telemetry.dll
$metricTelemetryPath = "$eceWinServicePath\content\ECEWinService\Microsoft.AzureStack.Solution.MetricTelemetry.dll"
$telemetryPath = "$eceWinServicePath\content\ECEWinService\Microsoft.AzureStack.Solution.Telemetry.dll"
if (Test-Path $metricTelemetryPath) {
[System.Reflection.Assembly]::LoadFile($metricTelemetryPath) | Out-Null
} elseif (Test-Path $telemetryPath) {
[System.Reflection.Assembly]::LoadFile($telemetryPath) | Out-Null
} else {
Write-Warning "Neither MetricTelemetry.dll nor Telemetry.dll found in $eceWinServicePath"
}
#Retrieve LCM User Username
Import-Module ECEClient 3>$null 4>$null
$eceClient = Create-ECEClusterServiceClient
$cloudDefinitionAsXmlString = (Get-CloudDefinition -EceClient $eceClient).CloudDefinitionAsXmlString
$cloudDefElements = [System.Xml.Linq.XElement]::Parse($cloudDefinitionAsXmlString)
$customerConfigurationObject = New-Object -TypeName 'CloudEngine.Configurations.CustomerConfiguration' -ArgumentList $cloudDefElements
$cloudRoleObject = [CloudEngine.Configurations.ConfigurationPathExtensions]::Find($customerConfigurationObject, 'Cloud')
[CloudEngine.Configurations.IInterface] $interface = $cloudRoleObject.Interface('Build')
$eceParams = $interface.GetInterfaceParameters()
$securityInfo = $ECEParams.Roles["Cloud"].PublicConfiguration.PublicInfo.SecurityInfo
$DAdmin = $securityInfo.DomainUsers.User | Where Role -eq "DomainAdmin"
$DAAdminUserCredential = $ECEParams.GetCredential($DAdmin.Credential)
if ($FetchUsername)
{
Write-AzsSecurityVerbose "Your LCM username is: $($DAAdminUserCredential.UserName)" -Verbose
return
}
$DAAdminUserPassword = $DAAdminUserCredential.GetNetworkCredential().password
$userProvidedPassword = $Credential.GetNetworkCredential().Password
$userProvidedUserName = $Credential.GetNetworkCredential().UserName
if ($DAAdminUserCredential.UserName -ne $userProvidedUserName)
{
Write-AzsSecurityWarning -Message "Provided LCM credential user name does not match the one in ECE." -Verbose
}
if ($DAAdminUserPassword -eq $userProvidedPassword)
{
Write-AzsSecurityVerbose -Message "Found matching credentials in ECE store." -Verbose
}
else
{
Write-AzsSecurityWarning -Message "Could not find matching credentials in ECE store. Please proceed with mitigation to update LCM user password in ECE." -Verbose
}
}
catch
{
Write-AzsSecurityWarning -Message "There was an issue verifying credentials. Error $_. Please reach out to Microsoft support for help" -Verbose
}
}
$lcmCredentials = Get-Credential
CredentialsHelper -Credential $lcmCredentialsPlease input your LCM user credentials when prompted. DO NOT include the domain as part of the username in the credential.
# Prompt for credentials
$credential = Get-Credential
# Validate credentials
try {
# Attempt to invoke a simple command (Get-Process) on the local machine to validate credentials
Invoke-Command -ScriptBlock { whoami } -Credential $credential -ErrorAction Stop -ComputerName localhost
Write-Host "Credential validation successful." -ForegroundColor Green
}
catch {
Write-Host "Credential validation failed. Please check the username and password." -ForegroundColor Red
return
}
# Import the necessary module
Import-Module "C:\Program Files\WindowsPowerShell\Modules\Microsoft.AS.Infra.Security.SecretRotation\PasswordUtilities.psm1" -DisableNameChecking
# Print the User Name
Write-Host "Username provided: $($credential.UserName)" -ForegroundColor Cyan
# Validate that the username provided is of the correct format. Username should be provided without domain and not contain any special characters.
if($credential.UserName -match '^[^\\]+(?=\\)|(?<=@).+$')
{
throw "Please provide user name without domain."
}
# Check the status of the ECE cluster group
$eceClusterGroup = Get-ClusterGroup | Where-Object { $_.Name -eq "Azure Stack HCI Orchestrator Service Cluster Group" }
if ($eceClusterGroup.State -ne "Online") {
Write-AzsSecurityError -Message "ECE cluster group is not in an Online state. Cannot continue with password rotation." -ErrRecord $_
}
# Update ECE with the new password
Write-AzsSecurityVerbose -Message "Updating password in ECE" -Verbose
$ECEContainersToUpdate = @(
"DomainAdmin",
"DeploymentDomainAdmin",
"CloudAdmin"
)
foreach ($containerName in $ECEContainersToUpdate) {
Set-ECEServiceSecret -ContainerName $containerName -Credential $credential 3>$null 4>$null
}
Write-AzsSecurityVerbose -Message "Finished updating credentials in ECE." -VerboseRun the following script on your HCI node to retrieve the LCM username:
NOTE: Please copy and run the 'CredentialsHelper' from the above "Validating LCM (User Deployment) Credentials Match the ECE Store" section.
CredentialsHelper -FetchUsernameVerify that you do not have policies in your domain controller that are restricting NTLM access. Please review Network security: Restrict NTLM: NTLM authentication in this domain.
Deployment fails with access denied when doing invoke-command or new-pssession from node1 -> node1
The following variations of invoke-command or new-pssession from node1 -> node1 fail
- new-pssession <node1 hostname>
- new-pssession localhost
- new-pssession <loopback IP>
The following variations of invoke-command or new-pssession succeed
- new-pssession <node1 hostname> -EnableNetworkAccess
- new-pssession <node1 hostname.fqdn>
- new-pssession <node2 hostname>
Repro the issue by running the following from node1:
new-pssession <node1 hostname>
$event = Get-WinEvent -LogName Security -FilterXPath "*[System[(EventID=4625)]]" | Select-Object -First 1
$event.messageIf you see the following error code in the message, this is an indication of the problem
Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006E
Sub Status: 0xC000006E
From the DC, run the following:
Get-ADUser -Identity <LCMUser> -Properties MemberOf | Select-Object -ExpandProperty MemberOf # Use LCM username without domain prefixIf the output has something similar to CN="...protectedusers..." the LCM user is part of the "Protected Users" group. If both validations passed, follow the mitigation below.
The LCM user must be removed from the "Protected Users" group.
Note this section applies to errors with Invoke-Command to the <ClusterName> only, not to the <HostName>
Invoke-Command from any host node to the <ClusterName> fails with:
Connecting to remote server <ClusterName> failed with the following error message : An unknown security error occured
The issue can be manually repro'd with the following:
Invoke-Command -ComputerName <ClusterName> -Credential <LCMUserCreds> -Authentication Credssp -ScriptBlock { hostname }The following scenarios still work:
- Invoke-Command to <ClusterName> without Credssp
- Invoke-Command to <HostName> with Credssp
From any host node check ClusterName's SPN for WSMAN/<ClusterName>
setspn -L <ClusterName>You should see a bunch of entries including HOST/<Clustername> and HOST/<ClusterName>.<FQDN>
If you see entries for either wsman/<ClusterName> or wsman/<ClusterName>.<FQDN> follow the mitigation
Note - this issue is only for the Cluster's SPN. Nodes with SPN's of wsman/<HostName> is expected
Remove the entries for wsman/<ClusterName> and wsman/<ClusterName>.<FQDN>
setspn -D wsman/<ClusterName> <ClusterName>
setspn -D wsman/<ClusterName>.<FQDN> <ClusterName>Deployment fails with Access is Denied during AddAsZHostToDomain step, right after node is joined to domain. An example of error message can be:
Type 'AddAsZHostToDomain' of Role 'BareMetal' raised an exception: Unable to connect to the <NODE_IP> using .\Administrator credentials. Please check the .\Administrator credentials.
When you recently made Group Policy change and applied to Azure Local clusters, your admin account, either local admin or domain account added to node's local Administrators group, fails to remotely logon to a node and gets Access is Denied error.
Please run the following script on your node through BMC console, or through another user that can still remotely connect (i.e., via PowerShell session or RDP) to the impacted node.
$ErrorActionPreference = 'Stop'
# SIDs to check
$Sid_LocalAdminCapability = 'S-1-5-114'
$Sid_AdministratorsGroup = 'S-1-5-32-544'
$isBlocked = $false
$TempCfg = Join-Path $env:TEMP 'secpol.cfg'
Write-Host "=== Checking SeDenyNetworkLogonRight ==="
# Export effective user rights
secedit /export /areas USER_RIGHTS /cfg $TempCfg | Out-Null
$denyLine = Select-String -Path $TempCfg `
-Pattern '^SeDenyNetworkLogonRight\s*=' `
-ErrorAction SilentlyContinue |
Select-Object -First 1
Remove-Item $TempCfg -Force -ErrorAction SilentlyContinue
if (-not $denyLine) {
Write-Host "SeDenyNetworkLogonRight: NOT configured"
Write-Host "Local admins blocked: NO"
}
else {
Write-Host "SeDenyNetworkLogonRight: CONFIGURED"
Write-Host "Raw value:"
Write-Host " $($denyLine.Line)"
$assignedSids = (($denyLine.Line -split '=', 2)[1] -split ',') |
ForEach-Object { $_.Trim().TrimStart('*') }
Write-Host "Assigned SIDs:"
foreach ($sid in $assignedSids) {
Write-Host " - $sid"
}
$blocked114 = $assignedSids -contains $Sid_LocalAdminCapability
$blocked544 = $assignedSids -contains $Sid_AdministratorsGroup
$isBlocked = $blocked114 -or $blocked544
if ($isBlocked) {
Write-Host "Local admins blocked: YES"
}
else {
Write-Host "Local admins blocked: NO"
}
}
if ($isBlocked)
{
Write-Host ""
Write-Host "=== Identifying applying GPO(s) ==="
$gpresultText = gpresult /scope computer /Z 2>$null
$gpos = $gpresultText |
Select-String 'Policy:\s+DenyNetworkLogonRight' -Context 3,3 |
ForEach-Object {
$_.Context.PreContext |
Select-String 'GPO:' |
ForEach-Object {
($_ -split 'GPO:\s*')[1].Trim()
}
} |
Select-Object -Unique
if ($gpos) {
foreach ($gpo in $gpos) {
Write-Host "GPO that contains the DenyNetworkLogonRight policy: $gpo"
}
}
else {
Write-Host "GPO that contains the DenyNetworkLogonRight policy: UNKNOWN (Run gpresult /scope computer /Z manually to confirm, or it may not be applied via Group Policy)"
}
}
Write-Host "=== Done ==="If you see Local admins blocked: YES in the output, that means the issue is validated.
Here is an example output from the validation script that validated the issue.
=== Checking SeDenyNetworkLogonRight ===
SeDenyNetworkLogonRight: CONFIGURED
Raw value:
SeDenyNetworkLogonRight = *S-1-5-32-544
Assigned SIDs:
- S-1-5-32-544
Local admins blocked: YES
=== Identifying applying GPO(s) ===
GPO that contains the DenyNetworkLogonRight policy: Test-GPO
=== Done ===
Please check the last section and find the GPO name that contains this DenyNetworkLogonRight policy. In Group Policy console it is displayed as Deny Access to this computer from the network. Go to the Group Policy management console and find the GPO with that name. Either uncheck the Define these policy settings checkbox for this policy, or remove Administrators group from the policy value.
Location where you can find the policy:

Once finished, run
gpupdate /forceon each impacted node. This will help mitigate the issue. Once this is done on all impacted nodes, you can then retry the failed action.