Set-AzStorageBlobContent throws error when trying to open file

[fmicle]

Description

I am migrating our Azure deployment scripts from AzureRM to Az and I ran into a problem with Set-AzStorageBlobContent throwing an error when trying to open the Azure template file. These Azure templates have been working and change very rarely. I tried full file name with backslashes, full name with forward slashes, relative file name without any special characters from the current template folder, in every situation the file name is resolved correctly to the full name, but it cannot be opened. It complains about "Illegal characters in path."

Script/Steps for Reproduction

PS C:\dev\pq\service\scripts\azure\NestedTemplates> Set-AzStorageBlobContent -Container "florin-container" -Context $storageAccount.Context -File "ApplicationInsights.json"

Module Version

Get-Module -ListAvailable

    Directory: C:\Program Files\WindowsPowerShell\Modules


ModuleType Version    Name                                ExportedCommands                                                                                                                              
---------- -------    ----                                ----------------                                                                                                                              
Script     1.2.1      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave...}                                   
Script     1.0.1      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOperation, Get-AzKeyVaultCertificateOperation...}         
Script     1.1.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment...}                                             
Script     1.1.0      Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryptionActivity, Set-AzSqlDatabaseTransparentDataEncryption...
Script     1.0.2      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey...}                                             
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}                                                                                         
Script     1.1.7.0    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}                                                                        
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}                                                                        
Script     3.4.0      Pester                              {Describe, Context, It, Should...}                                                                                                            
Script     1.6.0      PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}                                                                                  
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}                                                                                  
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Set-PSReadlineKeyHandler, Remove-PSReadlineKeyHandler, Get-PSReadlineOption...}                                    
Script     21.0.17279 SqlServer                           {Add-RoleMember, Add-SqlAvailabilityDatabase, Add-SqlAvailabilityGroupListenerStaticIp, Add-SqlAzureAuthenticationContext...}                 
Manifest   2.0.0.0    xSmbShare                                                                                                                                                                         
Manifest   1.19.0.0   xWebAdministration                                                                                                                                                                


    Directory: C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules


ModuleType Version    Name                                ExportedCommands                                                                                                                              
---------- -------    ----                                ----------------                                                                                                                              
Manifest   1.0.0.0    AppBackgroundTask                   {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgroundTaskDiagnosticLog, Set-AppBackgroundTaskResourcePolicy, Unregister-AppBackgrou...
Manifest   2.0.0.0    AppLocker                           {Get-AppLockerFileInformation, Get-AppLockerPolicy, New-AppLockerPolicy, Set-AppLockerPolicy...}                                              
Manifest   1.0.0.0    AppvClient                          {Add-AppvClientConnectionGroup, Add-AppvClientPackage, Add-AppvPublishingServer, Disable-Appv...}                                             
Manifest   2.0.0.0    Appx                                {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest, Remove-AppxPackage...}                                                            
Script     1.0.0.0    AssignedAccess                      {Clear-AssignedAccess, Get-AssignedAccess, Set-AssignedAccess}                                                                                
Manifest   1.0.0.0    BitLocker                           {Unlock-BitLocker, Suspend-BitLocker, Resume-BitLocker, Remove-BitLockerKeyProtector...}                                                      
Manifest   2.0.0.0    BitsTransfer                        {Add-BitsFile, Complete-BitsTransfer, Get-BitsTransfer, Remove-BitsTransfer...}                                                               
Manifest   1.0.0.0    BranchCache                         {Add-BCDataCacheExtension, Clear-BCCache, Disable-BC, Disable-BCDowngrading...}                                                               
Manifest   1.0.0.0    CimCmdlets                          {Get-CimAssociatedInstance, Get-CimClass, Get-CimInstance, Get-CimSession...}                                                                 
Manifest   1.0        ConfigCI                            {Get-SystemDriver, New-CIPolicyRule, New-CIPolicy, Get-CIPolicy...}                                                                           
Binary     1.0.0.0    Containers                          {Add-ContainerNetworkAdapter, Add-ContainerNetworkAdapterStaticMapping, Get-ContainerNetworkAdapterStaticMapping, Remove-ContainerNetworkAd...
Manifest   1.0        Defender                            {Get-MpPreference, Set-MpPreference, Add-MpPreference, Remove-MpPreference...}                                                                
Manifest   1.0.0.0    DirectAccessClientComponents        {Disable-DAManualEntryPointSelection, Enable-DAManualEntryPointSelection, Get-DAClientExperienceConfiguration, Get-DAEntryPointTableItem...}  
Script     3.0        Dism                                {Add-AppxProvisionedPackage, Add-WindowsDriver, Add-WindowsCapability, Add-WindowsImage...}                                                   
Manifest   1.0.0.0    DnsClient                           {Resolve-DnsName, Clear-DnsClientCache, Get-DnsClient, Get-DnsClientCache...}                                                                 
Manifest   1.0.0.0    EventTracingManagement              {New-EtwTraceSession, Get-EtwTraceSession, Set-EtwTraceSession, Send-EtwTraceSession...}                                                      
Manifest   1.0.0.0    HgsClient                           {Get-HgsAttestationBaselinePolicy, Get-HgsClientConfiguration, Set-HgsClientConfiguration, New-HgsGuardian...}                                
Manifest   1.0.0.0    HgsDiagnostics                      {New-HgsTraceTarget, Get-HgsTrace, Get-HgsTraceFileData, Test-HgsTraceTarget}                                                                 
Binary     1.0.0.0    HostComputeService                  {Get-ComputeProcess, Stop-ComputeProcess}                                                                                                     
Binary     2.0.0.0    Hyper-V                             {Add-VMAssignableDevice, Add-VMDvdDrive, Add-VMFibreChannelHba, Add-VMGpuPartitionAdapter...}                                                 
Binary     1.1        Hyper-V                             {Add-VMDvdDrive, Add-VMFibreChannelHba, Add-VMHardDiskDrive, Add-VMMigrationNetwork...}                                                       
Script     1.0.0.0    IISAdministration                   {Get-IISAppPool, Start-IISCommitDelay, Stop-IISCommitDelay, Get-IISSite...}                                                                   
Manifest   2.0.0.0    International                       {Get-WinDefaultInputMethodOverride, Set-WinDefaultInputMethodOverride, Get-WinHomeLocation, Set-WinHomeLocation...}                           
Manifest   1.0.0.0    iSCSI                               {Get-IscsiTargetPortal, New-IscsiTargetPortal, Remove-IscsiTargetPortal, Update-IscsiTargetPortal...}                                         
Script     1.0.0.0    ISE                                 {New-IseSnippet, Import-IseSnippet, Get-IseSnippet}                                                                                           
Manifest   1.0.0.0    Kds                                 {Add-KdsRootKey, Get-KdsRootKey, Test-KdsRootKey, Set-KdsConfiguration...}                                                                    
Manifest   1.0.1.0    Microsoft.PowerShell.Archive        {Compress-Archive, Expand-Archive}                                                                                                            
Manifest   3.0.0.0    Microsoft.PowerShell.Diagnostics    {Get-WinEvent, Get-Counter, Import-Counter, Export-Counter...}                                                                                
Manifest   3.0.0.0    Microsoft.PowerShell.Host           {Start-Transcript, Stop-Transcript}                                                                                                           
Manifest   1.0.0.0    Microsoft.PowerShell.LocalAccounts  {Add-LocalGroupMember, Disable-LocalUser, Enable-LocalUser, Get-LocalGroup...}                                                                
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path...}                                                                                
Script     1.0        Microsoft.PowerShell.ODataUtils     Export-ODataEndpointProxy                                                                                                                     
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {Get-Acl, Set-Acl, Get-PfxCertificate, Get-Credential...}                                                                                     
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Format-List, Format-Custom, Format-Table, Format-Wide...}                                                                                    
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Disable-WSManCredSSP, Enable-WSManCredSSP, Get-WSManCredSSP, Set-WSManQuickConfig...}                                                        
Manifest   1.0        MMAgent                             {Disable-MMAgent, Enable-MMAgent, Set-MMAgent, Get-MMAgent...}                                                                                
Manifest   1.0.0.0    MsDtc                               {New-DtcDiagnosticTransaction, Complete-DtcDiagnosticTransaction, Join-DtcDiagnosticResourceManager, Receive-DtcDiagnosticTransaction...}     
Binary     1.0.0.0    MSMQ                                {Clear-MsmqOutgoingQueue, Clear-MsmqQueue, Enable-MsmqCertificate, Get-MsmqCertificate...}                                                    
Manifest   2.0.0.0    NetAdapter                          {Disable-NetAdapter, Disable-NetAdapterBinding, Disable-NetAdapterChecksumOffload, Disable-NetAdapterEncapsulatedPacketTaskOffload...}        
Manifest   1.0.0.0    NetConnection                       {Get-NetConnectionProfile, Set-NetConnectionProfile}                                                                                          
Manifest   1.0.0.0    NetEventPacketCapture               {New-NetEventSession, Remove-NetEventSession, Get-NetEventSession, Set-NetEventSession...}                                                    
Manifest   2.0.0.0    NetLbfo                             {Add-NetLbfoTeamMember, Add-NetLbfoTeamNic, Get-NetLbfoTeam, Get-NetLbfoTeamMember...}                                                        
Manifest   1.0.0.0    NetNat                              {Get-NetNat, Get-NetNatExternalAddress, Get-NetNatStaticMapping, Get-NetNatSession...}                                                        
Manifest   2.0.0.0    NetQos                              {Get-NetQosPolicy, Set-NetQosPolicy, Remove-NetQosPolicy, New-NetQosPolicy}                                                                   
Manifest   2.0.0.0    NetSecurity                         {Get-DAPolicyChange, New-NetIPsecAuthProposal, New-NetIPsecMainModeCryptoProposal, New-NetIPsecQuickModeCryptoProposal...}                    
Manifest   1.0.0.0    NetSwitchTeam                       {New-NetSwitchTeam, Remove-NetSwitchTeam, Get-NetSwitchTeam, Rename-NetSwitchTeam...}                                                         
Manifest   1.0.0.0    NetTCPIP                            {Get-NetIPAddress, Get-NetIPInterface, Get-NetIPv4Protocol, Get-NetIPv6Protocol...}                                                           
Manifest   1.0.0.0    NetWNV                              {Get-NetVirtualizationProviderAddress, Get-NetVirtualizationGlobal, Get-NetVirtualizationLookupRecord, Get-NetVirtualizationCustomerRoute...} 
Manifest   1.0.0.0    NetworkConnectivityStatus           {Get-DAConnectionStatus, Get-NCSIPolicyConfiguration, Reset-NCSIPolicyConfiguration, Set-NCSIPolicyConfiguration}                             
Manifest   1.0.0.0    NetworkSwitchManager                {Disable-NetworkSwitchEthernetPort, Enable-NetworkSwitchEthernetPort, Get-NetworkSwitchEthernetPort, Remove-NetworkSwitchEthernetPortIPAddr...
Manifest   1.0.0.0    NetworkTransition                   {Add-NetIPHttpsCertBinding, Disable-NetDnsTransitionConfiguration, Disable-NetIPHttpsProfile, Disable-NetNatTransitionConfiguration...}       
Manifest   1.0.0.0    PcsvDevice                          {Get-PcsvDevice, Start-PcsvDevice, Stop-PcsvDevice, Restart-PcsvDevice...}                                                                    
Manifest   1.0.0.0    PKI                                 {Add-CertificateEnrollmentPolicyServer, Export-Certificate, Export-PfxCertificate, Get-CertificateAutoEnrollmentPolicy...}                    
Manifest   1.0.0.0    PnpDevice                           {Get-PnpDevice, Get-PnpDeviceProperty, Enable-PnpDevice, Disable-PnpDevice}                                                                   
Manifest   1.1        PrintManagement                     {Add-Printer, Add-PrinterDriver, Add-PrinterPort, Get-PrintConfiguration...}                                                                  
Manifest   1.1        PSDesiredStateConfiguration         {Set-DscLocalConfigurationManager, Start-DscConfiguration, Test-DscConfiguration, Publish-DscConfiguration...}                                
Script     1.0.0.0    PSDiagnostics                       {Disable-PSTrace, Disable-PSWSManCombinedTrace, Disable-WSManTrace, Enable-PSTrace...}                                                        
Binary     1.1.0.0    PSScheduledJob                      {New-JobTrigger, Add-JobTrigger, Remove-JobTrigger, Get-JobTrigger...}                                                                        
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}                                                                                  
Manifest   1.0.0.0    PSWorkflowUtility                   Invoke-AsWorkflow                                                                                                                             
Manifest   1.0.0.0    ScheduledTasks                      {Get-ScheduledTask, Set-ScheduledTask, Register-ScheduledTask, Unregister-ScheduledTask...}                                                   
Manifest   2.0.0.0    SecureBoot                          {Confirm-SecureBootUEFI, Set-SecureBootUEFI, Get-SecureBootUEFI, Format-SecureBootUEFI...}                                                    
Manifest   2.0.0.0    SmbShare                            {Get-SmbShare, Remove-SmbShare, Set-SmbShare, Block-SmbShareAccess...}                                                                        
Manifest   2.0.0.0    SmbWitness                          {Get-SmbWitnessClient, Move-SmbWitnessClient, gsmbw, msmbw...}                                                                                
Manifest   1.0.0.0    StartLayout                         {Export-StartLayout, Import-StartLayout, Get-StartApps}                                                                                       
Manifest   2.0.0.0    Storage                             {Add-InitiatorIdToMaskingSet, Add-PartitionAccessPath, Add-PhysicalDisk, Add-TargetPortToMaskingSet...}                                       
Manifest   2.0.0.0    TLS                                 {New-TlsSessionTicketKey, Enable-TlsSessionTicketKey, Disable-TlsSessionTicketKey, Export-TlsSessionTicketKey...}                             
Manifest   1.0.0.0    TroubleshootingPack                 {Get-TroubleshootingPack, Invoke-TroubleshootingPack}                                                                                         
Manifest   2.0.0.0    TrustedPlatformModule               {Get-Tpm, Initialize-Tpm, Clear-Tpm, Unblock-Tpm...}                                                                                          
Binary     2.1.639.0  UEV                                 {Clear-UevConfiguration, Clear-UevAppxPackage, Restore-UevBackup, Set-UevTemplateProfile...}                                                  
Manifest   2.0.0.0    VpnClient                           {Add-VpnConnection, Set-VpnConnection, Remove-VpnConnection, Get-VpnConnection...}                                                            
Manifest   1.0.0.0    Wdac                                {Get-OdbcDriver, Set-OdbcDriver, Get-OdbcDsn, Add-OdbcDsn...}                                                                                 
Manifest   1.0.0.0    WebAdministration                   {Start-WebCommitDelay, Stop-WebCommitDelay, Get-WebConfigurationLock, Remove-WebConfigurationLock...}                                         
Manifest   1.0.0.0    WindowsDeveloperLicense             {Get-WindowsDeveloperLicense, Unregister-WindowsDeveloperLicense, Show-WindowsDeveloperLicenseRegistration}                                   
Script     1.0        WindowsErrorReporting               {Enable-WindowsErrorReporting, Disable-WindowsErrorReporting, Get-WindowsErrorReporting}                                                      
Manifest   1.0.0.0    WindowsSearch                       {Get-WindowsSearchSetting, Set-WindowsSearchSetting}                                                                                          
Manifest   1.0.0.0    WindowsUpdate                       Get-WindowsUpdateLog                                                                                                                          


    Directory: C:\Program Files (x86)\Microsoft SQL Server\120\Tools\PowerShell\Modules


ModuleType Version    Name                                ExportedCommands                                                                                                                              
---------- -------    ----                                ----------------                                                                                                                              
Manifest   1.0        SQLASCMDLETS                        {Add-RoleMember, Backup-ASDatabase, Invoke-ASCmd, Invoke-ProcessCube...}                                                                      
Manifest   1.0        SQLPS                               {Backup-SqlDatabase, Add-SqlAvailabilityDatabase, Add-SqlAvailabilityGroupListenerStaticIp, Disable-SqlAlwaysOn...}                           


    Directory: C:\Program Files (x86)\Microsoft SQL Server\140\Tools\PowerShell\Modules


ModuleType Version    Name                                ExportedCommands                                                                                                                              
---------- -------    ----                                ----------------                                                                                                                              
Manifest   14.0       SQLPS                               {Backup-SqlDatabase, Save-SqlMigrationReport, Add-SqlAvailabilityDatabase, Add-SqlAvailabilityGroupListenerStaticIp...}                       

Environment Data

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.14393.2189
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.2189
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Debug Output

PS C:\dev\pq\service\scripts\azure\NestedTemplates> Set-AzStorageBlobContent -Container "florin-container" -Context $storageAccount.Context -File "ApplicationInsights.json"
DEBUG: 1:53:54 PM - Init Operation Context for 'SetAzureBlobContentCommand' with client request id  Azure-Storage-PowerShell-8174d299-b07b-4773-937c-c4a7e1a715c1. If you want to get more details,
please add "-Debug" to your command.
DEBUG: 1:53:54 PM - SetAzureBlobContentCommand begin processing with ParameterSet 'SendManual'.
DEBUG: 1:53:54 PM - Use storage account 'dhicockdeployqsa' from storage context.
Set-AzStorageBlobContent : Failed to open file C:\dev\pq\service\scripts\azure\NestedTemplates\ApplicationInsights.json: Illegal characters in path..
At line:1 char:1
+ Set-AzStorageBlobContent -Container "florin-container" -Context $stor ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Set-AzStorageBlobContent], TransferException
    + FullyQualifiedErrorId : TransferException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

DEBUG: 1:53:54 PM - SetAzureBlobContentCommand end processing, Start 28 remote calls. Finish 28 remote calls. Elapsed time 5829207.84 ms. Client operation id:
Azure-Storage-PowerShell-8174d299-b07b-4773-937c-c4a7e1a715c1.
DEBUG: 1:53:54 PM - SetAzureBlobContentCommand end processing.
PS C:\dev\pq\service\scripts\azure\NestedTemplates>

[blueww]

@fmicle
I can't repro the issue with the same module version, and same file path on my machine. (See following result on my machine).

Would you like to check do you have any invisible special char in the path?

PS C:\dev\pq\service\scripts\azure\NestedTemplates> Set-AzStorageBlobContent -Container "florin-container" -Context $ctx -File "ApplicationInsights.json"


   Container Uri: https://weistagelarge.blob.core.windows.net/florin-container

Name                 BlobType  Length          ContentType                    LastModified         AccessTier SnapshotTime         IsDeleted 
----                 --------  ------          -----------                    ------------         ---------- ------------         --------- 
ApplicationInsigh... BlockBlob 492476          application/octet-stream       2019-02-11 08:12:49Z Hot                             False     



PS C:\dev\pq\service\scripts\azure\NestedTemplates> get-module

ModuleType Version    Name                                ExportedCommands                                                                                                                        
---------- -------    ----                                ----------------                                                                                                                        
Script     1.2.1      Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount...}                                                             
Script     1.0.2      Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountNetworkRule, Disable-AzStorageDeleteRetentionPolicy, Disable-AzStorageStaticW...
Script     1.0.0.0    ISE                                 {Get-IseSnippet, Import-IseSnippet, New-IseSnippet}                                                                                     
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}                                                                      
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {ConvertFrom-SecureString, ConvertTo-SecureString, Get-Acl, Get-AuthenticodeSignature...}                                               
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}                                                                               
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Connect-WSMan, Disable-WSManCredSSP, Disconnect-WSMan, Enable-WSManCredSSP...}   

[fmicle]

I just tried again today, I removed the AzureRM modules, installed Az modules, executed Enable-AzureRmAlias and without making any other changes to my scripts I am seeing the same problem:

Set-AzureStorageBlobContent : Failed to open file C:\dev\pq\service\scripts\azure\NestedTemplates\ApplicationInsights.json: Illegal characters in path..
At C:\dev\pq\service\scripts\pq-deployment.psm1:838 char:14
+ $output = Set-AzureStorageBlobContent `
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Set-AzStorageBlobContent], TransferException
+ FullyQualifiedErrorId : TransferException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

There are no invisible characters in the path, since everything works as expected when using the AzureRM scripts...

[fmicle]

I created a very simple test case scenario, outside of our bigger script suite and can very easily recreate the problem. I put a file called test.json in c:, it contains an empty json object, just the curly braces, as you can see in the output below. I am using the AzureRM Alias scenario:

PS C:\> type test.json
{}
PS C:\> type C:\test.json
{}
PS C:\> Enable-AzureRmAlias
PS C:\> $sa = Get-AzureRmStorageAccount -ResourceGroupName florin-rg -Name florinsa
PS C:\> Set-AzureStorageBlobContent -Container florin-container -Context $sa.Context -File test.json -Blob test
Set-AzureStorageBlobContent : Failed to open file C:\test.json: Illegal characters in path..
At line:1 char:1
+ Set-AzureStorageBlobContent -Container florin-container -Context $sa. ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : CloseError: (:) [Set-AzStorageBlobContent], TransferException
+ FullyQualifiedErrorId : TransferException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

PS C:\>get-module

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     1.3.0      Az.Accounts                         {Add-AzEnvironment, Clear-AzCo
Script     1.0.2      Az.Storage                          {Add-AzRmStorageContainerLegal
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Ch
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {ConvertFrom-SecureString, Con
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-V
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Connect-WSMan, Disable-WSManC
Script     1.1.7.0    PackageManagement                   {Find-Package, Find-PackagePro
Script     1.6.0      PowerShellGet                       {Find-Command, Find-DscResourc
Script     0.0        pq-deployment                       {PQ-BootstrapOps, PQ-Build, PQ
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Get

[mprabhu11]

We are facing this issue from a PS automation context. What appear to be straightforward paths which used to work with the equivalent AzureRm cmdlet in the past are failing with the above error after changing to Set-AzStorage version.

[blueww]

@fmicle

I can't repro the issue on my machine, I tried 2 machines.
Could you help to open a new PowerShell console, repro the issue and collect more information by run following command just after Set-AzStorageBlobContent fail, and show me the result (hide if any credntial)

$error[0].Exception.ToString()

[environment]::OSVersion

Get-Childitem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full'

BTW, does this issue always happen when you upload any file with Set-AzStorageBlobContent, or only on specific files? From your comments, it looks like always repro?

[fmicle]

Yes, I can repro consistently on first attempt, and it happens with any file I try.

PS C:\> Set-AzureStorageBlobContent -Container florin-container -Context $sa.Context -File test.json -Blob test
Set-AzureStorageBlobContent : Failed to open file C:\test.json: Illegal characters in path..
At line:1 char:1
+ Set-AzureStorageBlobContent -Container florin-container -Context $sa. ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Set-AzStorageBlobContent], TransferException
    + FullyQualifiedErrorId : TransferException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

PS C:\> $error[0].Exception.ToString()
Microsoft.WindowsAzure.Storage.DataMovement.TransferException: Failed to open file C:\test.json: Illegal characters in path.. ---> System.ArgumentException: Illegal characters in path.
   at System.Security.Permissions.FileIOPermission.EmulateFileIOPermissionChecks(String fullPath)
   at System.Security.Permissions.FileIOPermission.QuickDemand(FileIOPermissionAccess access, String fullPath, Boolean checkForDuplicates, Boolean needFullPath)
   at Microsoft.WindowsAzure.Storage.DataMovement.LongPath.ToUncPath(String path)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.StreamedReader.<OpenInputStreamAsync>d__18.MoveNext()
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.StreamedReader.<OpenInputStreamAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.StreamedReader.<DoWorkInternalAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.SyncTransferController.<DoWorkInternalAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferControllers.TransferControllerBase.<DoWorkAsync>d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferScheduler.<ExecuteJobInternalAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.SingleObjectTransfer.<ExecuteAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Storage.DataMovement.TransferManager.<DoTransfer>d__72.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Commands.Storage.Common.DataMovementTransferHelper.<DoTransfer>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand.<Upload2Blob>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.WindowsAzure.Commands.Storage.Common.LimitedConcurrencyTaskScheduler.<RunConcurrentTask>d__26.MoveNext()
PS C:\> [environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.14393.0 Microsoft Windows NT 10.0.14393.0


PS C:\> Get-Childitem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full'


    Hive: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full


Name                           Property
----                           --------
1033                           CBS           : 1
                               Install       : 1
                               Release       : 461814
                               Servicing     : 0
                               TargetVersion : 4.0.0
                               Version       : 4.7.03062


PS C:\>

[blueww]

@fmicle

I see another customer which repro the issue in runbook: #8531
Just to confirm do you repro the issue in runbook or in local machine

We have repro the same issue in runbook, and find it's caused by the difference behavior of API [System.IO.Path]::GetFullPath() on different machine.

Would you please check if the following command will report error on your machine?
[System.IO.Path]::GetFullPath("\\?\c:\test.csv")

In DMlib (Powershell Depends on DMlib for blob upload/download), for support long file path, DMlib will convert local file path to UNC path format, and then do transfer. But on your machine, it seems the API not support UNC path, but on my machine, it support UNC path. I will contact netcore team to check the API issue. Will update you if get any result from them.

On my local machine (not repro the issue), the API works correctly:

PS C:\WINDOWS\system32> [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
\\?\c:\test.csv

On Runbook (which repro the issue), the API report error:
Script:
[System.IO.Path]::GetFullPath("\\?\c:\test.csv")
Result:

Exception calling "GetFullPath" with "1" argument(s): "Illegal characters in path."
At line:2 char:1
+ [System.IO.Path]::GetFullPath("\\?\c:\test.csv")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentException

[fmicle]

Yes, the error occurs on my local machine:

PS C:\> [System.IO.Path]::GetFullPath("\\?\c:\test.json")
Exception calling "GetFullPath" with "1" argument(s): "Illegal characters in path."
At line:1 char:1
+ [System.IO.Path]::GetFullPath("\\?\c:\test.json")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentException

PS C:\>

[mprabhu11]

@fmicle

The issue is now narrowed down to 'legacy path handing' settings in .Net.

For me the issue repros from a PS runbook using Az-* but does not repro from my desktop (using the same code).

Inspecting the relevant setting in the two environments gives me this:

$x = "blah"
$setting = [ref]$x
$z = [System.AppContext]::TryGetSwitch("Switch.System.IO.UseLegacyPathHandling", [System.Management.Automation.PSReference] $setting);
Write-Output("Setting Value: " + $setting.Value)

Output from Runbook:
Setting Value: True

Output on Desktop:
Setting Value: False

[blueww]

@fmicle
As we get from netcore team:

It's most likely the switch that enables device syntax isn’t set. While .NET 4.6.2+ supports the extended syntax, PowerShell (not Core) didn’t opt into it until the Windows 10 Anniversary update.

While PowerShell hosts the CLR it does specify the config file when creating the AppDomain. It will pass C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe.config. You can explicitly set the flags the “normal” way by: https://blogs.msdn.microsoft.com/jeremykuhne/2016/06/09/new-net-path-handling-sneak-peek/

I have tested:

1. Change the "Switch.System.IO.UseLegacyPathHandling" at begin of a PSH session not work in my test. It seems the setting is cached when the session load, so change it actually not change the value in use.

2. I also tried to change the powershell.exe.config, but seems the file can't be changes in my machine even with administrator.

So you might need to make sure:

1. Your OS is Windows 10 Anniversary or later
2. If not, you need follow https://blogs.msdn.microsoft.com/jeremykuhne/2016/06/09/new-net-path-handling-sneak-peek/ to change the powershell.exe.config.

Let me know if you get any update on this.

@mprabhu11
There's a typo in the script from you in the above comment, I have fixed it.

[fmicle]

For some reason the Windows 10 Anniversary update has been failing to install on my laptop since the day it was published. I tried a few tricks, but nothing worked, so I guess I need to take my laptop to IT, since my domain policy doesn't allow me to install updates manually...

I also tried the powershell.exe.config route and I can't get the setting to change...

[blueww]

@fmicle
Thanks for the update! Let me know if you need more assistant from us on this.

[ronhowe]

I'm still having issues with this.

Here are the contents of my powershell.exe.config.

<configuration>
    <runtime>
        <AppContextSwitchOverrides value="Switch.System.IO.UseLegacyPathHandling=false" />
    </runtime>
    <uri>
        <schemeSettings>
            <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/>
            <add name="https" genericUriParserOptions="DontUnescapePathDotsAndSlashes"/>
        </schemeSettings>
    </uri>
</configuration>

I run the following as a test:

$x = "blah"
$setting = [ref]$x
$z = [System.AppContext]::TryGetSwitch("Switch.System.IO.UseLegacyPathHandling", [System.Management.Automation.PSReference] $setting);
Write-Output("Setting Value: " + $setting.Value)

When I run that while RDP'd into my Azure Automation Hybrid Worker, I get $false.

When I run that remotely on that Azure Automation Hybrid Worker via a Runbook, I get $true.

[blueww]

@ronhowe
I am not sure how can you change the powershell.exe.config of runbook.
Automation team is trying to fix this in runbook, and before that is fixed, the Set/Get-AzStorageBlobContent should not work in Runbook.

You might can use the old cmdlet in Azure.Storage module as a workaround.

[blueww]

@ronhowe , @fmicle , @mprabhu11

The fix for cloud Azure Automation jobs is rolled out to all regions.
Hybrid jobs need to be handled separately. Automation team is working on getting the timeline for this. In the meantime, the temporary workaround idea: #8531 (comment)

And the automation runbook is fixed, I plan to close the issue. Let me know if you have any concent.
Feel free to contact us if you need any more assistant from us.

[npalmer1452]

This still is not working for me. How do you fix with Get-AzStorageBlobontent? I removed the powershell.exe.config file as stated in https://blogs.msdn.microsoft.com/jeremykuhne/2016/06/09/new-net-path-handling-sneak-peek/. Perhaps i am missing something but when i run:
PS> $x = "false"
PS> $setting = [ref]$x
PSs> [System.AppContext]::TryGetSwitch("Switch.System.IO.UseLegacyPathHandling", [System.Management.Automation.PSReference] $setting)
True

so ho do i change that to false? this is for Desktop

[fmicle]

The script above returns true for me as well, but the actual problem got fixed for me a while ago. Try creating this registry key:

$registryPath = "HKLM:\SOFTWARE\Microsoft.NETFramework\AppContext"
New-Item -Path $registryPath
New-ItemProperty -Path $registryPath -Name "Switch.System.IO.UseLegacyPathHandling" -Value "false"

[suman5488]

Hi All,

How to fix this illegal path issue? I am facing this issue when I trying to call Get-AzureStorageBlobContent. Please suggest to me. Recently I have uninstalled AzureRm and installed Az Module with Enable AzureRM Alias. Thanks

[karthick3d]

Bit frustrated after spending almost a day to get this working in hybrid Worker. I tried all the options mentioned here, i.e. update the registry and updated the powershell.exe.config file as well. none of this worked for me. Raised a formal ticket to Microsoft.
----------------- Update After some time--------------------
I went through the Event log ,I found that we have to create Orchestrator.Sandbox.exe.config which worked. I also found this in another link as well.