Import-AzDataLakeStoreItem can't find System.Buffers

[lotsahelp]

Description

Calling Import-AzDataLakeStoreItem v1.2.4 gives this error: Could not load file or assembly 'System.Buffers, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies.

Steps to reproduce

    Import-AzDataLakeStoreItem -AccountName $DataLakeStoreName `
          -Path $LocalOutputPath `
          -Destination $AdlsPath

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.14393.3053
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.3053
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands                                                                                                                                            
---------- -------    ----                                ----------------                                                                                                                                            
Script     1.6.4      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave...}                                                 
Script     1.1.0      Az.Advisor                          {Get-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Disable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration...}                               
Script     1.0.2      Az.Aks                              {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential...}                                                                                             
Script     1.1.1      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer...}                       
Script     1.3.2      Az.ApiManagement                    {Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiManagementRegion, Add-AzApiManagementUserToGroup...}                          
Script     1.0.2      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsights, Set-AzApplicationInsightsPricingPlan...}                               
Script     1.3.4      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-AzAutomationJobOutputRecord, Import-AzAutomationDscNodeConfiguration...}      
Script     2.0.1      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBatchAccount...}                                                                   
Script     1.0.1      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsumptionBudget...}                                                            
Script     1.4.0      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...}                                                                        
Script     1.2.1      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiveServicesAccountSku, Get-AzCognitiveServicesAccountType...}               
Script     3.0.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-AzAvailabilitySet...}                                                       
Script     1.0.1      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzContainerInstanceLog}                                                           
Script     1.1.0      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, Remove-AzContainerRegistry...}                                               
Script     1.4.1      Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFactoryV2...}                                                               
Script     1.0.1      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remove-AzDataLakeAnalyticsCatalogCredential, Set-AzDataLakeAnalyticsCatalog...
Script     1.2.4      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreFirewallRule, Set-AzDataLakeStoreTrustedIdProvider...}
Script     1.0.1      Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSource, Set-AzDeploymentManagerArtifactSource, Remove-AzDeploymentManagerArtifactS...
Script     1.0.0      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy...}                                         
Script     1.1.1      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecordSet...}                                                                   
Script     1.2.2      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGridTopicKey...}                                                              
Script     1.4.0      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remove-AzEventHubNamespace...}                                                  
Script     1.1.2      Az.FrontDoor                        {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor...}                                                                                  
Script     3.0.0      Az.HDInsight                        {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-AzHDInsightStreamingMapReduceJobDefinition...}                             
Script     1.0.0      Az.HealthcareApis                   {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthcareApisService, Get-AzHealthcareApisService}                                     
Script     2.0.0      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString, Get-AzIotHubJob...}                                                      
Script     1.3.1      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOperation, Get-AzKeyVaultCertificateOperation...}                       
Script     1.3.1      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration, Get-AzIntegrationAccountCallbackUrl...}   
Script     1.1.1      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlanUsageHistory, Remove-AzMlCommitmentPlan...}                           
Script     1.0.1      Az.ManagedServices                  {Get-AzManagedServicesAssignment, New-AzManagedServicesAssignment, Remove-AzManagedServicesAssignment, Get-AzManagedServicesDefinition...}                  
Script     1.0.1      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}                                                                                                            
Script     1.1.0      Az.Media                            {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability...}                                        
Script     1.4.0      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...}                                                                            
Script     2.0.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthenticationCertificate, New-AzApplicationGatewayAuthenticationCertificate,...
Script     1.1.0      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHubPNSCredential...}                        
Script     1.3.3      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCustomLogDataSource, Disable-AzOperationalInsightsLinuxCustomLogCollection...
Script     1.1.3      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemediation...}                                                                
Script     1.1.0      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKey, Get-AzPowerBIWorkspace...}               
Script     1.0.0      Az.PrivateDns                       {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPrivateDnsZone, New-AzPrivateDnsZone...}                                                              
Script     2.0.1      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecoveryServicesVaultSettingsFile, New-AzRecoveryServicesVault...}                
Script     1.1.1      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCachePatchSchedule, New-AzRedisCachePatchSchedule...}                          
Script     1.0.2      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRelayNamespace...}                                                              
Script     1.7.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment...}                                                           
Script     1.4.0      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace, Remove-AzServiceBusNamespace...}                                          
Script     2.0.0      Az.ServiceFabric                    {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricClusterCertificate, Add-AzServiceFabricNode, Add-AzServiceFabricNodeType...}                      
Script     1.1.0      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...}                                                                                       
Script     2.0.0      Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryptionActivity, Set-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDat...
Script     1.9.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey...}                                                           
Script     1.2.1      Az.StorageSync                      {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorageSyncService, Remove-AzStorageSyncService...}                                
Script     1.0.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, New-AzStreamAnalyticsFunction, Remove-AzStreamAnalyticsFunction...}         
Script     1.0.2      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFromEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Remove-AzTraffic...
Script     1.5.0      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppServicePlan...}     

Debug output

DEBUG: 8:41:58 PM - ImportAzureDataLakeStoreItem begin processing with ParameterSet 'NoDiagnosticLogging'.
DEBUG: 8:41:58 PM - using account id '{redacted}'...
FailedTransfer: {Destination Redacted} Could not load file or assembly 'System.Buffers, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
DEBUG: 2019-11-07 20:41:58.9582|DEBUG|adls.dotnet.FileTransfer|FileTransfer.Uploader, Src: {Source Redacted}, Dest: {Destination Redacted}, Threads: 128, TrackingProgress: True, OverwriteIfExist: False
DEBUG: 2019-11-07 20:41:58.9738|DEBUG|adls.dotnet.FileTransfer|FileTransfer.FileProduced, Name: {Source Redacted}, Dest: {Destination Redacted}, Length: 547918, Chunks: 0
DEBUG: 2019-11-07 20:41:58.9894|DEBUG|adls.dotnet.FileTransfer.Job|FileTransfer.CopyFile, Source: {Source Redacted}, Dest: {Destination Redacted}, Offset: 0, Length: 547918, Lat: 1, JobStatus: Failed, Error: Could not load file or assembly 'System.Buffers, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
{Destination Redacted}
DEBUG: AzureQoSEvent: CommandName - Import-AzDataLakeStoreItem; IsSuccess - True; Duration - 00:00:00.3928530; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 8:41:59 PM - ImportAzureDataLakeStoreItem end processing.
DEBUG: 8:41:59 PM - ImportAzureDataLakeStoreItem end processing.

Error output

DEBUG: 8:43:50 PM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
DEBUG: 8:43:50 PM - using account id '{redacted}'...
WARNING: Breaking changes in the cmdlet 'Resolve-AzError' :
WARNING:  - The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.

WARNING: NOTE : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
DEBUG: AzureQoSEvent: CommandName - Resolve-AzError; IsSuccess - True; Duration - 00:00:00.0265474; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 8:43:51 PM - ResolveError end processing.
DEBUG: 8:43:51 PM - ResolveError end processing.

[lotsahelp]

Version 1.2.3 does not have this issue.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams

[jschwanz]

+1 getting a similar issue with this

[jschwanz]

I've also confirmed it is broken with 1.2.4, but works with 1.2.3.

[jacksilverton]

Same error here using Az.DataLakeStore 1.2.5.

[tjpetz]

Same error with Az.DataLakeStore 1.2.6. Other data lake commands seem to be working, just the Import-AzDataLakeStoreItem is causing me this problem.

[jacksilverton]

Can confirm working in 1.2.3.
Not working in 1.2.4 onward.

To remove a newer version and install 1.2.3 instead, run in admin mode:
Remove-Module -Name Az.DataLakeStore
Install-Module -Name Az.DataLakeStore -RequiredVersion 1.2.3

[tjpetz]

Thanks I gave this a try. At first it did not work. Then I realized that I had both 32 and 64 bit versions installed. Once I downgraded both the 32 and 64 bit versions to 1.2.3 then it worked!

Thanks for your help.

[helloyesthisisdave]

This is now also causing problems when trying to load files into a Data Lake via an Azure Powershell task in Azure DevOps