Copy-ServiceFabricApplicationPackage fails regularily when trying to run SF application on local cluster

[Sossenbinder]

I am developing a Service Fabric application locally, and I, seemingly randomly, tend to see the following error message when trying to start from VS2017:

    4>Copying application to image store...
    4>Copy-ServiceFabricApplicationPackage : Failed to copy 1 items while copying from source 
    4>E:\Programmieren\ASP.NET_Core\MooMed\MooMed\MooMed.Fabric\pkg\Debug to destination 
    4>C:\SfDevCluster\Data\ImageStoreShare\MooMed.FabricType
    4>At C:\Program Files\Microsoft SDKs\Service 
    4>Fabric\Tools\PSModule\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1:274 char:9
    4>+         Copy-ServiceFabricApplicationPackage -ApplicationPackagePath  ...
    4>+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    4>    + CategoryInfo          : InvalidData: (:) [Copy-ServiceFabricApplicationPackage], InvalidDataException
    4>    + FullyQualifiedErrorId : CopyApplicationPackageErrorId,Microsoft.ServiceFabric.Powershell.CopyApplicationPackage

Apparently something goes wrong here, but I don't really know what the problem is.

Running Get-ServiceFabricApplicationType through powershell shows me the following errorcode: FABRIC_E_IMAGEBUILDER_UNEXPECTED_ERROR. However, I can barely find any kind of documentation.

I can't really say what might be the cause of this as well, as it tends to appear here and there, but it often goes away after rebooting my computer / resetting the local cluster.

Is there any information you can provide me? Documentation is really scarce regarding what happens in the Publish-NewServiceFabricApplication.ps1

Edit: I was able to run it with -Verbose flags, this is what I got now:

    4>Copying application to image store...
    4>VERBOSE: System.IO.InvalidDataException: Failed to copy 1 items while copying from source 
    4>E:\Programmieren\ASP.NET_Core\MooMed\MooMed\MooMed.Fabric\pkg\Debug to destination 
    4>C:\SfDevCluster\Data\ImageStoreShare\MooMed.FabricType
    4>
    4>   at System.Fabric.ImageStore.FolderCopy.Copy(String source, String destination)
    4>
    4>   at System.Fabric.ImageStore.FileImageStore.CopyCallerHoldsReaderLock(String source, String destination, CopyFlag 
    4>copyFlag, TimeoutHelper helper)
    4>
    4>   at System.Fabric.ImageStore.FileImageStore.UploadContent(String storeDestination, String localSource, 
    4>IImageStoreProgressHandler handler, TimeSpan timeout, CopyFlag copyFlag, Boolean acquireSourceReaderLock)
    4>
    4>   at Microsoft.ServiceFabric.Powershell.CommonCmdletBase.UploadToImageStore(String imageStoreConnectionString, String 
    4>src, String des, UploadProgressHandler progressHandler)
    4>
    4>   at Microsoft.ServiceFabric.Powershell.CopyApplicationPackage.ProcessRecord()
    4>Copy-ServiceFabricApplicationPackage : Failed to copy 1 items while copying from source 
    4>E:\Programmieren\ASP.NET_Core\MooMed\MooMed\MooMed.Fabric\pkg\Debug to destination 
    4>C:\SfDevCluster\Data\ImageStoreShare\MooMed.FabricType
    4>At C:\Program Files\Microsoft SDKs\Service 
    4>Fabric\Tools\PSModule\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1:274 char:9
    4>+         Copy-ServiceFabricApplicationPackage -ApplicationPackagePath  ...
    4>+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    4>    + CategoryInfo          : InvalidData: (:) [Copy-ServiceFabricApplicationPackage], InvalidDataException
    4>    + FullyQualifiedErrorId : CopyApplicationPackageErrorId,Microsoft.ServiceFabric.Powershell.CopyApplicationPackage
    4> 
    4>Finished executing script 'Deploy-FabricApplication.ps1'.

During the script run I notices that the application is indeed copied from from my pkg/Debug to the ImageStoreShare, but then disappears again. I'm not sure if this is due to the error and clean up in the background or whether something else could have deleted this.

[Sossenbinder]

@MicahMckittrick-MSFT

This is the post I have now transferred from the MSDN forums to github :) Right now, it is happening 100% of the time again. This has been the case about a week ago, but back then I noticed that my app was running "Refresh application" on a 5 node cluster, I changed it to 1 node and this only occured like 10% of the time anymore. However, I don't know if there's a relation between these or if it's just a coincidence.

Right now I can't get to debug or deploy my app at all anymore. Resetting cluster is of no help. If you want, I can therefore reliably attach logs for you to check.

Logs_2019-31-05-08-34-53.zip

[Adebeer]

FYI We are also experiencing the above issue quite frequently. I suspect the issue is due to network reliability/dropouts between our Build agent (Azure DevOps build agent running in Azure VM) and our on-premise Service Fabric cluster. Note: Using the Azure DevOps task "Deploy Service Fabric Application". Cluster secured via gMSA. SF Cluster version: 6.4.637.9590

Here's an example of what I'm seeing - sometimes issue is fixed by retrying immediately and/or at a different time of day (hence reason I think it relates to network)

2019-06-05T07:00:06.8301819Z Remove application instance succeeded
2019-06-05T07:00:08.4339917Z Unregister application type succeeded.
2019-06-05T07:00:12.3133238Z ##[debug]Join-Path "C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code" "ServiceFabricServiceModel.xsd"
2019-06-05T07:00:12.3133535Z ##[debug]C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\ServiceFabricServiceModel.xsd
2019-06-05T07:00:12.3133773Z ##[debug]Length: 94
2019-06-05T07:00:12.3133911Z ##[debug]
2019-06-05T07:00:12.3134076Z ##[debug]Test-Path "C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\ServiceFabricServiceModel.xsd"
2019-06-05T07:00:12.3134219Z ##[debug]True
2019-06-05T07:00:12.3134363Z ##[debug]
2019-06-05T07:00:28.8660982Z ##[debug]System.IO.InvalidDataException: Failed to copy 1 items while copying from source C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\applicationpackage to destination C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\s2bt2g3t.avt\applicationpackage
2019-06-05T07:00:28.8678267Z ##[debug]   at System.Fabric.Management.ImageStore.FolderCopy.Copy(String source, String destination)
2019-06-05T07:00:28.8695706Z ##[debug]   at System.Fabric.Management.ImageStore.FileImageStore.CopyCallerHoldsReaderLock(String source, String destination, CopyFlag copyFlag, TimeoutHelper helper)
2019-06-05T07:00:28.8712406Z ##[debug]   at System.Fabric.Management.ImageStore.FileImageStore.DownloadContent(String storeSource, String localDestination, IImageStoreProgressHandler handler, TimeSpan timeout, CopyFlag copyFlag)
2019-06-05T07:00:28.8729700Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageStoreWrapper.DownloadIfContentExists(String source, String destination, IImageStoreProgressHandler progressHandler, TimeSpan timeout)
2019-06-05T07:00:28.8746943Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageBuilder.InnerBuildApplicationType(String buildPackagePath, TimeSpan timeout, String localIncomingApplicationFolder, String progressFile, Boolean validateOnly, Boolean shouldSkipChecksumValidation, Boolean disableServerSideCopy)
2019-06-05T07:00:28.8763811Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageBuilder.ValidateApplicationPackage(String buildPackagePath, IDictionary`2 userParameters)
2019-06-05T07:00:28.8781424Z ##[debug]   at Microsoft.ServiceFabric.Powershell.ApplicationCmdletBase.TestSFApplicationPackage(String applicationPackagePath, Hashtable applicationParameters, String imageStoreConnectionString)
2019-06-05T07:00:29.5274243Z ##[debug]SYSTEM_JOBID: '47b38596-d4b1-584d-6e68-c59e4c826921'
2019-06-05T07:00:29.5533762Z ##[debug]Processed: ##vso[telemetry.publish area=TaskHub;feature=ServiceFabricDeploy]{"OperationId":"TestApplicationPackage","ExceptionData":"System.IO.InvalidDataException;|C:\\vsts-agent\\_work\\_tasks\\ServiceFabricDeploy_XXXXX\\1.7.31\\ServiceFabricSDK\\Publish-NewServiceFabricApplication.ps1|243","JobId":"47b38596-d4b1-584d-6e68-c59e4c826921","SDKVersion":"3.0.456.9494"}
2019-06-05T07:00:29.5543906Z ##[debug]Leaving C:\vsts-agent\_work\_tasks\ServiceFabricDeploy_XXXXX\1.7.31\deploy.ps1.
2019-06-05T07:00:29.5554362Z ##[debug]Caught exception from task script.
2019-06-05T07:00:29.5663675Z ##[debug]Error record:
2019-06-05T07:00:29.6361007Z ##[debug]Test-ServiceFabricApplicationPackage : Failed to copy 1 items while copying from source C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\applicationpackage to destination C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\s2bt2g3t.avt\applicationpackage
2019-06-05T07:00:29.6380169Z ##[debug]At C:\vsts-agent\_work\_tasks\ServiceFabricDeploy_XXXXX\1.7.31\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1:243 char:46
2019-06-05T07:00:29.6398278Z ##[debug]+                 $packageValidationSuccess = (Test-ServiceFabricApplicationPackag ...
2019-06-05T07:00:29.6415042Z ##[debug]+                                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2019-06-05T07:00:29.6431617Z ##[debug]    + CategoryInfo          : InvalidData: (:) [Test-ServiceFabricApplicationPackage], InvalidDataException
2019-06-05T07:00:29.6448588Z ##[debug]    + FullyQualifiedErrorId : TestApplicationPackageErrorId,Microsoft.ServiceFabric.Powershell.TestApplicationPackage
2019-06-05T07:00:29.6465272Z ##[debug] 
2019-06-05T07:00:29.6498525Z ##[debug]Script stack trace:
2019-06-05T07:00:29.6677818Z ##[debug]at Publish-NewServiceFabricApplication, C:\vsts-agent\_work\_tasks\ServiceFabricDeploy_XXXXX\1.7.31\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1: line 243
2019-06-05T07:00:29.6694480Z ##[debug]at <ScriptBlock>, C:\vsts-agent\_work\_tasks\ServiceFabricDeploy_XXXXX\1.7.31\deploy.ps1: line 194
2019-06-05T07:00:29.6712096Z ##[debug]at <ScriptBlock>, <No file>: line 1
2019-06-05T07:00:29.6729960Z ##[debug]at <ScriptBlock>, <No file>: line 22
2019-06-05T07:00:29.6747031Z ##[debug]at <ScriptBlock>, <No file>: line 18
2019-06-05T07:00:29.6763968Z ##[debug]at <ScriptBlock>, <No file>: line 1
2019-06-05T07:00:29.6797634Z ##[debug]Exception:
2019-06-05T07:00:29.6868744Z ##[debug]System.IO.InvalidDataException: Failed to copy 1 items while copying from source C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\applicationpackage to destination C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\s2bt2g3t.avt\applicationpackage
2019-06-05T07:00:29.6885737Z ##[debug]   at System.Fabric.Management.ImageStore.FolderCopy.Copy(String source, String destination)
2019-06-05T07:00:29.6902430Z ##[debug]   at System.Fabric.Management.ImageStore.FileImageStore.CopyCallerHoldsReaderLock(String source, String destination, CopyFlag copyFlag, TimeoutHelper helper)
2019-06-05T07:00:29.6919064Z ##[debug]   at System.Fabric.Management.ImageStore.FileImageStore.DownloadContent(String storeSource, String localDestination, IImageStoreProgressHandler handler, TimeSpan timeout, CopyFlag copyFlag)
2019-06-05T07:00:29.6936127Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageStoreWrapper.DownloadIfContentExists(String source, String destination, IImageStoreProgressHandler progressHandler, TimeSpan timeout)
2019-06-05T07:00:29.6952804Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageBuilder.InnerBuildApplicationType(String buildPackagePath, TimeSpan timeout, String localIncomingApplicationFolder, String progressFile, Boolean validateOnly, Boolean shouldSkipChecksumValidation, Boolean disableServerSideCopy)
2019-06-05T07:00:29.6969846Z ##[debug]   at System.Fabric.Management.ImageBuilder.ImageBuilder.ValidateApplicationPackage(String buildPackagePath, IDictionary`2 userParameters)
2019-06-05T07:00:29.6986465Z ##[debug]   at Microsoft.ServiceFabric.Powershell.ApplicationCmdletBase.TestSFApplicationPackage(String applicationPackagePath, Hashtable applicationParameters, String imageStoreConnectionString)
2019-06-05T07:00:29.7383657Z ##[error]Failed to copy 1 items while copying from source C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\applicationpackage to destination C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\s2bt2g3t.avt\applicationpackage
2019-06-05T07:00:29.7394160Z ##[debug]Processed: ##vso[task.logissue type=error]Failed to copy 1 items while copying from source C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\applicationpackage to destination C:\XXXXX\AppData\Local\Temp\TestApplicationPackage_199094445097268\s2bt2g3t.avt\applicationpackage
2019-06-05T07:00:29.7395597Z ##[debug]Processed: ##vso[task.complete result=Failed]
2019-06-05T07:00:29.7802147Z ##[section]Finishing: Deploy Service Fabric Application (Disabled Upgrades and Diff Packages)

[Adebeer]

I've changed our Azure DevOps deployment task to SF PowerShell script and invoking copy package manually - and, like before, still PERIODICALLY getting same error above.

This is the command I'm invoking from Powershell:

Copy-ServiceFabricApplicationPackage -ApplicationPackagePath $ApplicationPackagePath -ApplicationPackagePathInImageStore $ImageStoreCopyPath -CompressPackage -ShowProgress -TimeoutSec $CopyPackageTimeoutSec

[sarajang]

I have the same issue as well on local cluster manager. Any update on this issue?

[iyyappam]

This looks like issue with local file copy. We have seen similar issues where anti-virus was holding the files in those folders causing these issues. Could you exclude the folders you are trying to copy and try it out.

[sunilbindra]

we are facing the same issue on on prem cluster. I am using Octopus deploy, and running powershell script to copy the package. Script is running on Target Machine. Package is valid and I am able to connect on the cluster, just while copying it fails. The script works fine if RDP on machine and run the script. Any Idea why this is happening, We are using GMSA (Windows Security).

Copy-ServiceFabricApplicationPackage -ApplicationPackagePath $packageSourcePath
-ApplicationPackagePathInImageStore $ImageStoreName_$appVersion -ImageStoreConnectionString $ImageConnectionString.Value
-TimeoutSec 2700 -Verbose

[amitaysh]

Hi,
Had the same issue in our team, eventually found a weird solution that worked for us all... just run the "Publish-NewServiceFabricApplication.ps1" PowerShell command manually as admin, and only then run it via VisualStudio. (this script already includes the Copy-ServiceFabricApplicationPackage command)
You can take it from your VisualStudio ServiceFabric output pane, just take the actual command from inside the double quotes (don't forget the dot!)
The full command should look like this:
powershell -NonInteractive -NoProfile -WindowStyle Hidden -ExecutionPolicy Bypass -Command ". C:\.................. "
So you should run only the . C:\....... part (with the prefix dot)

Hope it will help you as well :)

[amitaysh]

found a better solution for this...
seems like sometimes the boolean values sent by VS converted to string.
this solves the issue in 100% of the times we tried:
in file C:\Program Files\Microsoft SDKs\Service Fabric\Tools\PSModule\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1
line 274 - Change $CompressPackage with $false