Release is invalid. Code is missing for service error in SF Application Upgrade using diffPackage.

[muradkhateeb78]

Hi,

SF Application details
I have an SF app and I had enabled application upgrade for my application while deploying on cloud. The architecture and a bit of other details regarding my application can be found at the git issue https://github.com/Azure/service-fabric-issues/issues/1372.

Issue/Error

Details about the error
I am using a diff package so I had intentionally removed the code and config packages for the service named in the error above, because I didn't want that service to be upgraded. The version number of this service in the service manifest is the same as the version number of the same service deployed on the cluster. Ideally, it should not be looking for the code package as it is not intended to be upgraded. Is there any other indicator other than the updated version number that the SF platform uses to judge whether a service is intended to be upgraded or not?

This issue does not come always. The upgrade gets completed most of the time. This issue also appears with other services at times.

R&D at my end
I had gone through a couple of related git issues like microsoft/service-fabric-issues#837 and microsoft/service-fabric-issues#595 and upon investigating I found that the SF SDK 's script Publish-UpgradedServiceFabricApplication.ps1 specifies a connection string parameter in Test-ServiceFabricApplicationPackage --which is defined in the cluster manifest. Also the version number has not been updated indicating the services has not intended to be upgraded and hence there is a diff package for this particular service.

Any thoughts?

[ashishnegi]

Did you try running the failing command Test-ServiceFabricApplicationPackage yourself and see what is the error ?

[muradkhateeb78]

Yes I had tried Test-ServiceFabricApplicationPackage command myself and the results are as follows.

1. The content in code package has changed but the version has not
   I was only updating the version number of the service to enable it to be upgraded and I used to leave to code and config version as they are. Ideally this should not matter as these code and config package are residing under a service package with a different version number as I had updated the service version number.
   Anyways, I had changed the code and config versions as well for now.

2. Service manifest has changed but the version number didn't
   For one of the service which I did not want to be upgraded I had not updated the version and as it was a diff package so it had missing code and config packages and there was only one service manifest file. Does service manifest also gets a different checksum as a result of package generation from a build pipeline as it happens with code and config files?

[ashishnegi]

In first case,

I was only updating the version number of the service to enable it to be upgraded and I used to leave to code and config version as they are.

@muradkhateeb78 So, what changed under that service for which you increased service version number ?

For second case,
@masnider Do we need to keep old ServiceManifest.xml for a service in a diff package that has not changed ?

[muradkhateeb78]

The changes were in the code section only, but I intentionally upgraded the service version in the service manifest only. I wanted both the code and the config packages to be updated during the upgrade.

I am of the view that if you update the service version number and provide it with the code and config packages then both of them will be updated during the upgrade, without noticing the code and config package versions as the service version has been updated and technically they are the code and config packages for a newer version of the service package.

[ashishnegi]

No, this is not right. Changing only Service version won't work.

Think of whole application package as a tree where version number should start changing from the leaf node. Non-Leaf nodes should have highest values of their children.

For the scenario, where you want to upgrade only the config, you will increase version of Service and Config section only but not of Code. In this case, the diff package won't have Code folder. This reduces the overall updated package size considerably.

[muradkhateeb78]

Ok, Great. Just one last question. If the upgrade fails on any one of the update domain and the specified failure action is Rollback, Will the upgrade be rolled back to the previous version for all of the update domains including those on which the upgrade had succeeded, or it will only be rolled backed on the current update domain to its previous version?

[muradkhateeb78]

@ashishnegi So everything works as expected and only those services get upgraded which are intended to, i.e whose version numbers have been updated. But once in a while suddenly the deployment fails resulting in the generation of same build layout invalid and package missing error. For this particular release it doesn't get fixed unless you updated the version numbers of all of the services which means you force all of your services to be upgraded. It looks that it is comparing the diff package to something else other than the package residing in the ImageStore? Could you please shed some light on what could be happening wrong?

[ashishnegi]

@muradkhateeb78 sorry, I didn't see your last question earlier.

Will the upgrade be rolled back to the previous version for all of the update domains

Yes, on all the upgrade domains. You should see only one version of your app running at a time, when no upgrade/rollback is going on or has finished.

[ashishnegi]

It looks that it is comparing the diff package to something else other than the package residing in the ImageStore?

Do you have reproducible steps ?
Do you see this on dev cluster or on multi node cluster ?
Can you share the SF traces ?