"sfctl application upgrade" seems to be broken

[hunterSThompson]

It appears that upgrading a service-fabric application via sfctl application upgrade is completely broken.

I'm attempting to upgrade using the stand procedure documented here: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-application-lifecycle-sfctl#upgrade-application

I can upload and provision the new version fine, and it shows up in the Service Fabric explorer. But running sfctl application upgrade --app-id sf-test --parameters "{}" --health-check-stable-duration 60 --upgrade-domain-timeout 1200 --upgrade-timeout 3000 --mode Monitored throws a 500 error:

Error occurred in request., RetryError: HTTPConnectionPool(host='localhost', port=19080): Max retries exceeded with url: /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 (Caused by ResponseError('too many 500 error responses',))

However, I can successfully upgrade the app with this PowerShell command:

Start-ServiceFabricApplicationUpgrade -ApplicationName "fabric:/sf-test" -ApplicationTypeVersion "2.0.0" -HealthCheckStableDurationSec 60 -UpgradeDomainTimeoutSec 1200 -UpgradeTimeout 3000 -FailureAction Rollback -Monitored

I also tried this on the EchoService and VisualObject samples in this repo: https://github.com/Azure-Samples/service-fabric-java-getting-started

Same deal.

My Environment:

Bash on Ubuntu 16.04 LTS, installed sfctl via: sudo curl -s https://raw.githubusercontent.com/Azure/service-fabric-scripts-and-templates/master/scripts/SetupServiceFabric/SetupServiceFabric.sh | sudo bash

[samedder]

Can you include the following information?

• Version of sfctl used
• Version of your cluster
• Output of application upgrade with the --debug flag specified?

[hunterSThompson]

Version of sfctl: 6.0.120.1
Version of cluster: I tried both a local cluster and one on Azure, they were both 6.0.120.1.

Here's the output with --debug:

hunt@hunt-VirtualBox:~/service-fabric-java-getting-started/Actors$ sfctl application upgrade --app-id sf-test --app-version 2.0.0 --parameters "{}" --health-check-stable-duration 60 --upgrade-domain-timeout 1200 --upgrade-timeout 3000 --mode Monitored --debug
Command arguments: ['application', 'upgrade', '--app-id', 'sf-test', '--app-version', '2.0.0', '--parameters', '{}', '--health-check-stable-duration', '60', '--upgrade-domain-timeout', '1200', '--upgrade-timeout', '3000', '--mode', 'Monitored', '--debug']
Event: Cli.PreExecute []
Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fa564cd4ea0>, <function OutputProducer.on_global_arguments at 0x7fa56485f620>, <function CLIQuery.on_global_arguments at 0x7fa5645e4268>]
Event: CommandInvoker.OnPreCommandTableCreate [<function CLILogging.remove_logger_flags at 0x7fa564cd4f28>]
Event: CommandLoader.OnLoadArguments []
Event: CommandInvoker.OnPostCommandTableCreate []
Event: CommandInvoker.OnCommandTableLoaded []
Event: CommandInvoker.OnPreParseArgs []
Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fa56485f6a8>, <function CLIQuery.handle_query_parameter at 0x7fa5645e42f0>]
msrest.pipeline : Configuring request: timeout=100, verify=True, cert=None
msrest.pipeline : Configuring redirects: allow=True, max=30
msrest.pipeline : Configuring proxies: ''
msrest.pipeline : Evaluate proxies against ENV settings: True
msrest.pipeline : Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90
urllib3.connectionpool : Starting new HTTP connection (1): localhost
urllib3.connectionpool : http://localhost:19080 "POST /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 HTTP/1.1" 500 76
urllib3.util.retry : Incremented Retry for (url='/Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0'): Retry(total=2, connect=3, read=3, redirect=None, status=None)
urllib3.connectionpool : Retry: /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0
urllib3.connectionpool : http://localhost:19080 "POST /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 HTTP/1.1" 500 76
urllib3.util.retry : Incremented Retry for (url='/Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0'): Retry(total=1, connect=3, read=3, redirect=None, status=None)
urllib3.connectionpool : Retry: /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0
urllib3.connectionpool : http://localhost:19080 "POST /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 HTTP/1.1" 500 76
urllib3.util.retry : Incremented Retry for (url='/Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0'): Retry(total=0, connect=3, read=3, redirect=None, status=None)
urllib3.connectionpool : Retry: /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0
urllib3.connectionpool : http://localhost:19080 "POST /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 HTTP/1.1" 500 76
msrest.exceptions : Error occurred in request., RetryError: HTTPConnectionPool(host='localhost', port=19080): Max retries exceeded with url: /Applications/sf-test/$/Upgrade?timeout=60&api-version=6.0 (Caused by ResponseError('too many 500 error responses',))

Thanks!
Hunt

[hunterSThompson]

Also I fixed the title... sorry about that, I was (clearly) in a rush.

[samedder]

@hunterSThompson that's not a version of sfctl that's published. Can you verify that's the version you have installed? You can get the install version by running pip show sfctl

[hunterSThompson]

So... interestingly I don't seem to have pip installed, so guess I must have done the manual installation.

Here's the output of apt-get -s install servicefabric:

hunt@hunt-VirtualBox:~$ sudo apt-get -s install servicefabric
Reading package lists... Done
Building dependency tree
Reading state information... Done
servicefabric is already the newest version (6.0.120.1).
servicefabric set to manually installed.
The following packages were automatically installed and are no longer required:
bridge-utils containerd runc ubuntu-fan
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 229 not upgraded.

Should the manual install give you the bleeding edge version?

[samedder]

@hunterSThompson you should have pip installed if you ran the install script you mentioned in the beginning. From the script:

#
# Setup Azure Service Fabric CLI
#

apt-get install python -f -y
ExitIfError $?  "Error@$LINENO: Failed to install python for sfctl setup."

apt-get install python-pip -f -y
ExitIfError $?  "Error@$LINENO: Failed to install pip for sfctl setup."

pip install sfctl
ExitIfError $?  "Error@$LINENO: sfctl installation failed."

If you don't, theres documentation on how to install sfctl.

[hunterSThompson]

I tried uninstalling whichever version I had installed-- then re-ran the script via: sudo curl -s https://raw.githubusercontent.com/Azure/service-fabric-scripts-and-templates/master/scripts/SetupServiceFabric/SetupServiceFabric.sh | sudo bash.

Now, pip show sfctl says I have version 2.0.0 installed... but running the upgrade command still fails with the same generic 500 error.

To be doubly sure I had the right version, I spun up a fresh Ubuntu 16.04 VM on VirtualBox, then installed GIT and curl, then installed sfctl via the script. Once again, I verified correct version via pip show sfctl.

I then pulled down the service-fabric-java-getting-started repo, installed it, and attempted to upgrade-- still the same error on the sfctl application upgrade command. I tried on both local and Azure clusters.

I'm still able to upgrade the application successfully via PowerShell though.

[samedder]

@hunterSThompson can you include the output with the --debug flag once more? Paste it into a gist or pastebin link if possible. Thanks!

[hunterSThompson]

Dang, knew I was forgetting something!

Paste-bin link: https://pastebin.com/0K8pVxyn

[samedder]

Thanks, one other piece of information @hunterSThompson:

Can you get the Resource ID from the Azure Portal for your cluster? This should be a string of the format:

/subscriptions/<guid>/resourcegroups/<string>/providers/Microsoft.ServiceFabric/clusters/<string>

[hunterSThompson]

No problem.

/subscriptions/3664c441-5d00-4d69-adfd-e230ce32edc8/resourceGroups/service-fabric-testing/providers/Microsoft.ServiceFabric/clusters/guacamole-test

[samedder]

@hunterSThompson can you try with --app-id fabric:/EchoServerApplication ?

[hunterSThompson]

Welp, I could have sworn I tried that, but I clearly did not as it's working now...

I see that the documentation clearly says to use the full URI scheme here: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-sfctl-application#sfctl-application-upgrade

But here: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-application-lifecycle-sfctl#upgrade-application and here: https://github.com/Azure-Samples/service-fabric-java-getting-started are both not using the fabric:/ convention.

That should be updated so more people don't get confused... seem's like the CLI could use some better error handling too :)

Thanks for you help!

[samedder]

Yep, this is then a duplicate of #37.

We're aware also the error responses are not helpful. This is something we're working on improving in future releases.

[hunterSThompson]

Awesome. Thanks again for your time