Agent gets stuck and hangs if automatic upgrade fails

[melezhik]

Have you tried trouble shooting?

yes

Agent Version and Platform

any version before 2.140.0 , have not tested yet for this version

Windows OS

VSTS Type and Version

VisualStudio.com

What's not working?

Based on #1696, especially #1696 (comment)

Agent gets stuck and hangs without EASY way to roll it back to the previous version ( unless manual cleanup and reinstall ) if for some reasons agents fails to autoupdate.

[bryanmacfarlane]

Updating an agent (copying down files, updating the service, etc...) is not an atomic operation. We have as many efforts to retry and absorb failures but if something fails in the script to run those sequence of steps fail, there isn't many options.

I'm afraid this issue is not actionable.

We do have a way to rollback, it's an xcopy copy extract and run config.

[melezhik]

We do have a way to rollback, it's an xcopy copy extract and run config.

It would be nice if agent handle this

I have around 6-7 agents get stuck every 2-3 weeks due autoupdate failure , and manual rollback is real chore.

[TingluoHuang]

@melezhik do you happen to know what is causing the auto-update failure? proxy/firewall?

[melezhik]

yeah, our build servers are under firewall, I am awaiting when maintainers add "vsts download url" to the white list. However the issue is not about it.

Imagine someone have many agents up and running, then at the moment ( when new version is released ) all those agents fail to autoupdate due to whatever reasons ... Well this automatic update breaks all the build infrastructure ... And even though manual fix is not that hard, if we have a dozens of agents it is an effort to recover.

In all the mature package managing system, software updates failures never lead to breakage of the system, because they are atomic ... - success or automatic rall back to the old state ( see linux examples - apt , yum so on )

Just to think about. Today it is only me who care about this, tomorrow other customer might complain ...

IMHO for now autoupdate operation is not safe and not transnational which might cause many issue to vsts users ...