Gets stuck on checking an update

[iangregsondev]

Hi,

Just started to use this, its working great but it seems to stop on a particular container, can you help ?

I mean I have quite a few services - and all isn't listed here but it seems to have stopped on the last run - there is no output saying "no updates...."

Enabling synchronous service updates


Trying to update service bitcoin-testnet_bcoin with image iangregsondev/bcoin:latest


No updates to service bitcoin-testnet_bcoin!


Trying to update service bitcoin-testnet_bitcoind with image iangregsondev/bitcoind:0.19.0.1


No updates to service bitcoin-testnet_bitcoind!


Trying to update service bitcoin-testnet_lnd1 with image iangregsondev/lnd:latest


No updates to service bitcoin-testnet_lnd1!


Trying to update service bitcoin-testnet_lnd2 with image iangregsondev/lnd:latest


No updates to service bitcoin-testnet_lnd2!


Trying to update service bitcoin-testnet_loop1 with image iangregsondev/loop:latest

Am I doing something wrong?

The only thing I can see with this particular service is that it's not currently deployed. Would this be a problem?

[djmaze]

@iangregsondev Well, if it shows 0/1 it is currently flapping/broken. You should have a look in the logs on why it is not starting correctly. That could be a problem why the service update hangs.

You can also try the update manually in order to see what goes wrong:

docker service update --detach=false --with-registry-auth --image iangregsondev/loop:latest bitcoin-testnet_loop1 

[iangregsondev]

Hi, thanks @djmaze, yes, in fact, that container was broken. I fixed it and all is working well now.

I just thought that shepherd would timeout if that container hadn't finished updating - so it would continue.

Anyway all working now!

Thanks again, by the way - great little container, thanks for your hard work - it makes my life easier.

[DDuarte]

In my case, I'm intentionally having services with

    deploy:
      restart_policy:
        condition: none

so they run once and exit.

It seems that docker service update gets stuck forever, so Shepherd also gets stuck:

$ docker service update "base_nfs_helper" --image="myimage:latest"
base_nfs_helper
overall progress: 0 out of 1 tasks
1/1:

Regardless of why a service update gets stuck, I think it makes sense for Shepherd to enforce a reasonable timeout and continue its job so that a single service "problem" doesn't stop other updates.

Should I send a PR skipping services that use restart_policy none?
This does not seem doable with FILTER_SERVICES.

[djmaze]

@DDuarte Sorry for the late answer. I am using swarm-cronjob for scheduling services. Those have their restart_policy to condition: none and replicas set to 0 (so they don't automatically start when being (re-)deployed). Still, it would be cool if their image reference could be updated automatically.

So I think we should not skip those services if replicas is set to 0. Does this make sense to you?

[DDuarte]

It does! In that case I think it would be good to review that timeout. I tried with the builtin timeout in the docker service update call but without much success.

Regardless of why a service update gets stuck, I think it makes sense for Shepherd to enforce a reasonable timeout and continue its job so that a single service "problem" doesn't stop other updates.

[djmaze]

So I think we should not skip those services if replicas is set to 0

Oh, I guess I wanted to write So I think we should not skip those services if replicas is not set to 0 (in order to skip only those services which might be restarted unintentionally by the update).

About the timeout, I am not sure what would be a good way to implement that. As I currently don't have a need for that myself, I am open for PRs!