Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Swarm Job support to Stack Deploy #2907

Merged
merged 1 commit into from
May 17, 2022
Merged

Added Swarm Job support to Stack Deploy #2907

merged 1 commit into from
May 17, 2022

Conversation

ollypom
Copy link
Contributor

@ollypom ollypom commented Jan 4, 2021

- What I did

Added the support for both global-job and replicated-job to Docker Stack deploy.

Follow up to #2262.

For replicated-jobs, the number of replicas adjusts the number of Concurrent Tasks as well as the required number of Completed Tasks. I have not added support for max-concurrent which allows the number of Concurrent Tasks and the number of Completed Tasks to be set independently. This is because max-concurrent is currently not defined in Compose, so I believe a follow up PR is required to add max-concurrent once support is in Compose.

- How to verify it
Locally verified with the following 3 Stack files.

$ cat global-job.yaml 
version: '3.9'

services:
  sleeper:
    deploy:
      mode: global-job
    image: ubuntu
    command:
      - sleep
      - "50"
$ cat replicated-job.yaml 
version: '3.9'

services:
  sleeper:
    deploy:
      mode: replicated-job
    image: ubuntu
    command:
      - sleep
      - "50"
$ cat max-replicas-per-node.yaml 
version: '3.9'

services:
  sleeper:
    deploy:
      mode: replicated-job
      replicas: 5
      placement: 
        max_replicas_per_node: 1
    image: ubuntu
    command:
      - sleep
      - "50"

- Description for the changelog
Swarm Job support in docker stack deploy

@EntraptaJ
Copy link

Anything blocking this from being merged? I'll admit, I don't know the process for this repo, but this not existing is blocking a project of mine.

@g0t4
Copy link

g0t4 commented Jun 21, 2021

I'm a fan of this.

  1. For consistency. In many ways 'docker stack deploy' vs 'docker service create' is a tradeoff of passing arguments via a compose file vs CLI arguments. Not that one can't just check a script into VCS and get many of the same benefits...

  2. ... And that's where the next purpose underscores the value of compose files for jobs too... even with jobs there is a desired state to be enforced and that is X jobs Completed (restart if they fail, etc) perhaps someday even some event based or schedule based triggers could be defined thus skewing toward a long running, periodic service.

  3. Regardless the future, even with Swarm Jobs in 20.10... a stack file is not just about docker service create... it also handles docker service update / rm and docker network create / update / rm & docker volume create / update / rm, same with docker config & docker secret ... and it even handles the notion of external vs internal defined resources... all together a very potent combination as I'm sure everyone arriving here is aware.

  4. When I first was exposed to jobs my natural inclination was to create a stack file and it was surprising to me to find that it failed. At a bare minimum it would be helpful if the CLI displayed an error message that indicated that jobs aren't (yet?) supported in stack deploys... because my next inclination was to find out the correct mode value assuming I put the wrong string in for mode... I began to wonder is it _ instead of - or is there no delimiter :) ... and those aren't easy questions to answer without digging deep into code and/or docs...

  5. Speaking of docs, I would love to see @ollypom's contribution to documenting Swarm Jobs get merged and released... most people won't know to google for his pull request that's still outstanding.

TLDR: Swarm Jobs in Stacks seems a must long term, unless jobs don't prove worthwhile as first class concepts.

+1 for everyone above getting the ball rolling on this!

@ollypom
Copy link
Contributor Author

ollypom commented Jun 25, 2021

@thaJeztah or @silvin-lubecki PTAL

Thanks 🙏

@grishin
Copy link

grishin commented Oct 8, 2021

This looks like a small change in code, but without yaml support jobs in Swarm are not usable. Most people who use Swarm for years emulated jobs with replicas: 0 in yaml. The overhead of converting our current hundred yaml scripts to cli style docker service update is HUGE. This one should definitely be a top priority bugfix.

@frabbit
Copy link

frabbit commented Oct 8, 2021

I would love to see support for this.

@JackMorganNZ
Copy link

Agree with you @grishin, we're holding off using jobs until it's supported in stack deploy.

Would really appreciate if @thaJeztah or @silvin-lubecki could provide some quick feedback for this, even if it's just an ETA on when this could be reviewed properly. Thanks!

@emmertf
Copy link

emmertf commented Jan 5, 2022

+1

@frapell
Copy link

frapell commented Apr 7, 2022

Any news on this?

@s4ke
Copy link

s4ke commented May 17, 2022

+1

Added the support for both `global-job` and `replicated-job` to Docker
stack deploy.

Signed-off-by: ollypom <oppomeroy@gmail.com>
@thaJeztah thaJeztah added this to the 22.06.0 milestone May 17, 2022
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sorry, this one somehow dropped off my radar (did a quick rebase to have a fresh run of CI)

@thaJeztah thaJeztah merged commit b6aa28b into docker:master May 17, 2022
@ollypom ollypom deleted the stack-deploy-swarm-jobs branch June 12, 2022 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.