Better stack list experience with Kubernetes and Swarm

[mat007]

- What I did

Made it easier to work with both swarm and kubernetes stacks combined, e.g.

• added support in stack ls for the value all in the --orchestrator flag which triggers the display of both swarm and kubernetes stacks
• allowed multiple --namespace for docker stack ls

- How I did it

1. The stack lists are now merged after having been queried for each orchestrator and before sorting them and applying the formatting
2. The namespace flag was moved from the stack command to all sub-commands thus allowing for the stack ls sub-command to accept several occurrences and loop over them

- How to verify it

# List both swarm and kubernetes with namespace
$ docker stack ls --orchestrator=all --namespace=tutu
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
dtc                 5                   Kubernetes          tutu
wall-e              2                   Swarm

# List stacks for both swarm and kubernetes in empty namespace
$ docker stack ls --orchestrator=all --namespace=pipo
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
wall-e              2                   Swarm

# List stacks for both swarm and kubernetes (all namespaces implied)
$ docker stack ls --orchestrator=all
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
abc                 2                   Kubernetes          molo
dtc                 5                   Kubernetes          tutu
wall-e              2                   Swarm

# List stacks for kubernetes in several namespaces
$ ./build/docker.exe stack ls --orchestrator=kubernetes --namespace=tutu,tutu,pipo,molo
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
abc                 2                   Kubernetes          molo
dtc                 5                   Kubernetes          tutu

- Description for the changelog

• List stacks for both Swarm and Kubernetes with --orchestrator=all in docker stack ls
• Allow several occurrences of --namespace for Kubernetes with docker stack ls

- A picture of a cute animal (not mandatory but encouraged)

[mat007]

⛑ comes after #991, only the 5 last commits are new.

[vdemeester]

This should stay

[mat007]

This was moved into each sub-command, so still there in a way.

[vdemeester]

Why making these flags per stack subcommand instead of putting the on the stack command and make the subcommand inherit ?

[mat007]

Because stack ls now needs a different behaviour than other stack sub-commands for --namespace and we cannot set it once with type string and another time with []string (e.g. several namespaces).
So the drawback is a bit of code duplication setting the flags for all sub-commands, which was factorized into kubernetes.InstallFlags

[vdemeester]

Will setting a default here will override the default namespace defined in the context ?

[mat007]

Hmm yes, I suppose it will, which is likely a bug.
However I don't think this has been introduced by this PR as the default for --namespace was already default.
Could this be fixed in a follow up?

[mat007]

Actually there is code in kubernetes.WrapCli to deal with that and it seems to be working as expected

> docker stack ls  --orchestrator=kubernetes
stacks.compose.docker.com is forbidden: User "uu" cannot list stacks.compose.docker.com in the namespace "default": access denied
> kubectl config set-context ucp_34.209.72.126:6443_uu --namespace=tutu
Context "ucp_34.209.72.126:6443_uu" modified.
> docker stack ls  --orchestrator=kubernetes
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
dtc                 5                   Kubernetes          tutu

[vdemeester]

all-namespaces should probably be marked as experimentalCLI too

[mat007]

Yes, I'll add that, thanks.

[vdemeester]

sort.Slice(stacks, func(i, j int) bool { …

[mat007]

OK

[vdemeester]

Same here

[vdemeester]

and here …

[vdemeester]

… and here

[vdemeester]

We may want to share this error somehow

[mat007]

OK!

[vdemeester]

I don't get this change, this feels really weird to add the flag twice (on cmd.Flags() and cmd.PersistentFlags())

[mat007]

"orchestrator" is the only flag added to cmd.PersistentFlags() in InstallFlags, all the other flags are still in cmd.Flags()

[mat007]

That being said if you would rather have me remove that last commit, I'd be fine with that!

[thaJeztah]

Slightly confused as well

[mat007]

OK, I'll just remove the related commit.

[mat007]

@vdemeester I fixed all comments except the ones under discussion about the flags e.g.

• persistence of --orchestrator
• multiple --namespace implementation

Would you kindly please take another look?

[thaJeztah]

This adds a lot of complexity to the codebase, and I'm not sure if all is needed. I left some comments inline, but must admit some of the flow is a bit hard to follow

[thaJeztah]

Looks like (default swarm) no longer matches reality (for example, having "orchestrator":"kubernetes" in my ~/.docker/config.json would set the default)

[mat007]

That's a good point. I'll remove the (default swarm).

[mat007]

Also this was there before but I don't thing with the docker cli brings anything and the other flag descriptions don't have it. I'll remove it too.

[thaJeztah]

Looks like this change to all-namespaces should be done in the other PR (#991)?

[mat007]

You're right, too late though as #991 has been merged in the meanwhile…

[thaJeztah]

Is there a reason we use two separate approaches for the namespace flag? Looks like we define it twice now: once through AddNamespaceFlag()

[mat007]

All docker stack sub-commands use AddNamespaceFlag and have a namespace flag defined as a StringVar except ls which has a namespace flag defined as a StringSliceVar.
This is because for all sub-commands but ls only one --namespace can be set whereas ls can have several, e.g. docker stack ls --namespace=tutu,default,bla or docker stack ls --namespace=tutu --namespace=default --namespace=bla.

[thaJeztah]

Wondering now if this should be a StringSliceVar or a StringVar; I think StringVar can be specified multiple times (--namespace=bla --namespace=foo), but does not support comma-separated values; we haven't documented using comma-separated values, and don't think we support it for other options (possibly --security-opt only)

(note that if a command only accepts a single namespace, we could still produce an error if multiple are given)

[mat007]

I'll take another look but from what I remember there was no way to make it look nice, for both cases, e.g.

$ docker stack ps  --help
…
Options:
…
      --namespace string    Kubernetes namespace to use

VS

$ ./build/docker.exe stack ls --help
…
Options:
…
      --namespace strings   Kubernetes namespaces to use

Also I don't see how a single StringVar can hold multiple --namespace values…

[mat007]

I think StringVar can be specified multiple times (--namespace=bla --namespace=foo), but does not support comma-separated values

In that case the latest value is silently selected (here foo), I suppose the flags are simply processed in order.
If we just let comma-separated values out of the question we could indeed emulate both behaviors with a StringSliceVar by using the last element when only one namespace is required.

(Incidentally passing --namespace=bla,foo to a StringVar flag yields a value of bla,foo…)

[mat007]

The best I could come up with by using []string on the stack command level is

$ ./build/docker.exe stack --help

Usage:  docker stack COMMAND

Manage Docker stacks

Options:
      --kubeconfig string   Kubernetes config file
      --namespace strings   Kubernetes namespace to use

This is wrong for all sub-commands as for most of them it says strings plural whereas only one is supported (and differs for instance from kubeconfig), and for docker stack ls which supports several namespaces the description should have namespaces…

I don't see how to have types and descriptions accurate for all sub-commands without duplicating the flag.

[thaJeztah]

Looks like this is to get rid of duplicates? This feels a bit like premature optimisation, also I'd expected this to return a []string (matching the type before removing duplicates)

[mat007]

Yes this is to remove duplicates, again to match kubectl behavior. I'll change the return type to a []string.

[mat007]

…and refactor it to

func removeDuplicates(namespaces []string) []string {
	mnms := map[string]struct{}{}
	for _, nm := range namespaces {
		mnms[nm] = struct{}{}
	}
	namespaces = []string{}
	for nm := range mnms {
		namespaces = append(namespaces, nm)
	}
	return namespaces
}

[thaJeztah]

There's now opts.Namespaces (for stacks) and opts.Namespace, correct?

[mat007]

Yes, there first one is of type []string and the latter string.

[thaJeztah]

Wouldn't len(opts.Namespaces) give the same here?

[mat007]

Oh yes, nice catch!

[thaJeztah]

This could just take (flags *flag.FlagSet, or even namespaces []string, then call it with getNamespaces(cmd.Flags()), but see my other comments

[mat007]

This has actually been changed to receive a flags *flag.FlagSet in a follow-up PR, but I might as well do it there.

[mat007]

Hmm sorry I meant []string and passing it opts.Namespaces…

[thaJeztah]

Both flags and namespace basically come from cmd.Flags(); why pass both, if we have all information in flags?

[mat007]

The way Kubernetes works implies that listing the stacks (or anything else) must be made within the context of a namespace, thus as we want to merge the lists of stacks from different namespaces we create a new kubernetes.Options for each namespace when iterating over them for docker stack ls.
I'll refactor the code to separate the different behaviors in respect to namespaces.

[thaJeztah]

in what situation would flags.GetString("namespace") give anything different than namespace[0]? If we only expect a single namespace to be passed, it should not take more than one

[mat007]

This is again due to the discrepancy between docker stack ls and the other docker stack sub-commands regarding the number of --namespace flags allowed.

[thaJeztah]

Slightly confused as well

[thaJeztah]

This needs a rebase now

[mat007]

@thaJeztah I believe the code to be much clearer now, thank you for your comments!
I merged the changes into 539654c as it was the commit which they all originated from.
Would you please take another look?

[wsong]

So if you have orchestrator=all and then you do docker stack ls --namespace=foobar, this will still return all stacks in every namespace?

[mat007]

This branch is only executed if opts.AllNamespaces || len(opts.Namespaces) == 0 e.g. there is already --orchestrator=all or no --namespace.
Here is what this gives me:

$ ./build/docker.exe --orchestrator=all stack ls
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
blaaaaaa            1                   Kubernetes          default
blaaaaaaaaa         1                   Kubernetes          default
dtc                 5                   Kubernetes          tutu
dtc                 5                   Kubernetes          tata
dtcccccc            5                   Kubernetes          default

$ ./build/docker.exe --orchestrator=all stack ls --namespace=tata
NAME                SERVICES            ORCHESTRATOR        NAMESPACE
dtc                 5                   Kubernetes          tata

[wsong]

Ah, I see now.

[mat007]

This looks like what we want because

docker stack deploy --namespace=foo --namespace=bar -c docker-compose.yml --orchestrator=swarm foo
unknown flag: --orchestrator

The orchestrator flag must be right after docker (I removed the commit which made it persistent).

docker --orchestrator=swarm stack deploy --namespace=foo --namespace=bar -c docker-compose.yml foo
"--namespace" is only supported on a Docker cli with kubernetes features enabled

Passing --orchestrator=swarm tells docker to target that orchestrator for which the --namespace makes no sense, hence the error.

Or am I missing something?

[thaJeztah]

The orchestrator flag must be right after docker (I removed the commit which made it persistent).

hm, right; I'm in doubt if we should only accept it as a global flag. I know users can get very confused as to "where" to pass a flag; existing examples are, e.g. docker container run -e foo mysql --host=bar (where --host=bar is a mysql option, but -e is a docker option); adding global flags makes this even more confusing.

[mat007]

@thaJeztah I brought the refactoring over from https://github.com/docker/cli/pull/1034/files/e0dbcb3b88f615d358fc5bed93ca3d14aa094d87..691eeff5a5db87be3fe5e0f1ca852024edf08f1c#diff-0ade8f01def39989ced9bb51bab08a69
Tell me if you'd like me to change the orchestrator flag, maybe we could move it after docker stack for instance? This could be addressed in another PR if needed though as this has not been touched in this one?

[mat007]

@vdemeester @thaJeztah I can remove the last commit if that helps moving forward with this PR (and either open a separate PR with it or just forget about it)?

[thaJeztah]

Code generally LGTM, but perhaps we should indeed move the refactor out of this PR (t.b.h., I don't recall suggesting that change, but it could've been me, idk)

[thaJeztah]

Thinking of rephrasing this (also in light of #770); perhaps;

No orchestrator specified. Use either "kubernetes" or "swarm"

(better suggestions welcome)

[mat007]

As an error message shouldn't it start with a lower case? I would probably avoid the dot too, e.g.

no orchestrator specified, use either "kubernetes" or "swarm"

or

no orchestrator specified: use either "kubernetes" or "swarm"

?

[thaJeztah]

I like this one:

no orchestrator specified: use either "kubernetes" or "swarm"

[thaJeztah]

Not for this PR, but we should consider passing an opts here, instead of a boolean (opts.AllNamespaces) to prevent future bloat in the signature

[thaJeztah]

I still have reservations on the magic relation between --orchestrator=all and --all-namespaces (copying my earlier comment #1031 (comment) here, as it was now hidden);

I'm having trouble conflating these; this may become very confusing:

• configuration has "orchestrator: all": docker stack ls shows all stacks (both swarm and k8s)
• I want to view only the k8s stacks, so I add --orchestrator=kubernetes. Now I no longer see all my stacks, only those in the default namespace
• confused

@chris-crone :

The thinking behind this was as follows: When you list stacks across orchestrators your intention is to see all the stacks running on your cluster. When you specify kubernetes as your orchestrator; you are doing something more specific and want similar behaviour to what kubectl gives you.

I can see how this might be confusing for your flow. There will be a NAMESPACE column which should help the user understand what has happened.

@chris-crone : Ah, see you're referring to the case when a user has set orchestrator to all in their config file. I don't think many users will do this as they will only be able to list resources. In order to manipulate resources, they will need to specify an orchestrator.

I don't think many users will do this as they will only be able to list resources.

I'm actually not sure: I can definitely see it being useful to just run any docker <foo> list or docker <foo> ps command, and see everything I have running, then use --filter <something> and/or --orchestrator=x to limit the list to what I'm looking for, or (when deploying / interacting with objects), pass --orchestrator=foo --namespace=bar as part of the steps to deploy.

Which brings me to two things to consider:

• Add orchestrator and namespace as filter options (--filter orchestrator=swarm) (no high-priority, but for consistency)
• When interacting with existing objects, only require --orchestrator and/or --namespace if the object refered to is ambiguous, for example, if there's only one stack with a given name, there would be no need to ask for --orchestrator.

[mat007]

I removed the commit with the tests refactoring and changed the error message when an orchestrator must be explicitly specified.

[thaJeztah]

@mat007 could you move the all orchestrators -> all namespaces changes to a separate PR to unblock this one?

It's three lines, so easy to review from a code-perspective, so if we get agreement on that one, we should be able to merge that one fast;

if dockerCli.ClientInfo().HasAll() {
	opts.AllNamespaces = true
}

[mat007]

@thaJeztah done!

[thaJeztah]

LGTM

sorry for all the hence-and-forth 😅

[vdemeester]

LGTM 🐯

[chris-crone]

Just to keep track, the all orchestrator all namespace work is in #1059.