Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression in 0.55.0: --spot is only valid with managed nodegroups #3929

Closed
nbusseneau opened this issue Jul 4, 2021 · 4 comments · Fixed by #3930
Closed

Regression in 0.55.0: --spot is only valid with managed nodegroups #3929

nbusseneau opened this issue Jul 4, 2021 · 4 comments · Fixed by #3930
Labels
kind/bug priority/important-soon Ideally to be resolved in time for the next release

Comments

@nbusseneau
Copy link

Hi,

We use eksctl on cilium/cilium-cli for deploying an EKS cluster and running tests on it via GitHub Actions (link).

First, we download the latest release:

      - name: Install eksctl CLI
        run: |
          curl -LO "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz"
          sudo tar xzvfC eksctl_$(uname -s)_amd64.tar.gz /usr/bin
          rm eksctl_$(uname -s)_amd64.tar.gz

Then, we deploy a cluster without a nodegroup:

      - name: Create EKS cluster without nodegroup
        run: |
          eksctl create cluster \
            --name ${{ env.clusterName }} \
            --tags "usage=${{ github.repository_owner }}-${{ github.event.repository.name }},owner=${{ steps.vars.outputs.owner }}" \
            --without-nodegroup

And finally, we setup a spot managed nodegroup:

      - name: Add managed spot nodegroup
        run: |
          eksctl create nodegroup \
            --cluster ${{ env.clusterName }} \
            --nodes 2 \
            --instance-types "t3.medium,t3a.medium" \
            --node-volume-type gp3 \
            --node-volume-size 10 \
            --managed \
            --spot \
            --node-private-networking

Looking at the list of scheduled runs (running every 6 hours) we can see that the last working run was on Jul 2 around 9 CEST, with the next run around 15 CEST failing at the managed spot nodegroup creation step, along with all subsequent runs:

Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

Nothing changed in how we setup EKS clusters between the last working and first failing runs mentioned above.

0.55.0 was released on Jul 2 at 14:27 CEST. I strongly suspect a regression in 0.55.0 introducing the issue. Looking at the list of changes, I feel like #3895 might be related?

I don't have access to my work machine at present but will try the following reproduction steps on Monday:

  • Download eksctl 0.54.0 and try to setup a spot managed nodegroup: I expect it will work.
  • Download eksctl 0.55.0 and try to setup a spot managed nodegroup: I expect it will not work.
@dz902
Copy link

dz902 commented Jul 5, 2021

I can reproduce this with 0.55 and 0.56-rc1. This is a fairly common command. It should not slipped under the radar for two releases.

tklauser added a commit to cilium/cilium-cli that referenced this issue Jul 5, 2021
The EKS workflows suddenly started failing with:

  Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

On Jul 2 at around 15 CEST
(https://github.com/cilium/cilium-cli/actions/runs/992727965). The
release of eksctl 0.55.0 coincides with this time frame. As we pin the
EKS workflows to the latest eksctl release, it is likely that this
caused the failures.

Pin the eksctl to 0.54.0 for now which doesn't seem to exhibit this
behavior.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser added a commit to cilium/cilium-cli that referenced this issue Jul 5, 2021
The EKS workflows suddenly started failing with:

  Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

On Jul 2 at around 15 CEST
(https://github.com/cilium/cilium-cli/actions/runs/992727965). The
release of eksctl 0.55.0 coincides with this time frame. As we pin the
EKS workflows to the latest eksctl release, it is likely that this
caused the failures.

Pin the eksctl to 0.54.0 for now which doesn't seem to exhibit this
behavior.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
@Callisto13
Copy link
Contributor

Thanks for reporting @nbusseneau, after a quick bisect I found the change came with #3887.

Fix will be along shortly 👍

tklauser added a commit to cilium/cilium-cli that referenced this issue Jul 5, 2021
The EKS workflows suddenly started failing with:

  Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

On Jul 2 at around 15 CEST
(https://github.com/cilium/cilium-cli/actions/runs/992727965). The
release of eksctl 0.55.0 coincides with this time frame. As we pin the
EKS workflows to the latest eksctl release, it is likely that this
caused the failures.

Pin the eksctl to 0.54.0 for now which doesn't seem to exhibit this
behavior.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
@nbusseneau
Copy link
Author

nbusseneau commented Jul 5, 2021

Can confirm I don't reproduce the issue with 0.54.0, but do with 0.55.0. Editing title.

EDIT: ah sorry, GitHub did not refresh the comment list so I had missed the previous comment 🙈 Thanks!

@nbusseneau nbusseneau changed the title Potential regression in 0.55.0: --spot is only valid with managed nodegroups Regression in 0.55.0: --spot is only valid with managed nodegroups Jul 5, 2021
@nikimanoledaki nikimanoledaki added the priority/important-soon Ideally to be resolved in time for the next release label Jul 5, 2021
@cPu1
Copy link
Collaborator

cPu1 commented Jul 6, 2021

Closed by #3930. The fix is out in https://github.com/weaveworks/eksctl/releases/tag/0.56.0-rc.1

@cPu1 cPu1 closed this as completed Jul 6, 2021
tklauser added a commit to cilium/cilium-cli that referenced this issue Jul 13, 2021
Commit 59785a9 (".github/workflows: use eksctl 0.54.0") pinnned
eksctl to v0.54.0 to work around an issue in eksctl v0.55.0. That issue
was fixed upstream and v0.56.0 released, so revert back to using the
latest eksctl version.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser added a commit to cilium/cilium-cli that referenced this issue Jul 13, 2021
Commit 59785a9 (".github/workflows: use eksctl 0.54.0") pinnned
eksctl to v0.54.0 to work around an issue in eksctl v0.55.0. That issue
was fixed upstream and v0.56.0 released, so revert back to using the
latest eksctl version.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
aditighag pushed a commit to aditighag/cilium-cli that referenced this issue Apr 21, 2023
The EKS workflows suddenly started failing with:

  Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

On Jul 2 at around 15 CEST
(https://github.com/cilium/cilium-cli/actions/runs/992727965). The
release of eksctl 0.55.0 coincides with this time frame. As we pin the
EKS workflows to the latest eksctl release, it is likely that this
caused the failures.

Pin the eksctl to 0.54.0 for now which doesn't seem to exhibit this
behavior.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
aditighag pushed a commit to aditighag/cilium-cli that referenced this issue Apr 21, 2023
Commit 59785a9 (".github/workflows: use eksctl 0.54.0") pinnned
eksctl to v0.54.0 to work around an issue in eksctl v0.55.0. That issue
was fixed upstream and v0.56.0 released, so revert back to using the
latest eksctl version.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
michi-covalent pushed a commit to michi-covalent/cilium that referenced this issue May 30, 2023
The EKS workflows suddenly started failing with:

  Error: couldn't create node group filter from command line options: --spot is only valid with managed nodegroups (--managed)

On Jul 2 at around 15 CEST
(https://github.com/cilium/cilium-cli/actions/runs/992727965). The
release of eksctl 0.55.0 coincides with this time frame. As we pin the
EKS workflows to the latest eksctl release, it is likely that this
caused the failures.

Pin the eksctl to 0.54.0 for now which doesn't seem to exhibit this
behavior.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
michi-covalent pushed a commit to michi-covalent/cilium that referenced this issue May 30, 2023
Commit 59785a9f9c7c (".github/workflows: use eksctl 0.54.0") pinnned
eksctl to v0.54.0 to work around an issue in eksctl v0.55.0. That issue
was fixed upstream and v0.56.0 released, so revert back to using the
latest eksctl version.

Ref: eksctl-io/eksctl#3929
Signed-off-by: Tobias Klauser <tobias@cilium.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug priority/important-soon Ideally to be resolved in time for the next release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants