Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iptables: fix check for rule existence in ip6tables v1.8.9 #2976

Conversation

markusboehme
Copy link
Member

@markusboehme markusboehme commented Apr 3, 2023

Issue number:

Closes #2975

Description of changes:

iptables v1.8.9 breaks the rule existence check in ip6tables. Fix this downstream until the issue has been patched in the upstream project.

I proposed this patch upstream: https://marc.info/?l=netfilter-devel&m=168055689214144

Testing done:

  • Locally: Reproducer from upstream bug report shows expected behavior and reports the missing rule
  • Locally: Valgrind doesn't report any out-of-bounds reads anymore
  • Bottlerocket: aws-k8s-1.24 on x86_64 builds and can launch pods
  • Bottlerocket: pod can have multiple host ports forwarded to it with IPv4 (confirmed via iptables -t nat -L on the host)
  • Bottlerocket: pod can have multiple host ports forwarded to it with IPv6 (on an IPv6 cluster; details see below)

Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: test-poly-host-ports
spec:
  restartPolicy: OnFailure
  containers:
    - name: test
      image: amazonlinux:2023
      command: ["sleep", "3600"]
      ports:
        - containerPort: 80
          hostPort: 10080
        - containerPort: 81
          hostPort: 10081

On the host (admin container --> sudo sheltie):

bash-5.1# ip6tables -t nat -L -n
[...]
Chain CNI-DN-b4a7dc28d99c75afb1337 (1 references)
target     prot opt source               destination         
CNI-HOSTPORT-SETMARK  6    --  2a05:d014:a18:a000:8e24::2  ::/0                 tcp dpt:10080
DNAT       6    --  ::/0                 ::/0                 tcp dpt:10080 to:[2a05:d014:a18:a000:8e24::2]:80
CNI-HOSTPORT-SETMARK  6    --  2a05:d014:a18:a000:8e24::2  ::/0                 tcp dpt:10081
DNAT       6    --  ::/0                 ::/0                 tcp dpt:10081 to:[2a05:d014:a18:a000:8e24::2]:81

Chain CNI-HOSTPORT-DNAT (2 references)
target     prot opt source               destination         
CNI-DN-b4a7dc28d99c75afb1337  6    --  ::/0                 ::/0                 /* dnat name: "aws-cni" id: "77c07952c638f66094ba7b92262990f5e9c4e680a038d91a81f1162d4beef863" */ multiport dports 10080,10081
[...]

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

iptables v1.8.9 breaks the rule existence check in `ip6tables`. Fix this
downstream until the issue has been patched in the upstream project.

Signed-off-by: Markus Boehme <markubo@amazon.com>
Copy link
Contributor

@etungsten etungsten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if IPv6 testing goes well

Copy link
Contributor

@stmcginnis stmcginnis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@markusboehme
Copy link
Member Author

I successfully tested the change on an IPv6 EKS cluster and updated the PR description with the steps I took.

@markusboehme markusboehme merged commit 948cb1d into bottlerocket-os:develop Apr 4, 2023
@yann-soubeyrand
Copy link

Hello,

I don’t know if this is the right place to ask this question, but do you know when new AMI will be released and available on EKS node groups and if the current AMI will be retracted in the interim?

@markusboehme
Copy link
Member Author

@yann-soubeyrand: There is no plan to retract the AMIs for Bottlerocket v1.13.x or their SSM parameters at this point as this may break other users.

The fix unfortunately missed the window for the just released v1.13.2. It will for sure be included in the next official release, be that another v1.13.x point release or v1.14.0. There is no fixed date for such a release yet, but we are looking into providing a v1.13.3. Please expect more information on that early next week.

For the time being, you may avoid the issue by creating a custom launch template referencing an AMI for Bottlerocket v1.12.0. The documentation for AWS EKS Managed Node Groups contains more information on this. You can determine suitable AMI IDs by querying SSM parameters, e.g. aws ssm get-parameter --region eu-central-1 --name '/aws/service/bottlerocket/aws-k8s-1.25/x86_64/1.12.0/image_id' --query Parameter.Value --output text (substituting the desired Bottlerocket variant, and arm64 for x86_64 if you're using Graviton-/Arm-based instances).

@markusboehme
Copy link
Member Author

@yann-soubeyrand A new point release of Bottlerocket (1.13.3) is in the works, including this fix.

@yann-soubeyrand
Copy link

Thanks @markusboehme!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ip6tables bug causing malfunctioning of host ports redirection on Kubernetes
6 participants