Skip to content
This repository has been archived by the owner on Oct 11, 2023. It is now read-only.

Set iptables rules at startup #446

Closed
moimael opened this issue Jul 21, 2015 · 26 comments
Closed

Set iptables rules at startup #446

moimael opened this issue Jul 21, 2015 · 26 comments

Comments

@moimael
Copy link

moimael commented Jul 21, 2015

Hi,

I use a direct routing load balancer to balance load against my rancher servers (https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Load_Balancer_Administration/s2-lvs-directrouting-VSA.html). One of them is under RancherOS and the only way (except using arptables, which isn't available on RancherOS) to make direct routing work is editing iptables rules. Is there a way to make them persistent across restart ?

Thanks !

@imikushin
Copy link
Contributor

I think there's a way: a privileged service container that runs on boot which executes a script applying the iptables rules.

iptables-rules:
  image: yourname/iptables-rules
  command: ./set-rules.sh  # can be omitted if set in Dockerfile with ENTRYPOINT or CMD
  net: host
  privileged: true
  labels:
    io.rancher.os.scope: system
    io.rancher.os.detach: false
    io.rancher.os.remove: true
  links:
  - network

This is a compose template for such a container. You can put it under under rancher.services in your cloud-config, or make it available via HTTPS and enable it with either cloud-config (under rancher.services_include) or with ros s enable <URL>

@gregory
Copy link

gregory commented Aug 10, 2015

@imikushin do you guys have documentation on all the labels and their meanings you guys are using?

@deniseschannon
Copy link

@gregory currently, there is no documentation on all the labels and their meanings, but we can look into adding it into our documentation. Created this issue, #476, to track it.

@gregory
Copy link

gregory commented Aug 10, 2015

thanks @deniseschannon !

@gregory
Copy link

gregory commented Aug 15, 2015

@moimael here below is an example where i used busybox run an init script at bootstrap:

#cloud-config
hostname: rancher-x
ssh_authorized_keys:
  - ssh-rsa ...[your ssh key]
write_files:
  - path: /opt/rancher/init/start.sh
    permissions: 0755
    content: |
      #!/bin/sh
      iptables-restore < /opt/rancher/init/rules-save
  - path: /opt/rancher/init/rules-save
    permissions: 0644
    owner: "root:root"
    content: |
      *filter
      :INPUT DROP [0:0]
      :FORWARD ACCEPT [0:0]
      :OUTPUT ACCEPT [0:0]
      :DOCKER - [0:0]
      -A INPUT -i lo -j ACCEPT
      -A INPUT -i eth1 -j ACCEPT
      -A INPUT -i docker0 -j ACCEPT
      -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
      -A INPUT -m conntrack --ctstate NEW -m multiport -p tcp --dports 22,80,443 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
      -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
      -A FORWARD -i docker0 -o eth1 -j ACCEPT
      -A FORWARD -i eth1 -o docker0 -j ACCEPT
      -A FORWARD -i eth0 -o docker0 -j ACCEPT
      -A FORWARD -i docker0 -o eth0 -j ACCEPT
      COMMIT
  - path: /etc/ssh/sshd_config
    permissions: 0600
    owner: "root:root"
    content: |
        UsePrivilegeSeparation sandbox
        Subsystem sftp internal-sftp
        PermitRootLogin no
        AllowUsers rancher
        PasswordAuthentication no
        ChallengeResponseAuthentication no
rancher:
  debug: true
  cloud_init:
    datasources:
      - ec2
  network:
    dns:
      nameserver:
        - 8.8.4.4
        - 4.2.2.3
  state:
    fstype: auto
    dev: LABEL=RANCHER_STATE
    autoformat:
      - /dev/vda
  services:
    init_script:
      image: busybox
      volumes:
        - /opt/rancher/init:/opt/rancher/init
      working_dir: /opt/rancher/init
      links:
        - network
      labels:
        - io.rancher.os.scope=system
      command: ./start.sh
      net: host
      privileged: true

@pwFoo
Copy link

pwFoo commented Sep 25, 2015

Hi @imikushin

your example at issuecomment-127362076 could be based on a RancherOS / os-service image? So existing image could be reused with correct file permissions and the same filesystem (base).

Regards

@quantverse
Copy link

@gregory tried your solution, but I have failed miserably

  1. the system fails to pull the busybox image, because network may not be available at the time when this service is started (solved by wait-for-network)
  2. /opt filesystem is not ready when booting first time (it fails on command not found)
  3. the recent version of busybox no longer contains iptables and iptables-restore commands
  4. when I use iptables from the host system within busybox, I get kernel panic
#cloud-config
hostname: rancher-os
ssh_authorized_keys:
  - ssh-rsa xxx
write_files:
  - path: /opt/rancher/init/iptables.sh
    permissions: "0755"
    owner: "root:root"
    content: |
      #!/bin/bash
      iptables -F
      iptables -P INPUT DROP
      iptables -P FORWARD ACCEPT
      iptables -P OUTPUT ACCEPT
      iptables -A INPUT -i lo -j ACCEPT
      iptables -A INPUT -i eth1 -j ACCEPT
      iptables -A INPUT -i docker0 -j ACCEPT
      iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      iptables -A INPUT -i eth0 -p tcp -m multiport --dports 2222 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
      iptables -A FORWARD -i docker0 -o eth1 -j ACCEPT
      iptables -A FORWARD -i eth1 -o docker0 -j ACCEPT
      iptables -A FORWARD -i eth0 -o docker0 -j ACCEPT
      iptables -A FORWARD -i docker0 -o eth0 -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type source-quench -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type echo-request -j ACCEPT
  - path: /home/rancher/.docker/config.json
    permissions: "0660"
    encoding: b64
    content: xxx
  - path: /etc/ssh/sshd_config
    permissions: "0644"
    owner: root:root
    content: |
      Port 2222
      AuthorizedKeysFile .ssh/authorized_keys
      UsePrivilegeSeparation sandbox
      ClientAliveInterval 180
      Subsystem sftp /usr/libexec/sftp-server
      UseDNS no
      PermitRootLogin no
      AllowGroups docker
rancher:
  cloud_init:
    datasources:
      - ec2
  network:
    dns:
      nameserver:
        - 8.8.4.4
        - 4.2.2.3
    interfaces:
      eth0:
        dhcp: true
      eth1:
        address: $V4_PRIVATE_IP/16
        mtu: 1450
  state:
   fstype: auto
   dev: LABEL=RANCHER_STATE
   autoformat:
     - /dev/vda
  services:
    init_script:
      image: busybox
      net: host
      uts: host
      pid: host
      ipc: host
      privileged: true
      volumes_from:
        - user-volumes
      volumes:
        - /sbin/iptables:/usr/bin/iptables:ro
      labels:
        io.rancher.os.scope: system
        io.rancher.os.after: wait-for-network, cloud-init
      command: /opt/rancher/init/iptables.sh

@quantverse
Copy link

Well at the end I just setup iptables rules in /opt/rancher/bin/start.sh. It is possibly incorrect, but seems to work (so far, on the first sight).

@pwFoo
Copy link

pwFoo commented Feb 2, 2016

A cloud config solution should be available after RancherOS 0.4.3 release.
#660

@deniseschannon
Copy link

This should be working in v0.4.4.

@sschueller
Copy link

sschueller commented May 6, 2016

How do we do this in v0.4.4.?

The following doesn't work

#cloud-config
write_files:
  - path: /var/lib/iptables/rules.sh
    permissions: "0755"
    owner: root:root
    content: |
      iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
      iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
      iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
      # the last line of the file needs to be a blank line or a comment
rancher:
  network:
    interfaces:
      eth0:
        dhcp: true
        post_up: 
        - bash /var/lib/iptables/rules.sh

@deniseschannon
Copy link

@sschueller The problem is that the file needs to be written in the network container so that the post_up can access it. We fixed this in v0.6.0 where you should be able to do this, where I added the container:network in the write_files so that the file is create in the network container.

#cloud-config
write_files:
  - container: network 
    path: /var/lib/iptables/rules.sh
    permissions: "0755"
    owner: root:root
    content: |
      iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
      iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
      iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
      # the last line of the file needs to be a blank line or a comment
rancher:
  network:
    interfaces:
      eth0:
        dhcp: true
        post_up: 
        - bash /var/lib/iptables/rules.sh

@camrossi
Copy link

Hi,

I have tried the above solution however it seems the network container do not comes with iptables.
/var/lib/iptables/rules.sh: line 1: iptables: command not found

what would be the best way to fix this issue?

Thanks!
Cam

@joshwget
Copy link
Contributor

@camrossi True, iptables is currently only available in the console. Can you file a separate issue for this?

@deoxxa
Copy link

deoxxa commented Jan 4, 2017

So I see this issue is closed, does that mean it's resolved? Or is this still not possible in rancher?

@joshwget
Copy link
Contributor

joshwget commented Jan 4, 2017

@deoxxa This is being tracked in #1362.

@deoxxa
Copy link

deoxxa commented Jan 4, 2017

Okay, so that's a no. Thanks.

@felixsanz
Copy link

So ... it's impossible to use iptables right now?

@quantverse
Copy link

Does not work in 1.0.0:

2017/04/16 11:26:07 Writing file to "/var/lib/iptables/rules.sh"
2017/04/16 11:26:07 Wrote file to "/var/lib/iptables/rules.sh"
> time="2017-04-16T11:26:07Z" level=info msg="Wrote file /var/lib/iptables/rules.sh to filesystem" 
> time="2017-04-16T11:26:07Z" level=info msg="Apply Network Config" 
Failed to write to log, write /dev/kmsg: invalid argument
> time="2017-04-16T11:26:07Z" level=info msg="Applying 127.0.0.1/8 to lo" 
> time="2017-04-16T11:26:07Z" level=info msg="Applying ::1/128 to lo" 
> time="2017-04-16T11:26:07Z" level=info msg="Applying 10.99.0.20/16 to eth1" 
> time="2017-04-16T11:26:07Z" level=info msg="Set 10.99.0.20/16 on eth1" 
> time="2017-04-16T11:26:07Z" level=info msg="removing  fe80::5800:ff:fe65:245/64 from eth1" 
> time="2017-04-16T11:26:07Z" level=info msg="Removed fe80::5800:ff:fe65:245/64 from eth1" 
> time="2017-04-16T11:26:07Z" level=info msg="Apply Network Config RunDhcp" 
> time="2017-04-16T11:26:07Z" level=info msg="Running DHCP on eth0: dhcpcd -MA4 --nohook resolv.conf eth0" 
> time="2017-04-16T11:26:07Z" level=error msg="exit status 1" 
> time="2017-04-16T11:26:08Z" level=info msg="Apply Network Config SyncHostname"

My network config:

write_files:
  - container: network
    path: /var/lib/iptables/rules.sh
    permissions: "0755"
    owner: root:root
    content: |
      #!/bin/bash
      iptables -F
      iptables -P INPUT DROP
      iptables -P FORWARD ACCEPT
      iptables -P OUTPUT ACCEPT
      iptables -A INPUT -i lo -j ACCEPT
      iptables -A INPUT -i eth1 -j ACCEPT
      iptables -A INPUT -i docker0 -j ACCEPT
      iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
      ... some rules for eth0
      iptables -A FORWARD -i docker0 -o eth1 -j ACCEPT
      iptables -A FORWARD -i eth1 -o docker0 -j ACCEPT
      iptables -A FORWARD -i eth0 -o docker0 -j ACCEPT
      iptables -A FORWARD -i docker0 -o eth0 -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type source-quench -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
      iptables -I INPUT -p icmp --icmp-type echo-request -j ACCEPT
      # the last line of the file needs to be a blank line or a comment
rancher:
  cloud_init:
    datasources:
      - ec2
  network:
    dns:
      nameservers:
        - 8.8.4.4
        - 4.2.2.3
    interfaces:
      eth0:
        dhcp: true
        post_up:
        - /var/lib/iptables/rules.sh
      eth1:
        address: $V4_PRIVATE_IP/16
        mtu: 1450
  state:
   fstype: auto
   dev: LABEL=RANCHER_STATE
   autoformat:
     - /dev/vda 

@SvenDowideit
Copy link
Contributor

so basically, this has never worked? reopening and I'll see what I can do for v1.0.2

@SvenDowideit SvenDowideit reopened this Apr 27, 2017
@SvenDowideit SvenDowideit added this to the v1.0.2 milestone Apr 27, 2017
@SvenDowideit
Copy link
Contributor

SvenDowideit commented May 19, 2017

so reading the code, it looks like the pre_up and post_up never worked for dhcp adresses - and there's no tests for any of it :(

I feel you should really be using rancher.network.post_cmds for your iptables script, but even that happens at the wrong time for DHCP:

rancher:
  cloud_init:
    datasources:
      - ec2
  network:
    dns:
      nameservers:
        - 8.8.4.4
        - 4.2.2.3
    pre_cmds:
    - echo "PRE_CMD"
    post_cmds:
    - echo "POST_CMD"
    interfaces:
      eth0:
        dhcp: true
        pre_up:
        - echo "PRE_UP eth0"
        post_up:
        - echo "POST_UP eth0"
        - /var/lib/iptables/rules.sh
      lo:
        dhcp: true
        pre_up:
        - echo "PRE_UP lo"
        post_up:
        - echo "POST_UP lo"
      eth1:
        address: 192.168.3.13/16
        mtu: 1450
        pre_up:
        - echo "PRE_UP eth1"
        post_up:
        - echo "POST_UP eth1"
[root@rancher-dev rancher]# system-docker logs network
DEBU[0054] Calling GET /v1.23/containers/network/json   
DEBU[0054] Calling GET /v1.23/containers/network/logs?stderr=1&stdout=1&tail=all 
DEBU[0054] logs: begin stream                           
DEBU[0054] logs: end stream                             
2017/05/19 05:16:52 Writing file to "/var/lib/iptables/rules.sh"
2017/05/19 05:16:52 Wrote file to "/var/lib/iptables/rules.sh"
> time="2017-05-19T05:16:52Z" level=info msg="Wrote file /var/lib/iptables/rules.sh to filesystem" 
> time="2017-05-19T05:16:52Z" level=info msg="Apply Network Config" 
> time="2017-05-19T05:16:52Z" level=debug msg="Config: &netconf.NetworkConfig{PreCmds:[]string{\"echo \\\"PRE_CMD\\\"\"}, DNS:netconf.DNSConfig{Nameservers:[]string{\"8.8.4.4\", \"4.2.2.3\"}, Search:[]string(nil)}, Interfaces:map[string]netconf.InterfaceConfig{\"lo\":netconf.InterfaceConfig{Match:\"\", DHCP:true, DHCPArgs:\"\", Address:\"\", Addresses:[]string(nil), IPV4LL:false, Gateway:\"\", GatewayIpv6:\"\", MTU:0, Bridge:\"\", Bond:\"\", BondOpts:map[string]string(nil), PostUp:[]string{\"echo \\\"POST_UP lo\\\"\"}, PreUp:[]string{\"echo \\\"PRE_UP lo\\\"\"}, Vlans:\"\"}, \"eth0\":netconf.InterfaceConfig{Match:\"\", DHCP:true, DHCPArgs:\"\", Address:\"\", Addresses:[]string(nil), IPV4LL:false, Gateway:\"\", GatewayIpv6:\"\", MTU:0, Bridge:\"\", Bond:\"\", BondOpts:map[string]string(nil), PostUp:[]string{\"echo \\\"POST_UP eth0\\\"\", \"/var/lib/iptables/rules.sh\"}, PreUp:[]string{\"echo \\\"PRE_UP eth0\\\"\"}, Vlans:\"\"}, \"eth1\":netconf.InterfaceConfig{Match:\"\", DHCP:false, DHCPArgs:\"\", Address:\"192.168.3.13/16\", Addresses:[]string(nil), IPV4LL:false, Gateway:\"\", GatewayIpv6:\"\", MTU:1450, Bridge:\"\", Bond:\"\", BondOpts:map[string]string(nil), PostUp:[]string{\"echo \\\"POST_UP eth1\\\"\"}, PreUp:[]string{\"echo \\\"PRE_UP eth1\\\"\"}, Vlans:\"\"}}, PostCmds:[]string{\"echo \\\"POST_CMD\\\"\"}, HTTPProxy:\"\", HTTPSProxy:\"\", NoProxy:\"\"}" 
Failed to write to log, write /dev/kmsg: invalid argument
> time="2017-05-19T05:16:52Z" level=debug msg="Apply pre call runCmds(): [echo \"PRE_CMD\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmds(on ): [echo \"PRE_CMD\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmd(on ): echo \"PRE_CMD\"" 
> time="2017-05-19T05:16:52Z" level=info msg="Running command echo [PRE_CMD]" 
PRE_CMD
> time="2017-05-19T05:16:52Z" level=debug msg="Config(eth1): netconf.InterfaceConfig{Match:\"eth1\", DHCP:false, DHCPArgs:\"\", Address:\"192.168.3.13/16\", Addresses:[]string(nil), IPV4LL:false, Gateway:\"\", GatewayIpv6:\"\", MTU:1450, Bridge:\"\", Bond:\"\", BondOpts:map[string]string(nil), PostUp:[]string{\"echo \\\"POST_UP eth1\\\"\"}, PreUp:[]string{\"echo \\\"PRE_UP eth1\\\"\"}, Vlans:\"\"}" 
> time="2017-05-19T05:16:52Z" level=info msg="Applying 192.168.3.13/16 to eth1" 
> time="2017-05-19T05:16:52Z" level=debug msg="call pre runCmds(on eth1): [echo \"PRE_UP eth1\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmds(on eth1): [echo \"PRE_UP eth1\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmd(on eth1): echo \"PRE_UP eth1\"" 
> time="2017-05-19T05:16:52Z" level=info msg="Running command echo [PRE_UP eth1]" 
PRE_UP eth1
> time="2017-05-19T05:16:52Z" level=debug msg="call pre runCmds(on eth1): [echo \"POST_UP eth1\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmds(on eth1): [echo \"POST_UP eth1\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmd(on eth1): echo \"POST_UP eth1\"" 
> time="2017-05-19T05:16:52Z" level=info msg="Running command echo [POST_UP eth1]" 
POST_UP eth1
> time="2017-05-19T05:16:52Z" level=debug msg="Apply post call runCmds(): [echo \"POST_CMD\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmds(on ): [echo \"POST_CMD\"]" 
> time="2017-05-19T05:16:52Z" level=debug msg="runCmd(on ): echo \"POST_CMD\"" 
> time="2017-05-19T05:16:52Z" level=info msg="Running command echo [POST_CMD]" 
POST_CMD
> time="2017-05-19T05:16:52Z" level=info msg="Apply Network Config RunDhcp" 
> time="2017-05-19T05:16:52Z" level=debug msg=RunDhcp 
> time="2017-05-19T05:16:52Z" level=info msg="Running DHCP on eth0: dhcpcd -MA4 -e force_hostname=true --nohook resolv.conf eth0" 
> time="2017-05-19T05:16:52Z" level=error msg="exit status 1" 
> time="2017-05-19T05:16:52Z" level=debug msg="dhcpcd -u eth1: " 
> time="2017-05-19T05:16:53Z" level=info msg="Apply Network Config SyncHostname" 

Still hoping to fix it today tho.

@SvenDowideit
Copy link
Contributor

ok, looks like the proper fix is too complicated to 1.0.2 - it'll come out with 1.1.0

however - @felixsanz @quantverse @deoxxa rancher.network.post_cmds should work for you currently?

can you please have a go and tell me if it does/not?

@SvenDowideit SvenDowideit modified the milestones: v1.1.0, v1.0.2 May 19, 2017
@deoxxa
Copy link

deoxxa commented May 21, 2017

@SvenDowideit can't test it - gave up on Rancher months ago. Sorry!

@stszap
Copy link

stszap commented May 24, 2017

How can I allow access to specific port (which is forwarded to container) only for specific ip? I tried cloud-config like this:

#cloud-config

write_files:
  - container: network
    path: /var/lib/iptables/rules.sh
    permissions: "0755"
    owner: root:root
    content: |
      #!/bin/bash
      iptables -F
      iptables -I FORWARD -i eth0 -p tcp --dport 1234 -s 8.8.8.8 -j ACCEPT
      # the last line of the file needs to be a blank line or a comment
rancher:
  network:
    interfaces:
      "mac=08:00:27:78:2e:5c":
        gateway: 192.168.99.1
        address: 192.168.99.100/24
        dhcp: false
        post_up:
        - /var/lib/iptables/rules.sh

After system starts 'iptables -L FORWARD -nv' shows:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2454  206K DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 1227  103K DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 1227  103K ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 1227  103K ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  eth0   *       8.8.8.8              0.0.0.0/0            tcp dpt:1234

So my rule is useless (because it's at the bottom). Docker will release a solution soon moby/libnetwork#1675, but will it work with Rancher os?

@stszap
Copy link

stszap commented May 26, 2017

After rereading the documentation I managed to solve my problem with config like this (just an example, not an actual config):

write_files:
  - path: /etc/rc.local
    permissions: "0755"
    owner: root
    content: |
      #!/bin/bash
      wait-for-docker
      iptables -I FORWARD -s 8.8.8.8 -j ACCEPT

I also tested the following config in VirtualBox (with dhcp enabled, Rancher os v1.0.2-rc2):

rancher:
  network:
    post_cmds:
    - iptables -I INPUT -s 8.8.8.8 -j ACCEPT

And after booting 'iptables -L INPUT -nv' showed

Chain INPUT (policy ACCEPT 1017 packets, 73428 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      *       8.8.8.8              0.0.0.0/0

So it seems that post_cmds works with dhcp at least in my case. I hope this helps.

@SvenDowideit
Copy link
Contributor

this should me more reliable in v1.1.0 due to #1869

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests