Calico with Kubernetes and Digital Ocean firewalls

[dghubble]

Kubernetes clusters can use Calico for container networking on Digital Ocean if traffic uses IPv4 encapsulation (ipip protocol i.e. IP protocol 4). However, creating a Digital Ocean firewall blocks such traffic and firewall rules for protocols other than TCP, UDP, and ICMP aren't supported. In practice, this means its not very reasonable to use Calico on Digital Ocean.

Calico tutorials for DO are getting around this by simply never enabling firewall rules on droplets (!). On AWS and Google Cloud, their firewalls allow rules for the "ipip" protocol so there is no issue. I've spoken with DO and requested support (initial issue and vote for the feature)

Possible Solution

In the mean time, are there any alternative configurations folks have successfully used on Digital Ocean? According to #315, Digital Ocean does not allow peering with its networking so IP encapsulation may be the only choice, but the author mentions "there should be no problem". Recommendations?

Steps to Reproduce

1. Launch Kubernetes cluster on Digital Ocean with Calico
2. Delete firewall
3. Verify pod to pod connectivity works
4. Enable firewall again. Pod to pod connectivity no longer works. There are no firewalls you can add to allow the traffic.

Context

Aim is to have Calico be the default for our Kubernetes clusters across cloud providers. Still using flannel on DO due to this.

Your Environment

• Calico version: 2.5.1
• Orchestrator version: Kubernetes v1.7.3
• Operating System and version: CoreOS 1409.9.0
• Link to your project: https://github.com/poseidon/typhoon

rel:

• Digital Ocean firewalls don't allow IP-tunneling protocol for Calico poseidon/typhoon#11
• https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
• https://tools.ietf.org/html/rfc2003

cc @ozdanborne @gunjan5

[ozdanborne]

@dghubble thanks for submitting. My initial thought is that you could use Flannel for networking, as it uses VXLAN which runs over udp. The canal project bundles flannel with Calico for Kubernetes. Hopefully DO won't block packets when using that.

[dghubble]

Yeah, flannel with vxlan for networking and Calico for policy is what we're using on Digital Ocean currently. https://typhoon.psdn.io/digital-ocean/#optional Its ok for now, but my goal is to get to Calico networking at some point. DO did accept the filed "idea", saying there are several requests for additional protocols they want to add.

[lucasyvas]

@dghubble Thanks a lot for reporting this. I spent hours trying to figure out what the hell was going on today. I have a very simple firewall that wraps all the Kubernetes nodes by tag. I presume switching to Canal may just magically work since my UDP traffic is wide open between my droplets... I'll find out soon enough!

[wardviaene]

Had the same issue. Switching to Canal indeed just works. Calico only will only work if DigitalOcean adds a feature to their firewall to enable IP-in-IP.

[caseydavenport]

Calico v3.7 will have native VXLAN capabilities: projectcalico/felix#1989

So, won't need to use canal for this - will be similar configuration model as IPIP is today.