Documents for romana on multipe subnets

[nightmeng]

I have install ramana on my k8s cluster according to the following document:

https://github.com/romana/romana/tree/master/containerize

The document says:

An automated setup only supports a single subnet in this release. Multiple subnets are supported, but automating that configuration is coming in a future release.

But I can not find any document describes romana on multiple subnets, where I can fint it?

[chrismarino]

Hi. Support for multiple subnets gets really complicated really fast. The installer doesn't do that and we're in the middle of some changes to make that possible. Unfortunately, it won't be ready for several weeks. And I'm afraid that until then there are not any simple alternatives.

Are you running in ASW VPCs? Or in your own datacenter?

[nightmeng]

Thank you very much, I'm running romana in my own datacenter, not AWS.
Does romana support cross-subnet communication by IPinIP， just like calico？
Before the cross-subnet version release, is there another way to support
cross-subnet?

[chrismarino]

Hi, Romana does not have an ip-ip mode. Right now we are working on and testing an agent that will announce the host stub networks up to ToRs. Absent that, you could add the routes manually. Building tunnels would work as well, but thats going to be a lot more work. Perhaps we can get you the new agent to try out. Let me look in to that for you.

[flashvoid]

Hello @nightmeng, we have development branch with BGP speaker enabled on every kubernetes node that can be configured to advertise it's routes to some BGP capable router that would handle cross-subnet communication.
Alternatively, BGP router can be enabled on every k8s node, that would enable cross-subnet communication without a dedicated router.

Would that satisfy your requirements?

[nightmeng]

@chrismarino @flashvoid Yes, it satisfies my requirements, and I think it is a great design!
I can not add routes manually now, but I want to try the development branch. Which branch is the development branch you menthioned above?
Thank you.

[flashvoid]

@nightmeng it's not visible in public repo, I will build the binaries and we need to discuss how to pass it on to you, probably will build special containers or something else.

[nightmeng]

Containers please, thank you!

[flashvoid]

Hello @nightmeng, we're finished containers and installer changes for bgp announcement and are ready to assist you in a deployment scenario.

below is a copy of our internal instruction that applies to containers with enable-bird tag.

you should also receive an invite into our internal slack

---

Note: experimental changes.

This feature lets a user configure the agent to use bird as a route publisher.
They need to do a few steps to make it work

1. Use a different yaml file.
2. Provide a config map containing at least two files.

The yaml changes are:

• add another item to the args section of services: - --x-route-publisher=bird
• add another item to the args section of agent: - --x-route-publisher=bird
• add another mountPath to the agent

        - name: route-publisher-bird
          mountPath: /var/lib/romana/route-publisher/bird

• connect that mountPath to a configmap

      - name: route-publisher-bird
        configMap:
          name: romana-route-publisher-bird

The configmap should contain one file that configures bird, and another to provide a host or network-specific template.
The first file must be named bird.conf and an example is:

protocol kernel {
	scan time 60;
	import none;
}

protocol device {
	scan time 60;
}

include "conf.d/*.conf";

The last line is really important.

The other file(s) should be named as one of the following:

• host ip address + ".conf", eg: 192.168.99.10.conf
• host network address + ".conf", eg: 192.168.99.0.conf
• default.conf
  This file is a Go template that is used by agent to inject new bgp configuration.
  The first matching file will be used and configured for the agent.

An example (that doesn't work perfectly) as a network-specific config is: 192.168.99.0.conf

protocol static static_bgp {
	{{range .Networks}}
	route {{.}} reject;
	{{end}}
}

protocol bgp bgp_def {
	export where proto = "static_bgp";
	local as {{.LocalAS}};
	neighbor 192.168.99.10 as {{.LocalAS}};
}

(This doesn't work right because that neighbor isn't really a proper BGP neighbor.)
To create the configmap, create the files in an otherwise empty directory (eg: ~/bird, and run: kubectl -n kube-system create configmap --from-file=bird/ romana-route-publisher-bird