Cilium Operator panics when starting without Kubernetes #24767
Comments
EtienneM
added
kind/bug
|
I can see that the code which panics has been added in #21764. Could it be possible that the use case without Kubernetes was not taken into account at the time? |
|
Given that I authored #21764, I can confidently say that this is not something I took into account. The ability of the operator to function without kubernetes is not documented or very well known for that matter. Given that we implicitly accepted this requirement in the past, I think we should continue to unless otherwise decided. But to be honnest, I don't know what the operator is supposed to do without api-server. A few other tings of note:
It would be good to also add a test to the CI which starts the operator outside of k8s to assert that works so we don't regress on this again. |
dylandreimerink
removed
the
needs/triage
|
Thanks a lot for your answer :)
Thanks for the warning. We are not interested in multi-cluster setup so that's OK for us. Our use case would be: We have multiple customers hosting apps and databases in a single cluster. We want all apps and databases from one customer to be in some kind of private network (e.g. using VXLAN), isolated from the other customers apps and databases. The traffic between apps and databases should be encrypted.
Yes we used it as a base but are aware that some things may be deprecated or no relevant anymore. But for us that's already a good starting point :) |
Sorry, I meant to say multi node setup. Without kubernetes you are missing a lot of inter node communication which a lot of feature need. I believe the L4 LB case is the only scenario we currently support. Even for that we have had to add the ability to configure cilium via the CLI/API. So you might need to make some changes to get everything to do what you want. |
|
If that's the case it would be a really sad news. Moreover a setup without Kubernetes is advertised on the website and in the documentation so we expect it to work. That being said, we still think that it may work. We found various mentions of running Cilium without Kubernetes in GitHub issues (e.g. #18334), and a comprehensive answer from @joestringer seems to indicate that it could work with the CNI plugin (#18334 (comment)). Hence we expect that with some extra glue, we could make our setup work. |
|
I think that if the community sees value in Cilium use cases without k8s, then it's up to those community members to propose patches to support those cases, develop test cases to avoid regressions, and so on. Currently the core Cilium team is primarily focused on k8s environments so we can expect those cases to have the best support. That said, if there are sufficient community members interested in other platforms then the code should be (made to be) generic enough to support those platforms. I would say that the CI today doesn't provide any guarantees that Cilium works outside k8s environments, otherwise Dylan's submission would have failed that test and it would have been updated to consider this. But given that a few community members have been able to run Cilium without k8s with just a few tweaks, I think that the code is not that far from that capability. At least so far, I haven't seen any fundamental decisions that differ or maintenance burden that the core Cilium team needs to take on in order to support this, we're just relying on developers who care to submit the patches to make it work. |
|
One more note, while we do commonly rely on Kubernetes for defining the configuration schemas for Cilium functionality, Kubernetes is in no way a requirement for multi-node connectivity. Before Cilium supported Kubernetes, it already did a bunch of state sharing across nodes directly via etcd. We continue to support that deployment case in core Cilium (supplemented by k8s state distribution for various features like network policies). |
|
This issue has been automatically marked as stale because it has not |
github-actions
bot
added
the
stale
|
This issue has not seen any activity since it was marked stale. |
Is there an existing issue for this?
What happened?
We are trying to use Cilium with Docker containers without Kubernetes. I explained our setup in this Slack message.
For now, I'm just working on a proof of concept:
On my laptop, I start a three nodes etcd cluster with no TLS and no authentication.
I'm also starting a VM (using Vagrant). In it I start 3 Docker containers using Docker Compose: an agent, a Docker plugin and an operator.
This is inspired from this documentation page from Cilium 1.9.
I noticed the PR #21344 which adds the
enable-k8soption. It brings confidence about having a working proof of concept for this setup.Docker Compose configuration file
The operator panics when starting:
Operator logs
I'm not sure to understand why there is a call to a function which seems useful for Kubernetes:
The two other containers (agent and Docker plugin) successfully start:
Agent logs
Docker plugin logs
Vagrantfile
Cilium Version
Client: 1.13.1 a6be57e 2023-03-15T19:39:01+01:00 go version go1.19.6 linux/amd64
Daemon: 1.13.1 a6be57e 2023-03-15T19:39:01+01:00 go version go1.19.6 linux/amd64
Kernel Version
Linux ubuntu-focal 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Kubernetes Version
n/a
Sysdump
No response
I'm still working on figuring out how to make this work.
Relevant log output
No response
Anything else?
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: