feat(new source): Inital kubernetes source implementation

[ktff]

Implements 1. Decentralized configuration part of #260.

Implementation differs from showed example in:

• Instead of passing vector configuration as argument, vector configuration is mounted as vector.toml file, and it's path passed as argument.

• include_namespace option was added as it proved to be necessary in tests. (Edit: removed because it's implementation relied on unspecified behavior)

Tests are currently behind kubernetes-integration-tests feature flag as they require:

• Access to Kubernetes cluster.

• Tested vector image present on Kubernetes hosts. Currently they are pulling vector image from my repository.

Before merger, find a way to:

• spin up local Kubernetes cluster
• build/find tested vector image and copy it to every Kubernetes host
• enable tests
• test and accommodate, where possible, other non Docker container runtimes. Only distributed.yaml can be possibly changed by this. This is required mostly because *.log files ,which are being read, are symlinks to other files. Which means that they also have to be mounted to be accessible.

[LucioFranco]

@ktff Looking great! I will give this a proper review on Monday. For testing, I feel like setting up kube in our current test-stable job might be too much. So maybe we want to see if we can setup a specific test stage to spin up minikube and just run the kube tests.

[ktff]

For testing, I feel like setting up kube in our current test-stable job might be too much. So maybe we want to see if we can setup a specific test stage to spin up minikube and just run the kube tests.

@LucioFranco I agree.

The stage should have installed kubectl and minikube. If possible, minikube that is running directly on host so that we can avoid step 2. That would also eliminate the need for virtualization. There are guides for that in Kubernetes documentation:

• kubectl
• minikube

Then during testing stage:

1. build/find tested vector image
2. copy it to every Kubernetes host. (this could be avoided)
3. minikube start
4. run tests
5. minikube delete

[LucioFranco]

@ktff looks like there are already a few issues and people have solved adding minikube to circleci. I would like to see this added before we merge here as well as some docs. I will give the main code a review today.

[LucioFranco]

This is looking really good!! Left some comments inline.

[LucioFranco]

@binarylogic I think the docs are ready to go. I would like to write more thorough guides for kubernetes but I don't think the docs need much more since right now this implementation is simple.

All that is left is your approval and the new svg.

[binarylogic]

@LucioFranco we need to improve the docs quite a bit more. For example, there is no Kubernetes platform installation page. This is an important enough source that it warrants being thoughtful about the documentation and setup experience. I would like to review and work on the docs before we merge.

[binarylogic]

Can you enlighten me how a developer would run these tests locally? Are there docs for this? Even better, is there a way to embed this into the current workflow?

[LucioFranco]

All of the kubernetes tests are behind a kubernetes-integration-tests feature flag and not enabled by default.

minikube start
cargo test --features kubernetes-integration-tests

It will use your kube config that exists in ~/.kube/config. Minikube will place one for you or you can run this command to fetch the config from an existing cluster. It just needs some kube cluster.

curl http://<some kube cluster>/config > ~/.kube/config

The idea here is that if you have access to a kube cluster via kubectl then you can run the tests.

[binarylogic]

Where is this documented?

[LucioFranco]

Happy to add this to the contributing guide, will do that tomorrow. We can also provide a separate test command to test with minikube.

[LucioFranco]

@binarylogic I'm not sure what setup there is? Most of the setup stuff I can think about would involve being a guide since they would need to talk about deployment strategies. So I was not sure what direction we wanted to take these docs versus deployment strategies.

For example, there is no Kubernetes platform installation page

I'm not sure what you mean by this, do we have any examples of a platform installation page? I tried to follow how the other sources documented things.

Another thing to note is that this source only works if you run it as a DaemonSet and not as a sidecar which we want to support as another deployment strategy. So, in theory, this may not be the only source that users use in a kubernetes deployment. So because of that, I'm not sure this should contain our high-level kubernetes guides/setup but be a reference point for using this source with a per-node deployment style.

[binarylogic]

Remove this for now.

[binarylogic]

I don't think this belongs here, this is also hard to follow. Shouldn't we provide commands that user can copy/paste to quickly get setup?

[binarylogic]

Just providing an update. I have merged in all of the upstream changes from master. This includes the new website/documentation. I am working on the installation docs as well as the use case page for Kubernetes so that we can announce it properly.

[binarylogic]

Nice work on this! I'm going to merge so that we can unblock work. Documentation is being worked on.