vector-agent with multiple vector_sinks

[jsbloo]

Hello,
I have a vector agent, I want it to have N vector_sinks.
The vector port on my vector aggregators is 6000,
when I have a config like (in my agent customConfig):

sinks:
  vector_sink_1:
    type: vector
    inputs: [k8_logs]
    address: vector.ns1.svc.cluster.local:6000
  vector_sink_2:
    type: vector
    inputs: [k8_logs]
    address: vector.ns2.svc.cluster.local:6000

I get a fatal kubernetes error , saying that I cannot have duplicate port numbers?
Seems the way the helm-chart is configured is wrong? I can maybe just ignore the error, but it worries me.

It still deploys and seems to work, but how do I get around it? I cannot see ANY examples, showing this usecase am I doing something wrong?

[jszwedko]

Can you show the actual error that Kubernetes is reporting to you?

[jsbloo]

stderr: Error: UPGRADE FAILED: failed to create resource: Service "vector"\ is invalid: spec.ports[2]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:6000, TargetPort:IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}

[jszwedko]

Interesting. And you see this on the clients and not the aggregator, correct? That is: the instances that are running the config you shared above with the vector sinks and not the ones with the vector source.

[jsbloo]

Yeah, so this example config is for my vector agent. I want that to forward
onto my aggregators on several namespaces, the actual config I have is
programmatic, but I've hard coded it like this and got the same error.

The aggregators just listen on 0.0.0.0:6000 for the incoming logs, weirdly
even with throwing this error they seem to be recieving the logs just fine

[jszwedko]

Can you share your full values.yaml? That would be helpful for reproducing.

[jsbloo]

Yeah, so this example config is for my vector agent. I want that to forward onto my aggregators on several namespaces, the actual config I have is programmatic, but I've hard coded it like this and got the same error. The aggregators just listen on 0.0.0.0:6000 for the incoming logs, weird even with throwing this error they seem to be recieving the logs

…

On Mon, 13 Jan 2025, 18:22 Jesse Szwedko, ***@***.***> wrote: Interesting. And you see this on the clients and not the aggregator, correct? That is: the instances that are running the config you shared above with the vector sinks and not the ones with the vector source. — Reply to this email directly, view it on GitHub <#22189 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD265NG6BKHYSGFU2HWLZ2T2KP775AVCNFSM6AAAAABVCUEND6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBSGMZTONQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jsbloo]

@jszwedko Hi sorry for the late reply, do you have any idea? How can I sink to multiple namespaces? thanks

[jszwedko]

Hey! Apologies for the delayed response. I think this is a bug in in the Helm chart that templates ports based on the Vector config. It thinks that the Vector sink is binding to the ports, when it isn't. This logic is encoded here: https://github.com/vectordotdev/helm-charts/blob/d11451c72148735d9fd2cb4a4a250e913663386e/charts/vector/templates/_helpers.tpl#L144-L159

As a workaround, you can manually configure the ports in your Helm values.yaml (as containerPorts) rather than have them be generated.