Replies: 92 comments 48 replies
-
|
I can validate that this works properly for me with Traefik 2.2 in a 3 node Ubuntu cluster with |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond I don't believe so. However, I'm using a managed cloud k3s service. |
Beta Was this translation helpful? Give feedback.
-
|
Just 3 bare metal nodes running k3s on Ubuntu 19.10. Flannel is in host-gw mode, using Traefik 2.2 for ingress, and MetalLB in bgp mode for external services. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @brandond |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond , any update on this? |
Beta Was this translation helpful? Give feedback.
-
|
@jawabuu I'm now using Traefik 3.1 with the KubernetesCRD provider here's my traefik service: ---
apiVersion: v1
kind: Service
metadata:
name: traefik
namespace: traefik
labels:
app: traefik
chart: "traefik-3.1.0"
release: "traefik"
heritage: "Helm"
spec:
type: LoadBalancer
selector:
app: traefik
release: traefik
ports:
- port: 80
name: web
targetPort: "web"
- port: 443
name: websecure
targetPort: "websecure"
- port: 9000
name: traefik
targetPort: "traefik"
loadBalancerIP: 10.0.3,80
loadBalancerSourceRanges:
- "0.0.0.0/0"
externalTrafficPolicy: LocalFor metallb, I found that using bgp works best. I have a Ubiquiti USG as my router, and set it up to peer with all three of my nodes. ---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
peers:
- peer-address: 10.0.1.1
peer-asn: 64512
my-asn: 64512
hold-time: 120s
address-pools:
- name: manual
auto-assign: false
protocol: bgp
addresses:
- 10.0.3.10-10.0.3.99
- name: default
auto-assign: true
protocol: bgp
addresses:
- 10.0.3.100-10.0.3.254 |
Beta Was this translation helpful? Give feedback.
-
|
Lucky me, I also use the Ubiquiti USG as my router so I am also interested how to peer with all three of your nodes. |
Beta Was this translation helpful? Give feedback.
-
|
Here's what {
"protocols": {
"bgp": {
"64512": {
"neighbor": {
"10.0.1.20": {
"remote-as": "64512"
},
"10.0.1.21": {
"remote-as": "64512"
},
"10.0.1.22": {
"remote-as": "64512"
}
},
"parameters": {
"router-id": "10.0.1.1"
}
}
}
},
"service": {
"dns": {
"forwarding": {
"cache-size": "10000",
"except-interface": [
"eth0"
],
"options": [
"filterwin2k",
"local-ttl=60",
"host-record=usg.khaus,10.0.1.1",
"host-record=sw-core.khaus,10.0.1.2",
"host-record=ap-house.khaus,10.0.1.6",
"host-record=ap-garage.khaus,10.0.1.7",
"host-record=seago.khaus,10.0.1.20",
"host-record=maersk.khaus,10.0.1.21",
"host-record=sealand.khaus,10.0.1.22",
"srv-host=_etcd-client._tcp,seago.khaus,2379",
"srv-host=_etcd-client._tcp,maersk.khaus,2379",
"srv-host=_etcd-client._tcp,sealand.khaus,2379",
"srv-host=_etcd-server._tcp,seago.khaus,2380",
"srv-host=_etcd-server._tcp,maersk.khaus,2380",
"srv-host=_etcd-server._tcp,sealand.khaus,2380",
"server=/k3s.khaus/10.0.3.53",
"server=75.75.75.75",
"server=75.75.76.76"
]
}
}
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond I have narrowed down the issue - as best as I could :-) to using flannel as CNI. Calico works with Traefik 2.2 and MetalLB to resolve Real Client IP without any further configuration. |
Beta Was this translation helpful? Give feedback.
-
|
@jawabuu Those comments all seem to be about routing issues, not obtaining the original client IP? Either way, I'm using flannel in host-gw mode with traefik and metallb and getting the original address in the headers, so I know that it is doable. All you have to do is use |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond you are correct. The comments are just meant to provide context that the default installation of the latest k3s with flannel may be having some issues.
Ideally points 2 and 3 should be tested to identify edge cases. |
Beta Was this translation helpful? Give feedback.
-
|
thanks to both of you @brandond and @jawabuu |
Beta Was this translation helpful? Give feedback.
-
|
Hey @brandond, upon further testing with flannel, I have found out that it is not necessary to use |
Beta Was this translation helpful? Give feedback.
-
|
i have the same issue using |
Beta Was this translation helpful? Give feedback.
-
|
Hey @mschneider82 You will not be able to get source ip using the embedded flannel in k3s. containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq=false
- --kube-subnet-mgr
- --iface=${interface} |
Beta Was this translation helpful? Give feedback.
-
|
yes - its a production setup with AWS spot instances setup.
…On Mon, Apr 5, 2021 at 12:42 PM jawabuu ***@***.***> wrote:
@sandys <https://github.com/sandys> I am interested in how you are using
k3s in the cloud.
Is this a production setup?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYU2B4NTJEFQD7HOV6TTTHFPHXANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
How do you do HA? Do you have virtual IPs? And if yes, how are you announcing them? I have done that outside of K3s with Vultr, they are offering a BGP-based setup. I suppose one could get that to work with K3s as well using MetalLB in BGP mode. |
Beta Was this translation helpful? Give feedback.
-
|
@sandys That's impressive!! Any challenges to watch out for so far? |
Beta Was this translation helpful? Give feedback.
-
|
How do you do HA? Do you have virtual IPs? And if yes, how are you
announcing them?
yes - we do outside of k3s. using AWS load balancer.
I'm less keen on managing the HA system outself. I'd rather pay some more
money to AWS.
Any challenges to watch out for so far?
source ip. you have to uninstall klipper and use haproxy ingress for proxy
protocol support.
AWS integration - you have to set this up yourself. this is funky. cluster
autoscaling, cloud controller, etc. This is not at all trivial to setup.
https://itnext.io/how-to-integrate-k3s-with-the-cloud-controller-36bd5020b8f7
Calico. Flannel has problems. #1880
…On Mon, Apr 5, 2021 at 1:28 PM jawabuu ***@***.***> wrote:
@sandys <https://github.com/sandys> That's impressive!! Any challenges to
watch out for so far?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYU3GQU67RNHSAF7N2H3THFUQ5ANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
I suppose that the AWS load balancer then preserve Client IP, so that HAProxy Ingress can set those headers. Do you know how that works, i. e. how the AWS load balancer passes Client IP to HAProxy Ingress? I suspect that not every cloud LB will preserve Client IP, so this is another potential pitfall to be aware of. Specifically, if your cloud LB terminates SSL, it can freely manipulate all data of the original request. |
Beta Was this translation helpful? Give feedback.
-
|
@realulim I have a POC that tests k3s with a number of cloud providers. |
Beta Was this translation helpful? Give feedback.
-
|
AWS LB sends the Client IP as proxy protocol headers - which everything and
everyone supports. Including Traefik. Haproxy also supports it.
Every cloud LB does this. u can check my bug that i filed on kube-vip for
this. kube-vip/kube-vip#184
…On Mon, Apr 5, 2021 at 2:31 PM Ulrich Mayring ***@***.***> wrote:
I suppose that the AWS load balancer then preserve Client IP, so that
HAProxy Ingress can set those headers. Do you know how that works, i. e.
how the AWS load balancer passes Client IP to HAProxy Ingress?
I suspect that not every cloud LB will preserve Client IP, so this is
another potential pitfall to be aware of. Specifically, if your cloud LB
terminates SSL, it can freely manipulate all data of the original request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYU3E73DKUWE3SNJXNKDTHF4ABANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
You have been mentioning before that the HAProxy Ingress not only reads the Proxy Protocol headers, but can also set them, if they are not present. What would be a scenario, where I use this feature? |
Beta Was this translation helpful? Give feedback.
-
|
If you use haproxy as the load balancer. We did this for a while. Works
great actually. Very few load balancer/ingress can actually set proxy
protocol headers. ( That's why I filed the issue on kube-vip to incorporate
support for proxy protocol injection.)
Haproxy is a fully battle tested ingress+lb.
Then we moved to AWS built in lb.
…On Mon, 5 Apr, 2021, 15:40 Ulrich Mayring, ***@***.***> wrote:
You have been mentioning before that the HAProxy Ingress not only reads
the Proxy Protocol headers, but can also set them, if they are not present.
What would be a scenario, where I use this feature?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYU2SFA4J7XEIZL7AYHDTHGECTANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I suppose you mean HAProxy Ingress, which then would mean no HA or load balancing, because you always have to point to one node. Which is why you moved to AWS LB. If HAProxy external LB, then you would set the Proxy Protocol headers there and could also use Traefik or NGINX Ingress, since they can also read the headers. I haven't quite gotten my head around how Kube-VIP is supposed to work, but I have upvoted :) |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I
suppose you mean HAProxy Ingress, which then would mean no HA or load
balancing, because you always have to point to one node. Which is why you
moved to AWS LB.
either ways (and yes - ingress would mean no HA) - you will get proxy
protocol headers with haproxy.
If HAProxy external LB, then you would set the Proxy Protocol headers
there and could also use Traefik or NGINX Ingress, since they can also read
the headers.
yes you could. But no use really having a different software for lb and a
different one for ingress. Haproxy is just as impressive as traefik in
features...and possibly more battle tested.
I haven't quite gotten my head around how Kube-VIP is supposed to work,
but I have upvoted :)
kube-vip is a replacement for klipper (the built in LB in k3s). Its fairly
well supported. Just like MetalLB basically.
…On Mon, Apr 5, 2021 at 4:04 PM Ulrich Mayring ***@***.***> wrote:
Do you mean HAProxy external LB or HAProxy Ingress as LB within K3s? I
suppose you mean HAProxy Ingress, which then would mean no HA or load
balancing, because you always have to point to one node. Which is why you
moved to AWS LB.
If HAProxy external LB, then you would set the Proxy Protocol headers
there and could also use Traefik or NGINX Ingress, since they can also read
the headers.
I haven't quite gotten my head around how Kube-VIP is supposed to work,
but I have upvoted :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2997 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAASYUYZ2NCL7ORVB5R5MM3THGG25ANCNFSM4YMJZ6EQ>
.
|
Beta Was this translation helpful? Give feedback.
-
|
As far as I have understood Kube-VIP wants to solve HA, which MetalLB can't (it needs an external BGP service). Kube-VIP apparently can work with a virtual IP, which many cloud providers offer (I believe only Vultr offers BGP and you still have to install your own BGP daemon). |
Beta Was this translation helpful? Give feedback.
-
|
Basically, Kube-VIP uses ARP to elect a leader node and assign it a virtual IP and all other nodes then use BGP to advertise the virtual IP. This should take care of HA. As far as load balancing is concerned, there still needs to be a router that does it based on the cost-equivalent routes published by all the nodes. |
Beta Was this translation helpful? Give feedback.
-
|
This is how i solved the Client IP issue with my single-node k3s cluster(v1.21.5+k3s2), i think it's almost out-of-box since it need only one line patched, woks with the built-in ServiceLB/Traefik2.
curl -sfL https://get.k3s.io | sh -
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable servicelb" sh -
kubectl patch svc traefik --patch '{"spec":{"externalIPs":["<local ip>"]}}' -nkube-system
Test if X-Real-IP works using whoami kubectl create ns dev
kubectl apply -f https://gist.github.com/lazywalker/4d48eaad0ef111f5e370c9c25570cd39
# on other machine
curl -H Host:whoami.k3s.local <k3s master ip>
Hostname: web-8557b59f65-mmmxx
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.9
IP: fe80::74cb:efff:fe15:75ad
RemoteAddr: 10.42.0.1:40918
GET / HTTP/1.1
Host: whoami.k3s.local
User-Agent: curl/7.79.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 172.16.36.142
X-Forwarded-Host: whoami.k3s.local
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: k3s
X-Real-Ip: 172.16.36.142Clean up kubectl delete ns devThere you go! if you add agent nodes you may still need to set |
Beta Was this translation helpful? Give feedback.
-
|
the |
Beta Was this translation helpful? Give feedback.
-
|
This is not working for me. I do have I no configuration I was able to get the remote host IP on a plain, vanilla k3s installation with default settings. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, the 2 commands |
Beta Was this translation helpful? Give feedback.
-
|
For what its worth I have been trying this with a dual stack setup and with Cluster was installed with: Traefik config override: apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
service:
spec:
externalTrafficPolicy: Local
ipFamilyPolicy: RequireDualStack |
Beta Was this translation helpful? Give feedback.
-
|
Update: same setup seems to be working on v1.23.2+k3s1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks. Applying the traefik config override fixed the problem for me. |
Beta Was this translation helpful? Give feedback.
-
|
I‘d call it “works as designed“ as this is documented, but know you locked yourself down to a single-node “cluster“ as traffic can now potentially be spread imbalanced. |
Beta Was this translation helpful? Give feedback.
-
|
Can any of this apply to an UDP service ? I'm running pihole in k3s and only see the internal IP, not the clients. |
Beta Was this translation helpful? Give feedback.
-
|
so, without a load balancer, what are our options to use traefik and still have the hability to get real client ip? apparently if i set it up as a nodeport and externalTrafficPolicy:local i can get the real ip as "X-Real-Ip: 192.168.x.x" and "X-Forwarded-For: 192.168.x.x" headers... |
Beta Was this translation helpful? Give feedback.
-
|
hi all , deploying a mailserver on a deployment on a single node I tried with loadbalancer /nodeport service mode (tried also calico and port forwarding), but unable to have real client ip for SPF mail validation.Which is fastest way to have this work in single node k3s?thanks |
Beta Was this translation helpful? Give feedback.
-
|
I am all kinds of stuck on this. I feel that I tried just about everything - except that I am sticking with a rootless set-up. |
Beta Was this translation helpful? Give feedback.
-
|
H! What I need to do now to achieve this in multi-node cluster? |
Beta Was this translation helpful? Give feedback.
-
|
I am running a multi (3) node cluster as well. Using If however, you directly hit the node that the service is sitting on, it wil show you the real IP. I really hope there is a way to get this working, I would really like to be able to see useful access logs inside Traefik. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I found a simple solution for this problem. Basically the mode by you contact your service change from: to: than you don't contact the cni anymore. once you contact one url that is an ingress in k3s you enter in the node, than in the load balancer, than in traefik and finally on your service. I only have listened allthing writed here and I have setup traefik with hostNetwork: true and the service with externalTrafficPolicy: Local Thank you for this tread and I hope I will be helpfull. my os: virt kernel and hardening. my k3s: my traefik: klipper version: v0.4.4 what I have done for testing: create a namespace called apache-n deploy a base http container for testing: deploy a service: create an ingress: when contacting the example.example.com I have a response from the container: as you can see X-Forwarded-For: is right and X-Real-Ip: also. |
Beta Was this translation helpful? Give feedback.
-
|
thanks for your post. I was looking for a way to confirm what headers are being set by the proxy and your deployment does this. |
Beta Was this translation helpful? Give feedback.
-
|
I have same issue, Info"controller": {
"kind": "DaemonSet",
"allowSnippetAnnotations": True,
"service": {
"externalTrafficPolicy": "Local",
},
"config": {
"enable-real-ip": True,
"use-forwarded-headers": True,
"compute-full-forwarded-for": True,
"use-proxy-protocol": True,
"proxy-add-original-uri-header": True,
"forwarded-for-header": "proxy_protocol",
"real-ip-header": "proxy_protocol",
},
},InfoInfoInfoInfoI also tried default configuration that is |
Beta Was this translation helpful? Give feedback.
-
|
is there any standard setting that would allow you to send a real IP through a traefik proxy? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, use of See here for an example of hostNetwork #2997 (comment) And yes, hostNetwork means you lose IP isolation, so you need to assign unique ports from what I remember. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, (Note the different than other user reports specification of after applying and trying with |
Beta Was this translation helpful? Give feedback.
-
|
Preserving client source IPs in general is not a problem, but making it work with a scaled ingress controllers or cluster with more than one node is hard. Setting |
Beta Was this translation helpful? Give feedback.
-
|
The problem here was getting the client IP in the context of k3s, your other words are coming directly from the page you're linking and are well known and proven by others that have written about using metallb in front of k3s |
Beta Was this translation helpful? Give feedback.
and others
-
Is your feature request related to a problem? Please describe.
I am unable to obtain Real Client IP when using k3s and Traefik v2.2. I always get the cluster IP.
Describe the solution you'd like
I would like to obtain the client IP.
Describe alternatives you've considered
I already set
externalTrafficPolicy: Localin Traefik's Service.
Additional context
Issue can be reproduced by deploying containous/whoami image in cluster
Expected Response
Current Response
Service LoadBalancer Logs
Related
#955
Related Discussion
#679 (comment)
@erikwilson @btashton
Beta Was this translation helpful? Give feedback.
All reactions