New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hostnetwork support #561

Closed
hunter opened this Issue Apr 9, 2017 · 19 comments

Comments

Projects
7 participants
@hunter
Copy link
Contributor

hunter commented Apr 9, 2017

We run some of our Ceph clusters in K8s with hostNetworking enabled to allow easier access to Ceph from external services along with faster network performance. With Kube 1.6 the ClusterFirstWithHostNet has been added to make it slightly easier to handle DNS lookups for hostNetworking.

Is this something to be supported in Rook?

@kokhang

This comment has been minimized.

Copy link
Member

kokhang commented Apr 12, 2017

Hi @hunter, using ClusterFirstWithHostNet is something that i was going to experiment but instead im started working on a external Rook volume provisioner for Rook v0.4 (#538). With this provisioner, the monitor IPs would be discoverable by the operator and you would not need to rely on DNS lookups for the ceph mons.

@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Apr 13, 2017

The external provisioner will be a great addition :)

I do think a hostNetwork option will be a useful addition even with a new provisioner. Performance is pretty critical for running a stable production ceph cluster. I'd submit a patch but I'm not sure my Go knowledge is up to the task :)

@kokhang

This comment has been minimized.

Copy link
Member

kokhang commented Apr 13, 2017

@hunter Apart from accessing the Ceph mons from external services, what other benefits wouldhostNetwork provide? Wouldnt a service with type loadbalancer or NodePort grant ceph access to the external services? I just want to understand the use case better and how it helps with performance :)

@bzub

This comment has been minimized.

Copy link
Contributor

bzub commented Apr 13, 2017

Unfortunately Ceph clients don't like to talk to Mons through a proxy. If the destination IP doesn't match the Mon's real IP it will fail, at least when I tried using the kernel RBD client this way. So it seems the only options are hostNetwork, or Pod IPs that are routable from outside of the cluster.

Overlay network performance is something that interests me but I haven't looked into it. I'd love to see a comparison between Flannel, Calico, and no overlay.

@kokhang

This comment has been minimized.

Copy link
Member

kokhang commented Apr 13, 2017

@bzub I got it to work using a headless service. The only issue was that the DNS name was not resolvable from the host. This should change with ClusterFirstWithHostNet in 1.6 as @hunter mentioned but i havent tried it. I mentioned this back a while ago in #355 (comment).

@bzub

This comment has been minimized.

Copy link
Contributor

bzub commented Apr 13, 2017

Oh right I forgot about a headless service. The pod IPs would still need to be routable outside the cluster unless I'm forgetting something else.

@bzub

This comment has been minimized.

Copy link
Contributor

bzub commented Jun 11, 2017

On gitter @TimJones brought up this use-case for hostNetwork support:

I have a Ceph cluster on 3 bare-metal nodes, each with 2x1GbE and 2x10GbE. The 1GbE are for admin\service networks, and the 10GbE are for Ceph (public & cluster networks) and I would like Rook to be able to replicate the same using the dedicated links for cluster access and replication.

This could possibly be accomplished via Services that specify ExternalIPs, possibly as well.

@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Jul 14, 2017

Just looping back on this. We have a similar requirement to @TimJones.

Is it too hard/late to get something like this into 0.5?

@travisn

This comment has been minimized.

Copy link
Member

travisn commented Jul 14, 2017

@hunter I wonder if #806 will get us most of the way with this. With that change, there is now a stable ip address for monitors via a k8s service. The mon service looks like below. When ClusterFirstWithHostNet is enabled, would the ClusterIP on the service be the host ip? If that's the case, I think we're there.

$ kn get svc rook-ceph-mon0 -o yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: rook-ceph-mon
    mon: rook-ceph-mon0
    mon_cluster: rook
  name: rook-ceph-mon0
  namespace: rook
spec:
  clusterIP: 10.3.0.232
  ports:
  - name: rook-ceph-mon0
    port: 6790
    protocol: TCP
    targetPort: 6790
  selector:
    app: rook-ceph-mon
    mon: rook-ceph-mon0
    mon_cluster: rook
  sessionAffinity: None
  type: ClusterIP
@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Jul 15, 2017

It’s definitely a move in the right direction but for our current implementation (with ceph-docker) we need to be able to avoid the overhead of the Kube network and run on a separate bonded network

@travisn

This comment has been minimized.

Copy link
Member

travisn commented Jul 15, 2017

Got it, it's about the host network. Perhaps if we added a setting hostNetwork: true in the rook-cluster.yaml, then the operator could coordinate:

  • hostNetwork: true on all the pods
  • ClusterFirstWithHostNet for dns
  • The mons would use the host ip instead of a service ip
@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Jul 15, 2017

Great!! Something very similar is done with ceph-docker. We found a few headaches with hostnames not matching pod name and monmaps/services but I think #806 will make a big difference there

@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Aug 16, 2017

Is it possible to bump this into 0.6? We'll try to contribute if we can get some spare cycles. For now we can't do any production hardware testing until we can run on the hostNetwork.

@bassam

This comment has been minimized.

Copy link
Member

bassam commented Aug 18, 2017

this would be a good addition to 0.6. @hunter do you think you can help with the implementation?

@hunter

This comment has been minimized.

Copy link
Contributor

hunter commented Aug 18, 2017

Our team is pretty busy but I'll see what can be done to help!

@jbw976 jbw976 added this to the 0.6 milestone Aug 23, 2017

@jbw976 jbw976 added this to TODO in v0.6 Aug 23, 2017

@galexrt

This comment has been minimized.

Copy link
Member

galexrt commented Sep 9, 2017

Has anyone worked on adding hostNetwork feature? I have some time to try to add this.

@bassam

This comment has been minimized.

Copy link
Member

bassam commented Sep 9, 2017

@galexrt that would be awesome. its much needed!

@galexrt

This comment has been minimized.

Copy link
Member

galexrt commented Sep 9, 2017

#975
As written in the PR, I couldn't test it yet. I'll try to upload a version to my Docker namespace, so anyone can test it :)

@travisn travisn moved this from TODO to In Progress in v0.6 Sep 11, 2017

@DanKerns DanKerns moved this from In Progress to In Review in v0.6 Sep 13, 2017

@travisn travisn moved this from In Review to Done in v0.6 Sep 18, 2017

@galexrt

This comment has been minimized.

Copy link
Member

galexrt commented Sep 18, 2017

The hostNetwork feature has now been implemented in #975.

If you have any questions about it, please let me know. :)


Closed by #975.

@galexrt galexrt closed this Sep 18, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment