New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

discussion / plan LXC integration #6862

Closed
mswart opened this Issue Apr 15, 2015 · 34 comments

Comments

Projects
None yet
@mswart

mswart commented Apr 15, 2015

As discussed in https://groups.google.com/forum/#!topic/google-containers/aWZ0U_SIA6A supporting plain LXC containers (or lmctfy / lxd) requires some or a lot of changes.

I am very interested in managing LXC containers with kubernetes and also implementing this. But I would like to get some hints what needs to be done/how much work this would be:

  • What changes are needed/what components need to be extended?
  • What architectural decision needs to be made before the implementation is reasonable?
  • What are the minimal changes?
  • Are there any changes that should/need to be done before a integration is practical?
@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Apr 15, 2015

Member

The ongoing work to support Rocket should be clearing a lot of the
underbrush for you. I think the biggest issues are

  1. What parts of the API need to be changed/extended/removed to capture an
    LXC runtime?
  2. What parts of Kubelet " " " " ...

Dawn is overseeing the Rocket work, so I'll let her be more detailed, but
I'd be happy to see LXC support.

On Wed, Apr 15, 2015 at 7:27 AM, Malte Swart notifications@github.com
wrote:

As discussed in
https://groups.google.com/forum/#!topic/google-containers/aWZ0U_SIA6A
supporting plain LXC containers (or lmctfy / lxd) requires some or a lot of
changes.

I am very interested in managing LXC containers with kubernetes and also
implementing this. But I would like to get some hints what needs to be
done/how much work this would be:

  • What changes are needed/what components need to be extended?
  • What architectural decision needs to be made before the
    implementation is reasonable?
  • What are the minimal changes?
  • Are there any changes that should/need to be done before a
    integration is practical?


Reply to this email directly or view it on GitHub
#6862.

Member

thockin commented Apr 15, 2015

The ongoing work to support Rocket should be clearing a lot of the
underbrush for you. I think the biggest issues are

  1. What parts of the API need to be changed/extended/removed to capture an
    LXC runtime?
  2. What parts of Kubelet " " " " ...

Dawn is overseeing the Rocket work, so I'll let her be more detailed, but
I'd be happy to see LXC support.

On Wed, Apr 15, 2015 at 7:27 AM, Malte Swart notifications@github.com
wrote:

As discussed in
https://groups.google.com/forum/#!topic/google-containers/aWZ0U_SIA6A
supporting plain LXC containers (or lmctfy / lxd) requires some or a lot of
changes.

I am very interested in managing LXC containers with kubernetes and also
implementing this. But I would like to get some hints what needs to be
done/how much work this would be:

  • What changes are needed/what components need to be extended?
  • What architectural decision needs to be made before the
    implementation is reasonable?
  • What are the minimal changes?
  • Are there any changes that should/need to be done before a
    integration is practical?


Reply to this email directly or view it on GitHub
#6862.

@erictune

This comment has been minimized.

Show comment
Hide comment
Member

erictune commented Apr 15, 2015

@monokal

This comment has been minimized.

Show comment
Hide comment
@monokal

monokal May 10, 2015

LXC/LXD support in Kubernetes would be awesome. Forming containers around a single process (hey Docker) is obviously the ideal in terms of a micro-service infrastructure, but a lot of us don't yet have this pleasure (e.g, those of us tasked with moving an organisation in to 'DevOps' but with legacy services to support in the meantime).

This is obviously not trivial to implement, but it shouldn't be the goliath task it was pre-LXD now that we have a means of nicely transferring and migrating LXC containers around hosts, given, we're early days. I've been looking at Mesos/Marathon, CoreOS, etc, etc, but nothing seems to solve the problem of actually orchestrating a cluster of non-Docker containers.

I believe the first task would be to implement the runtime abstraction, which is currently in progress with rkt in mind.

monokal commented May 10, 2015

LXC/LXD support in Kubernetes would be awesome. Forming containers around a single process (hey Docker) is obviously the ideal in terms of a micro-service infrastructure, but a lot of us don't yet have this pleasure (e.g, those of us tasked with moving an organisation in to 'DevOps' but with legacy services to support in the meantime).

This is obviously not trivial to implement, but it shouldn't be the goliath task it was pre-LXD now that we have a means of nicely transferring and migrating LXC containers around hosts, given, we're early days. I've been looking at Mesos/Marathon, CoreOS, etc, etc, but nothing seems to solve the problem of actually orchestrating a cluster of non-Docker containers.

I believe the first task would be to implement the runtime abstraction, which is currently in progress with rkt in mind.

@vmarmol

This comment has been minimized.

Show comment
Hide comment
@vmarmol

vmarmol May 11, 2015

Contributor

The runtime abstraction is complete and rkt is using it today. A similar thing could be done with LXC.

Contributor

vmarmol commented May 11, 2015

The runtime abstraction is complete and rkt is using it today. A similar thing could be done with LXC.

@monokal

This comment has been minimized.

Show comment
Hide comment
@monokal

monokal May 11, 2015

Brilliant, good work. For anyone following up on LXC/LXD support, please head over here: https://groups.google.com/forum/#!topic/google-containers/aWZ0U_SIA6A

monokal commented May 11, 2015

Brilliant, good work. For anyone following up on LXC/LXD support, please head over here: https://groups.google.com/forum/#!topic/google-containers/aWZ0U_SIA6A

@sl1pm4t

This comment has been minimized.

Show comment
Hide comment

sl1pm4t commented Nov 5, 2015

+1

@jgnagy

This comment has been minimized.

Show comment
Hide comment

jgnagy commented Apr 5, 2016

+1

@bj916b

This comment has been minimized.

Show comment
Hide comment
@bj916b

bj916b May 6, 2016

@thockin have there been any other discussions/work around this issue? i noticed it's a little old in kubernetes-land, but it would be sort of a nice feature to have for existing organizations who have LXC-based deployments. i'm only curious.

bj916b commented May 6, 2016

@thockin have there been any other discussions/work around this issue? i noticed it's a little old in kubernetes-land, but it would be sort of a nice feature to have for existing organizations who have LXC-based deployments. i'm only curious.

@tebanep

This comment has been minimized.

Show comment
Hide comment

tebanep commented May 10, 2016

+1

@bprashanth

This comment has been minimized.

Show comment
Hide comment
@bprashanth

bprashanth May 10, 2016

Member

@kubernetes/sig-node

Member

bprashanth commented May 10, 2016

@kubernetes/sig-node

@yujuhong

This comment has been minimized.

Show comment
Hide comment
@yujuhong

yujuhong May 10, 2016

Contributor

We are in the process (#22964) of redefining the container runtime interface, which serves as the main interface for runtime integration, to improve feature velocity and extensibility.
Now would be a great time to join the discussions If you have any input/suggestions/concerns.

Contributor

yujuhong commented May 10, 2016

We are in the process (#22964) of redefining the container runtime interface, which serves as the main interface for runtime integration, to improve feature velocity and extensibility.
Now would be a great time to join the discussions If you have any input/suggestions/concerns.

@malix0

This comment has been minimized.

Show comment
Hide comment
@malix0

malix0 May 13, 2016

+1 to have LXC/LXD support

malix0 commented May 13, 2016

+1 to have LXC/LXD support

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh May 13, 2016

Member

It would help if someone can enumerate the reasons for wanting lxc support...

Member

vishh commented May 13, 2016

It would help if someone can enumerate the reasons for wanting lxc support...

@v1k0d3n

This comment has been minimized.

Show comment
Hide comment
@v1k0d3n

v1k0d3n May 13, 2016

@vishh would this be similar to PR #13079?

application containers are not the same as machine containers from a security perspective, since there is no control over declaring the kernel security posture. i'm not claiming to know what it would take to make this happen in kubernetes-land, but I definitely see a lot of interesting brewing on our side for additional options.

v1k0d3n commented May 13, 2016

@vishh would this be similar to PR #13079?

application containers are not the same as machine containers from a security perspective, since there is no control over declaring the kernel security posture. i'm not claiming to know what it would take to make this happen in kubernetes-land, but I definitely see a lot of interesting brewing on our side for additional options.

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh May 13, 2016

Member

@v1k0d3n Nope. There will be a proposal out (soon) to define a more user-friendly interface against which lxd can be integrated pretty easily.
It would help if you can enumerate a few advanced/additional options that you need from Kubelet.

Member

vishh commented May 13, 2016

@v1k0d3n Nope. There will be a proposal out (soon) to define a more user-friendly interface against which lxd can be integrated pretty easily.
It would help if you can enumerate a few advanced/additional options that you need from Kubelet.

@bactisme

This comment has been minimized.

Show comment
Hide comment
@bactisme

bactisme May 14, 2016

@vishh I am not saying that docker's one-process-per-container is broken ... but some people do 😊

If your application talk to a log daemon or simple mail server or ntp, have cron tasks etc. Docker require you to spin as much container. Of course it is better for horizontal scalability and process management ... but it also bring a damageable complexity ...

Lxc can bring simpler deployment: careffully build your box, ship.

@vishh I am not saying that docker's one-process-per-container is broken ... but some people do 😊

If your application talk to a log daemon or simple mail server or ntp, have cron tasks etc. Docker require you to spin as much container. Of course it is better for horizontal scalability and process management ... but it also bring a damageable complexity ...

Lxc can bring simpler deployment: careffully build your box, ship.

@monokal

This comment has been minimized.

Show comment
Hide comment
@monokal

monokal May 15, 2016

I think the majority of people want LXD/LXC support in order to allow for
legacy services. Not everyone's fortune enough to have a completely
green-field environment, myself included.

Mesosphere's DC/OS is now open-source so that's an option, but Kubernetes
support would allow for a single unified platform.
On 14 May 2016 7:46 p.m., "Baptiste M." notifications@github.com wrote:

@vishh https://github.com/vishh I am not saying that docker's
one-process-per-container is broken ... but some people do 😊

If your application talk to a log daemon or simple mail server or ntp,
have cron tasks etc. Docker require you to spin as much container. Of
course it is better for horizontal scalability and process management ...
but it also bring a damageable complexity ...

Lxc can bring simpler deployment: careffully build your box, ship.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#6862 (comment)

monokal commented May 15, 2016

I think the majority of people want LXD/LXC support in order to allow for
legacy services. Not everyone's fortune enough to have a completely
green-field environment, myself included.

Mesosphere's DC/OS is now open-source so that's an option, but Kubernetes
support would allow for a single unified platform.
On 14 May 2016 7:46 p.m., "Baptiste M." notifications@github.com wrote:

@vishh https://github.com/vishh I am not saying that docker's
one-process-per-container is broken ... but some people do 😊

If your application talk to a log daemon or simple mail server or ntp,
have cron tasks etc. Docker require you to spin as much container. Of
course it is better for horizontal scalability and process management ...
but it also bring a damageable complexity ...

Lxc can bring simpler deployment: careffully build your box, ship.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#6862 (comment)

@vishh

This comment has been minimized.

Show comment
Hide comment
@vishh

vishh May 18, 2016

Member

I'm convinced. Once #22964 takes some shape (hopefully very soon), we can take in community contributions for lxc/lxd integration. As part of #22964, we are attempting to define a client-server model for integrating with alternate runtimes and it should be (hopefully) straightforward to integrate with lxc/lxd.

Member

vishh commented May 18, 2016

I'm convinced. Once #22964 takes some shape (hopefully very soon), we can take in community contributions for lxc/lxd integration. As part of #22964, we are attempting to define a client-server model for integrating with alternate runtimes and it should be (hopefully) straightforward to integrate with lxc/lxd.

@bj916b

This comment has been minimized.

Show comment
Hide comment
@bj916b

bj916b May 18, 2016

@vishh this is great news! very much looking forward to this. i'll start watching #22964 more closely.

bj916b commented May 18, 2016

@vishh this is great news! very much looking forward to this. i'll start watching #22964 more closely.

@dragon9783

This comment has been minimized.

Show comment
Hide comment

+1

@nick-walt

This comment has been minimized.

Show comment
Hide comment
@nick-walt

nick-walt Jul 11, 2016

LXD is the way forward in the full system-container use case and it definitely has a place there with Rkt and Docker. With LXD's natural API (ugh, might even be better than Docker's?) and its capability to also be used as an incredibly lightweight single app-container, as well as a full system-container, we can expect to see increased usage of this technology. So it would be awesome to have native support in Kuber.

LXD is the way forward in the full system-container use case and it definitely has a place there with Rkt and Docker. With LXD's natural API (ugh, might even be better than Docker's?) and its capability to also be used as an incredibly lightweight single app-container, as well as a full system-container, we can expect to see increased usage of this technology. So it would be awesome to have native support in Kuber.

@abcfy2

This comment has been minimized.

Show comment
Hide comment

abcfy2 commented Sep 16, 2016

+1

@sequoiar

This comment has been minimized.

Show comment
Hide comment

sequoiar commented Oct 9, 2016

+1

@dronnix

This comment has been minimized.

Show comment
Hide comment

dronnix commented Oct 19, 2016

+1

@pouledodue

This comment has been minimized.

Show comment
Hide comment

+111

@techtonik

This comment has been minimized.

Show comment
Hide comment
@techtonik

techtonik Oct 20, 2016

Is anyone hiring to work on that?

Is anyone hiring to work on that?

@SpiLLeR

This comment has been minimized.

Show comment
Hide comment

SpiLLeR commented Oct 22, 2016

+1

@thockin

This comment has been minimized.

Show comment
Hide comment
@thockin

thockin Oct 22, 2016

Member

EVERYONE PLEASE STOP +1ing this. GitHub has a sentiment button - look at the first message, click the thumbs-up. Every time you +1 it in a message, dozens or hundreds of people get email.

Like so many issues, this requires someone to act as the driver for it. We're doing everything we can to make it possible to integrate things like this, but we (Google team and the larger Kube-team) do not have the bandwidth to do every cool idea.

Member

thockin commented Oct 22, 2016

EVERYONE PLEASE STOP +1ing this. GitHub has a sentiment button - look at the first message, click the thumbs-up. Every time you +1 it in a message, dozens or hundreds of people get email.

Like so many issues, this requires someone to act as the driver for it. We're doing everything we can to make it possible to integrate things like this, but we (Google team and the larger Kube-team) do not have the bandwidth to do every cool idea.

@pouledodue

This comment has been minimized.

Show comment
Hide comment
@pouledodue

pouledodue Oct 22, 2016

@vishh for orchestration of stateful containers like database, replicas and their associated data

see: https://news.ycombinator.com/item?id=10609247

pouledodue commented Oct 22, 2016

@vishh for orchestration of stateful containers like database, replicas and their associated data

see: https://news.ycombinator.com/item?id=10609247

@malix0

This comment has been minimized.

Show comment
Hide comment
@malix0

malix0 Jan 25, 2017

Canonical release is own distribution of kubernetes
https://jujucharms.com/canonical-kubernetes/bundle/
but does not add LXC/LXD support

malix0 commented Jan 25, 2017

Canonical release is own distribution of kubernetes
https://jujucharms.com/canonical-kubernetes/bundle/
but does not add LXC/LXD support

@malix0

This comment has been minimized.

Show comment
Hide comment

malix0 commented Mar 2, 2017

The google group discussion has an updated URL https://groups.google.com/forum/m/#!topic/kubernetes-users/aWZ0U_SIA6A

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Dec 21, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Jan 20, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

@fejta-bot

This comment has been minimized.

Show comment
Hide comment
@fejta-bot

fejta-bot Feb 19, 2018

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment