Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes container #12

Open
bhack opened this issue May 2, 2019 · 51 comments

Comments

@bhack
Copy link

@bhack bhack commented May 2, 2019

Support for containers running in kubernetes (pod, deployment etc..)

@Chuxel

This comment has been minimized.

Copy link
Member

@Chuxel Chuxel commented May 2, 2019

@bhack Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 2, 2019

Often it is remote specially when I need special (hardware/data) resources.
I need always to use kubectl cp when I edit a file in vscode.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 2, 2019

Have you collaboration with https://github.com/Azure/vscode-kubernetes-tools Azure team?

@Chuxel

This comment has been minimized.

Copy link
Member

@Chuxel Chuxel commented May 2, 2019

@bhack Definatley something we've been thinking about which is partly why I asked about the scenario. We actually have a dev container defintion that allows you to use a Docker sandbox container with all the needed tools (kubctl, helm docker, etc) so you can containerize your build toolchain and still use Kubernetes extension and deploy to a Kubernetes cluster from there: https://github.com/Microsoft/vscode-dev-containers/tree/master/containers/kubernetes-helm

F1 > Create Container Configuration File... > Kubernetes & Helm

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 2, 2019

F1 > Remote-Containers: Attach to Running Container... for remote K8s could be a good initial feature here.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 2, 2019

Just to clarify I meant, as first useful feature, the same use case at (but for K8s):
https://code.visualstudio.com/docs/remote/containers#_attaching-to-running-containers

While using VS Code to spin up a new container can be useful in many situations, it may not match your workflow and you may prefer to "attach" VS Code to an already running container.

Cause my K8S remote workflow "may not match" VS Code spin up.

@Chuxel

This comment has been minimized.

Copy link
Member

@Chuxel Chuxel commented May 3, 2019

@bhack Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI. K8s configs are more complicated to manage in general so starting with an attach flow makes sense.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 3, 2019

I've a question about the server side/extensions. Cause the container/pod nature it is ephemeral how do you handle server side extensions over containers? Do you need to re-configure the remote container extensions every time?

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 3, 2019

Also I don't understand what could be the case between remote vs container for remote-container K8s case:
image
image
We need to suppose that the running code is in the container (i.e. from init container, persitent volume, remote mount, etc) but remote/local edits need to be in sync with the local/remote files.
So there in not a volume mount of a local path like in your current "local" container case.
We need to have an auto-sync over the "source code block" in the diagram.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 5, 2019

About source code block sync check ernoaapa/kubectl-warp#10.
As mentioned there I don't know if a sidecar injector could be a good strategy.

@chrmarti

This comment has been minimized.

Copy link
Contributor

@chrmarti chrmarti commented May 6, 2019

Extensions running in the container need to be installed in each container. That happens automatically for the extensions listed in the devcontainer.json.

The files don't have to be local for VS Code to work on them. You could just check out the files in the container.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 6, 2019

No the problem is that you work on remote files but generally you git commit changes locally. So we need a sync.

@chrmarti

This comment has been minimized.

Copy link
Contributor

@chrmarti chrmarti commented May 6, 2019

If you commit / push through the Git viewlet in VS Code, you do it on the remote side currently.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 6, 2019

Generally you don't have git push credentials on remote containers.

@chrmarti

This comment has been minimized.

Copy link
Contributor

@chrmarti chrmarti commented May 6, 2019

We currently register a credentials helper for Git in the container (if git is installed) that forwards requests to the local Git to lift that limitation.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented May 6, 2019

@chrmarti I still think that a sync is optionally required over the source code about volume block in the diagram.
A very frequent use case is that you test a change on a small scale in a local container (i.e. Docker container) and then you want to test on an active developing Pod (container in k8s).
Generally you don't commit until changes are tested locally and on the pod resources.
So if the change works locally then you want to sync "code block" on the remote existing developing pod/container. If all works also remotely probably you can commit also with your mentioned helper.
So a local to existing pod sync it is required IMHO.

@kdvolder

This comment has been minimized.

Copy link

@kdvolder kdvolder commented Jul 12, 2019

@Chuxel said:

Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

If it'd allow me to somehow connect to pods inside my local k8s cluster... yes it would (at least for my current local experimenations). But I don't think that this actually works right now (is it supposed to work already?).

When I use the "Remote-Containers: Attach to Running Container" it shows a list of things that seems to correspond to the stuff you get running docker ps command locally. But the pods running in my k8s cluster (i.e. the things that you get running kubdctl get pods) is not amongst the things it lets me choose from.

So what we need really is what you suggested here:

Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI.

That, by the way, would then also work with any k8s cluster be it local or remote.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 11, 2019

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

@mlgill

This comment has been minimized.

Copy link

@mlgill mlgill commented Sep 11, 2019

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 11, 2019

@mlgill Yes but It Is better than nothing cause there Is anything planned in the next iteraton #1390

@mlgill

This comment has been minimized.

Copy link

@mlgill mlgill commented Sep 11, 2019

I get that. Just want to be clear about the limitations.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 11, 2019

@mlgill Cause It Is closed source we cannot do anything more that upvote this. So, if you can, please try to help to collect upvotes here in your GitHub network of vscode users.

@kdvolder

This comment has been minimized.

Copy link

@kdvolder kdvolder commented Sep 11, 2019

I experimented a bit with using the ssh support to connect to a k8s container. Was able to get this working but the main drawback of the approach is that you have to explicitly install ssh daemon into your 'development contiainer'. It's also a fairly involved process to set it all up. Anyhow... just in case anyone is interested. I put this up here: https://github.com/kdvolder/vscode-remote-java-dev-env

@kdvolder

This comment has been minimized.

Copy link

@kdvolder kdvolder commented Sep 11, 2019

@bhack said

Cause It Is closed source

Is that true? If so, I am very surprised! I assumed otherwise given that we are here, filing issues and discussing on github (which is a platform for open source software development). But perhaps it is true after all since when I go and try to find the source code, indeed I can't seem to find any.

I've filed this ticket #1402 as I think the github readme and docs for vscode-remote are really not clear on that point.

@rberrelleza

This comment has been minimized.

Copy link

@rberrelleza rberrelleza commented Sep 12, 2019

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

@mlgill if you follow the instructions on the blog post that @bhack linked, you'll be able to use all the features of VS Code Remote-SSH, including debugging. You'll also have your development environment running in Kubernetes and with bidirectional file synchronization (this helps if you want to commit from your local machine, instead of forwarding your credentials).

I'm one of the maintainers of github.com/okteto/okteto. We're working on a smother integration with VS Code Remote, your feedback on the scenario and approach would be great.

@mlgill

This comment has been minimized.

Copy link

@mlgill mlgill commented Sep 12, 2019

@rberrelleza

This comment has been minimized.

Copy link

@rberrelleza rberrelleza commented Sep 12, 2019

Thanks for the reply. Using Oketo requires admin access to the Kubernetes cluster to install a package, correct? If so, this is not an option for me.

On Thu, Sep 12, 2019, at 01:56, Ramiro Berrelleza wrote: >> Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime: >> https://okteto.com/blog/vs-code-remote-development-in-kubernetes/ > To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct? > Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container. > I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH. @mlgill https://github.com/mlgill if you follow the instructions on the blog post https://okteto.com/blog/vs-code-remote-development-in-kubernetes/ that @bhack https://github.com/bhack linked, you'll be able to use all the features of VS Code Remote-SSH, including debugging. You'll also have your development environment running in Kubernetes and with bidirectional file synchronization (this helps if you want to commit from your local machine, instead of forwarding your credentials). I'm one of the maintainers of github.com/okteto/okteto. We're working on a smother integration with VS Code Remote, your feedback on the scenario and approach would be great. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=AANNAOK3CQVO5O7O24IR2ALQJHK2VA5CNFSM4HKE3FZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6QX4AY#issuecomment-530677251>, or mute the thread https://github.com/notifications/unsubscribe-auth/AANNAOMERQZ6R6XLTHUZDLDQJHK2VANCNFSM4HKE3FZA.

@mlgill You don't need admin permissions to the cluster, just deploy privileges on a namespace. If you can deploy an application to your cluster, you can use Okteto. We have an okteto channel in the Kubernetes slack, we can continue the conversation there 😄 .

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 12, 2019

@rberrelleza

The dev image includes an SSH server

It Is still quite unmpratical to hack every image but we haven't any other solution currently.

@kwlzn

This comment has been minimized.

Copy link

@kwlzn kwlzn commented Sep 27, 2019

is there some reason that the plugin couldn't use e.g. kubectl exec the same way e.g. Remote SSH uses ssh to install and start up the remote vscode process group?

@chrmarti

This comment has been minimized.

Copy link
Contributor

@chrmarti chrmarti commented Sep 27, 2019

There are a few other Docker-specific things we use. (E.g., an NPM package to inspect containers / images and to listen for events.)

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 27, 2019

There are many NPMs also for Kubernetes like https://www.npmjs.com/package/kubernetes-client

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 27, 2019

I think that also some code could be reused from the other Microsoft plugin https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Sep 30, 2019

This issue now is the 2nd most upvoted feature request in the container sub-component/label.
I hope it will start to join in the imminent October roadmap cycle ticket.

@vanbroup

This comment has been minimized.

Copy link

@vanbroup vanbroup commented Oct 9, 2019

We created a proof of concept ssh proxy for visual studio code that deploys a pod in your selected namespace with a custom docker image (make sure you have wget or curl in your image) on Kubernetes for the authenticated user.

https://github.com/digitorus/remote-vsc-k8s

Please check the repository of you want to give it a try or want to help with contributions. Feedback is welcome!

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Oct 15, 2019

October iteration plan was just opened:
#1661

I hope we will see something about this in the container section.

@kdvolder

This comment has been minimized.

Copy link

@kdvolder kdvolder commented Oct 15, 2019

I hope we will see something about this in the container section.

Agreed. I hope so too.

In the mean time it might be interesting for people who are looking to connect / run a development environment to something deployed in a k8s cluster to consider using Theia instead of vscode. Interesting fact, Theia is fully compatible with vscode extension api. You can drop a vscode .vsix file into Theia and it usually just works (tried it with several of our own vscode extensions with great success in the gitpod version of Theia). Theia also provides a very similar look and feel based on the same Monacco editor as vscode, and even down to the level of keyboard shortcuts it looks and feel like you are using vscode inside a web browser.

So if vscode remote support for k8s does not seem to be coming soon, it is reassuring to know that perhaps Theia might be an alternative way for us to provide a k8s based IDE experience. Our team will start exploring this a bit soon.

@davidwindell

This comment has been minimized.

Copy link

@davidwindell davidwindell commented Oct 16, 2019

I also strongly recommend looking at Eclipse Che which provides Their based K8's workspaces.

@pchico83

This comment has been minimized.

Copy link

@pchico83 pchico83 commented Oct 29, 2019

We have released a VS Code plugin for remote development in Kubernetes:
Release blog post: https://okteto.com/blog/remote-kubernetes-development
Marketplace: https://marketplace.visualstudio.com/items?itemName=okteto.remote-kubernetes
@bhack it does not require to modify your image to include an ssh server any more

@ggaaooppeenngg

This comment has been minimized.

Copy link

@ggaaooppeenngg ggaaooppeenngg commented Oct 30, 2019

Suppose I want to multiplex clients requests through a proxy, ssh can not help, I would like other protocol (Websocket HTTP) than SSH to remotely control my pod.

@pchico83

This comment has been minimized.

Copy link

@pchico83 pchico83 commented Oct 30, 2019

@ggaaooppeenngg what is the use case for having multi clients?

@ggaaooppeenngg

This comment has been minimized.

Copy link

@ggaaooppeenngg ggaaooppeenngg commented Oct 30, 2019

@pchico83
I am searching for a tool like vscode to provide coding environment in cloud to customers.
For ssh, I need to provide a direct external tcp service to user, in cloud environment it could be ip address:port or a loadbalancer which costs time to be established, while if through HTTP protocol I can dynamically config a route from proxy to to a pod like Jupyter Notebook.
BTW, we don't provide K8S access control to users.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Oct 30, 2019

Kubectl operatios like exec use API:
https://erkanerol.github.io/post/how-kubectl-exec-works/

I suppose we are talking about a solution without SSH for Kubernetes.

@pchico83

This comment has been minimized.

Copy link

@pchico83 pchico83 commented Oct 30, 2019

@ggaaooppeenngg We can support your use case, ping me at twitter (@pchico83) if you want to connect.
@bhack why do you like it better via kubectl exec? Our ssh connection is done via port-forward to the pod. The ssh server is injected in the pod itself. ssh is more standard and supports things like reverse proxying. We can adapt our solution to work via kubectl exec + kubectl port-forward but we would need to understand the benefits better because it requires much more work.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Oct 30, 2019

@pchico83 Cause I've seen that in the last two years this upstream proposal didn't got momentum:
kubernetes/kubernetes#42950

@bhack bhack closed this Oct 30, 2019
@bhack bhack reopened this Oct 30, 2019
@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Oct 30, 2019

If you follow that upstream ticket It Is not clear why they have not adopted ssh protocol or kubectl ssh subcommand. So I am not sure what are the upstream drawback.

@ggaaooppeenngg

This comment has been minimized.

Copy link

@ggaaooppeenngg ggaaooppeenngg commented Oct 30, 2019

@pchico83

I found http CONNECT method can tunnel ssh.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/elazarl/goproxy"
)

func httpsHandler(host string, ctx *goproxy.ProxyCtx) (*goproxy.ConnectAction, string) {
        // apply some rules to fake-host and proxy to real host
	return goproxy.OkConnect, "your-real-ssh-host"
}

func main() {
	proxy := goproxy.NewProxyHttpServer()
	proxy.Verbose = true
	proxy.OnRequest().HandleConnectFunc(httpsHandler)
	log.Fatal(http.ListenAndServe(":8892", proxy))
}

ssh user@fake-host -o "ProxyCommand=nc -X connect -x 127.0.0.1:8892 %h %p" could connect to the real-ssh-host

@pchico83

This comment has been minimized.

Copy link

@pchico83 pchico83 commented Oct 31, 2019

@ggaaooppeenngg thanks for the code snippet 😊
Yes, we could leverage the ssh connection through https very easily.

@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Nov 3, 2019

Seems that this Is not in the November Plan:

#1775

@bhack bhack referenced this issue Nov 4, 2019
5 of 17 tasks complete
@bhack

This comment has been minimized.

Copy link
Author

@bhack bhack commented Nov 7, 2019

November plan was finalized. No Kubernetes initial support this month.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.