Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for interactive `deis run bash` session #117

Closed
gabrtv opened this issue Aug 20, 2013 · 25 comments

Comments

Projects
None yet
@gabrtv
Copy link
Member

commented Aug 20, 2013

While #70 provided us with the ability to run one-off admin commands, we really need an interactive shell into the ephemeral docker containers.

This is quite tricky, as it requires a full duplex channel (HTTP Chunked Encoding? WebSockets?) as well as a select loop on the API server to handle routing stdin, stdout and stderr.

https://github.com/paramiko/paramiko/blob/master/demos/interactive.py#L39 provides a good example of a terminal-enabled select loop.

@tombh

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2014

Reminded of this from #465. Just to add another example alongside the paramiko one; I've implemented socket communication from the CLI client in Danabox using Python's Twisted framework. I called the socket server 'rendevous' after Heroku's implementation, though it's nowhere near as functional. All it does at the moment is stream the output from a server-side deploy command and receive heartbeats from active docker containers.

The main thing I'd like to improve is authentication. Ideally I think you'd want to reuse the cookie from the HTTP session, I haven't looked into that, so I'm not even sure if it's possible.

@mboersma

This comment has been minimized.

Copy link
Member

commented Jan 16, 2014

Cool! We had done websockets / socket.io with Twisted in a previous product at OpDemand, and I've done plenty of Twisted in the past, but I think we want to avoid that this time around because the code is more difficult to read (and write), and a priority for Deis is keeping things approachable for developers. Although your socket server here is nice and clean...hmm.

There's a websockets package using django, gevent, and redis, all of which are already in our toolbox. (Redis will be very soon, see the containerize branch.) I'm going to give that a go today and see how it works, but I'm open to other approaches. I want the implementation to work for deis run bash as well as deis logs -f.

@tombh

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2014

Yeah Twisted code is very hard to read :/ If you can come up with a simpler way I'll definitely use it Danabox as well :)

@bacongobbler

This comment has been minimized.

Copy link
Member

commented Feb 4, 2014

Why not leverage docker-py for the CLI? It already supports attaching to containers and log streaming, so it seems like the problem has already been solved for us. The only issue I see is that we have to figure out how an API call to the controller would let us grab this information to run and attach to the container.

@gabrtv gabrtv removed this from the 0.5.0 milestone Feb 10, 2014

@mboersma mboersma self-assigned this Feb 17, 2014

@mboersma mboersma removed their assignment Apr 9, 2014

@bacongobbler bacongobbler added this to the 0.9.0 milestone Apr 24, 2014

@bacongobbler bacongobbler self-assigned this Apr 24, 2014

@gabrtv gabrtv modified the milestones: 0.10.0, 0.9.0 May 19, 2014

@andyshinn

This comment has been minimized.

Copy link
Contributor

commented May 29, 2014

I wonder if something like https://github.com/progrium/termshare would work for this? You can run server in a container and proxy it through the router. Then you could run this interactively on a instance, get the URL ID, and either present the user with a URL for the browser or wrap up the termshare client somehow.

@bacongobbler bacongobbler modified the milestones: 0.11.0, 0.10.0 Jun 6, 2014

@bacongobbler bacongobbler modified the milestones: 1.1, 0.11.0 Jul 14, 2014

@gabrtv gabrtv added this to the 0.12.0 milestone Jul 30, 2014

@bacongobbler

This comment has been minimized.

Copy link
Member

commented Aug 25, 2014

Update on this bug: I'm currently researching different ways on how we can tackle this. The problem lies with the fact that Django was never initially designed for long-running requests such as websockets. However, there are Django plugins such as https://github.com/stephenmcd/django-socketio which help solve this problem. Need to do a bit more deep-sea diving as I haven't delved much into the area of websockets before :)

Also to note that there's a very informative article on using Django with socketio and gevent. I'll experiment with this as well.

@bacongobbler bacongobbler modified the milestones: 0.12.0, 0.13.0 Sep 11, 2014

@bacongobbler bacongobbler removed this from the 0.13.0 milestone Sep 22, 2014

@bacongobbler bacongobbler removed their assignment Sep 22, 2014

@gabrtv gabrtv removed the production label Oct 10, 2014

@evadne

This comment has been minimized.

Copy link

commented Dec 7, 2014

Is it possible to just use SSH?

  1. Query the latest version of the application thru etcdctl / Deis Controller
  2. Query location of the private registry
  3. Assemble image name
  4. $ fleetctl ssh machine_id docker run -ti $IMAGE_NAME /bin/bash -c "cd /app && $COMMAND"
@bacongobbler

This comment has been minimized.

Copy link
Member

commented Dec 7, 2014

@evadne it's possible but we wouldn't want users to be able to access the cluster through fleet. If it's in a public PaaS setting (such as someone OEM'ing Deis or something), it's a security concern to give users host-level access.

@fbjork

This comment has been minimized.

Copy link

commented Oct 21, 2015

Any update on this? This is as pretty big limitation of Deis compared to Heroku.

@gabrtv

This comment has been minimized.

Copy link
Member Author

commented Oct 21, 2015

@fbjork I would have liked to ship this earlier, but as we've discussed in the open planning meetings, this feature is on hold pending Deis v2. Reason being the plumbing we need for this already exists in kubectl exec, versus having to write and maintain a bi-directional channel from scratch -- only to throw it away for v2.

We expect to have an RC for Deis v2 available by the end of the year.

@shidenkai0

This comment has been minimized.

Copy link

commented Nov 17, 2015

Hi everyone,
I wrote a python script that opens a rails console directly from your deis app folder, using TLS to authenticate to the docker remote API on your cluster nodes, to use it, you have to enable Docker remote API with TLS, see: https://coreos.com/os/docs/latest/customizing-docker.html
I personally put it in my PATH directory (/usr/local/bin).
You may find it in my repo: https://github.com/hajimenomomo/deis_rails_console.git
Note that you may use the framework of this script to execute any kind of interactive command in a deis cluster.
I hope this helps.

@bacongobbler bacongobbler added the v2 label Nov 18, 2015

helgi pushed a commit to helgi/deis that referenced this issue Dec 29, 2015

Merge pull request deis#117 from mboersma/fix-exception-handlers
fix(api): delete new_release only if it exists
@rstacruz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

The workaround is to SSH into one of your CoreOS nodes:

# try the one in DEISCTL_TUNNEL
ssh core@1.2.3.4

then check out the docker containers running:

docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                                                                                      NAMES
bc11d8da5e52        1.2.3.4:5000/staging:v10              "/runner/init start w"   2 minutes ago       Up 2 minutes        0.0.0.0:32869->5000/tcp                                                                    staging_v10.web.1

Then use that container ID for docker exec:

docker exec -i -t bc11d8da5e52 bash
@nathansamson

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

If you have a more recent deis (v1.11+ for sure) version you can just do

deisctl dock your_app /runner/init yourcmd"

you can even pick a specific container with
deisctl dock your_app_v10.web.1 /runner/init yourcmd"

I am not sure if this is intended to work (it originally was supposed to be for deis components itself) but I find it very useful for application containers as well

@rstacruz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

Can confirm... that works brilliantly. Be sure to have your deis key in ssh-add and be sure you have DEISCTL_TUNNEL set.

@krancour

This comment has been minimized.

Copy link
Member

commented Jan 7, 2016

The one potential problem with that workaround, however, is that deisctl is an admin tool whilst deis is in many cases a developer tool. In an org with many developers and good separation of roles, the sysadmins / devops guys or gals likely would not have widely distributed the key that would allow a developer to use deisctl.

@rstacruz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

however, is that deisctl is an admin tool whilst deis is in many cases a developer tool

sadly so! hope a proper implementation of interactive deis run would land soon exactly because of this.

@nathansamson

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

@krancour While thats certainly true, ssh-ing into the server has the exact same problem (only the admins should be able to do so).

So while we all are eagerly waiting for deis run to work interactively natively, deisctl is a very handy trick to work with...

@bacongobbler

This comment has been minimized.

Copy link
Member

commented Feb 24, 2016

FYI this won't be immmediately implemented in Deis v2 as we are targeting for v1 compatibility while rebasing on top of kubernetes, so the workaround in v2.0.0 will be to run kubectl exec --namespace=myapp -it <pod name> bash. This will achieve the same behavior as running docker exec -it <my container> bash in v1.

@sheerun

This comment has been minimized.

Copy link

commented Jan 18, 2017

My ugly workaround:

export NAMESPACE=xxx PROCESS=web && kubectl exec -it --namespace $NAMESPACE "$(kubectl get pods --namespace $NAMESPACE -o name | grep $NAMESPACE-$PROCESS | cut -d '/' -f 2)" -- bash -c '. /app/.profile.d/ruby.sh && PATH=/app/.heroku/node/bin:$PATH /app/bin/rails c'

It's a shame deis doesn't support interactive output..

@bacongobbler

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@sheerun you can simplify that bash incantation with -- /runner/init rails c if you're using Heroku buildpacks.

@bacongobbler

This comment has been minimized.

Copy link
Member

commented Jan 18, 2017

@sheerun

This comment has been minimized.

Copy link

commented Jan 18, 2017

@bacongobbler thanks! here's my new deis-run command:

deis-run() { export NAMESPACE="$1" && PROCESS="$2" && shift && shift && kubectl exec -it --namespace $NAMESPACE "$(kubectl get pods --namespace $NAMESPACE -o name | grep $NAMESPACE-$PROCESS | cut -d '/' -f 2)" -- /runner/init "$@"; }

Usage:

# deis-run <application> <process> <command>
deis-run my-app web rails c
@deis-admin

This comment has been minimized.

Copy link

commented Jan 19, 2017

This issue was moved to deis/controller#1215

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.