New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supporting arbitrary web applications without Jupyter server present #258

Open
yuvipanda opened this Issue Nov 10, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@yuvipanda
Member

yuvipanda commented Nov 10, 2017

One of our goals is to have first class support for non-jupyter applications on Binder (such as RStudio, arbitrary command execution ala ReproServer, etc).

Currently, the only things that binderhub expects the process started inside to do are:

  1. Accept a token and enforce using that (via a cookie, header or query param) for authentication

From prior discussion, the simplest way to allow arbitrary execution seems to be to build a standalone no-deps simple application that does the following:

  1. Authentication with token + proxying (including websockets) to any other process
  2. Process supervision of the backend process, so it gets restarted as appropriate

This program should be completely statically linked, and then dropped into the image as a final step. This unfortunately probably means we can't use Python (unless cx_freeze exceeds all my expectations), and need to write this in Go or Rust.

Once this program exists, we can use it for supporting arbitrary web backends without having to impose too many requirements on the environment that is present inside the container image.

@yuvipanda

This comment has been minimized.

Member

yuvipanda commented Nov 10, 2017

We sortof already do this with a combination of the nbserverproxy + nbrsessionproxy for rstudio, and that mostly works when you have Jupyter already installed. However, I think to really be inclusive of non-Jupyter & non-Python users, we need to not have Jupyter present inside as a requirement.

@yuvipanda

This comment has been minimized.

Member

yuvipanda commented Nov 10, 2017

From the binderhub side, this would mean that we support the following:

  1. Customizing what 'cmd' is run inside the container by the API
  2. Formalizing how we pass in the token to the container as a stable API
  3. Formalizing how we expect the container process to use the token as a stable API

Currently all these three are jupyter specific.

@betatim

This comment has been minimized.

Member

betatim commented Nov 10, 2017

Supporting arbitrary web applications

I picked that instead of your less broad first line of the first comment because it is more radical ;) My question is: is this really our goal? For me this sounds like heroku and friends, which already exists. My first reaction is that binderhub is about "telling stories with data", "data narratives", and "encouraging reuse and exploration". For me this implies a notebook like interface. Something that encourages me to poke around inside and encourages the creator to not add so many layers that it is a nice and polished product which makes it harder to explore around and reuse it in a different project.

From a technical point of view a solid and robust way to launch things like RStudio and co that doesn't rely on too many "hacks" because we started with jupyter notebooks is something I welcome.

Maybe this is just a question of wording but I felt the urge to tell you :)

@minrk

This comment has been minimized.

Member

minrk commented Nov 10, 2017

It also doesn't have to be in go/rust because we could use a multi-container pod where the auth proxy is in a different container, right? Is there a reason the proxy must be in the container with the target application?

I'm all for using go/rust, as long as there is already a proxy that properly supports http + websockets without being told where those will be. I don't think developing and supporting our own proxy in go/rust ought to be within scope for this project, but adding simple auth to an existing one should be quite doable. So the first step, for me, is identifying the landscape of extensible proxy libraries in go and/or rust.

@yuvipanda

This comment has been minimized.

Member

yuvipanda commented Nov 10, 2017

@betatim perhaps 'arbitrary interactive web applications that are traditionally single-user'? I don't think we'd be heroku, since our model is:

  1. Autobuild some container image I
  2. Launch image I in pod P
  3. Allow authenticated access for the individual user who initiated process to pod P

While heroku's model is more like:

  1. Autobuild some container image I
  2. Launch image I in one or many pods P
  3. Allow arbitrary access to everyone for the things in pods P at a stable URL

IMO we should make sure that 'our model' does not require Jupyter to exist in the image I, and that's in scope if we want to consider RStudio and future similar projects (Eclipse Che maybe?! plain Terminado?) as first class citizens.

@yuvipanda

This comment has been minimized.

Member

yuvipanda commented Nov 10, 2017

@minrk after listening to you and @remram44 talk about this I'm now convinced that having a sidecar container with a proxy is the right thing to do.

i definitely do not want us to write any new code if possible, and splitting the proxy and process supervision parts lets us do that. It also means we are not actually restricted to single binary proxies, which brings nginx back into play...

@minrk

This comment has been minimized.

Member

minrk commented Nov 10, 2017

Given that we're talking about simple token auth, an nginx proxy seems like it would be a pretty good fit as a side-car, and not too complicated since it's not conferring with an external service.

We may want to consider whether activity-tracking should be part of this API, as well (either in this proxy pod, or in the user application), as we are moving away from tracking at the CHP level.

Step one is multi-container pod spec support in KubeSpawner, I think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment