native openshift: support for nested nulecule applications

[dustymabe]

Currently nested nulecule applications won't work when running oc new-app because the pod that runs the atomicapp code doesn't have access to the docker daemon so it can't pull down other containers. I'm not sure of the best way to solve this right now but we need to support running nested nulecules via oc new-app.

[rtnpro]

I will take a look into it.

[dustymabe]

@rtnpro make sure that, whatever you do, you don't try to interface with the docker registry directly via REST API calls. I already have a POC implementation of that but have been told it's not going to be a solution we are happy with.

[dustymabe]

@rtnpro openshift has a ./oc rsync command. Try it out with --strategy=tar and see if that is something we can use.

Basically we would need to start the container (in a pod) and then make it sleep or something. Then we would run the equivalent of the oc rsync against the sleeping pod and have it copy the files to the local install pod (the one started with oc new-app).

[rtnpro]

@dustymabe

Correct me if I am wrong?

So, if the atomicapp code doesn't have access to docker daemon, then no docker commands will work, e.g., docker cp, etc. Or, is it just prohibited to access Docker registries?

If only docker pull is forbidden, then the solution is pretty simple:

def unpack(...):
    if openshift_env:
         call oc run image with dummy entrypoint, and this will pull the image
    # Then others step can follow as usual:
    Run docker container
    Extract contents for docker container... 

[dustymabe]

we don't have access to the docker daemon at all. no docker commands will work.

Here is an example using just oc commands to grabbing a file from an image:

[root@f23 dir]# cat /tmp/httpdsleep.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: sleephttpd
spec:
  containers:
  - image: projectatomic/helloapache
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: sleephttpd
  restartPolicy: Always
[root@f23 dir]# oc create -f /tmp/httpdsleep.yaml                                                                                                                                                    
pod "sleephttpd" created
[root@f23 dir]# 
[root@f23 dir]# oc get pods sleephttpd
NAME         READY     STATUS    RESTARTS   AGE
sleephttpd   1/1       Running   0          1m
[root@f23 dir]# 
[root@f23 dir]# oc exec sleephttpd -- bash -c "tar -cz /application-entity/ 2>/dev/null" | tar -xzv
application-entity/
application-entity/Dockerfile
application-entity/Nulecule
application-entity/README.md
application-entity/artifacts/
application-entity/artifacts/docker/
application-entity/artifacts/docker/hello-apache-pod_run
application-entity/artifacts/kubernetes/
application-entity/artifacts/kubernetes/hello-apache-pod.json
[root@f23 dir]# 
[root@f23 dir]# find ./application-entity/
./application-entity/
./application-entity/Nulecule
./application-entity/Dockerfile
./application-entity/artifacts
./application-entity/artifacts/docker
./application-entity/artifacts/docker/hello-apache-pod_run
./application-entity/artifacts/kubernetes
./application-entity/artifacts/kubernetes/hello-apache-pod.json
./application-entity/README.md

now we just need to figure out the rest requests to make to get this to happen without the oc commmand.

[rtnpro]

@dustymabe this was the worst thing I was fearing to do :(

[rtnpro]

@dustymabe I am working on writing some code to achieve this using openshift API. I will put up a PR as soon as I have a working version.

[dustymabe]

@rtnpro there is this for exec a command in a pod:
https://docs.openshift.com/enterprise/3.0/rest_api/kubernetes_v1.html#connect-get-requests-to-exec-of-pod

[dustymabe]

@rtnpro so it looks like we have hit a roadblock..

The exec api calls require us to upgrade our protocol to SPDY vs HTTPS. python-requests does not support this so it seems like this is not a valid option for us for now. We are back to square 1.

[kadel]

@dustymabe Today I have been playing with k8s API /exec and there is also another option. You can use websockets instead of HTTP/2 (SPDY).

I found this because calling /exec using websockets is what OpenShift web console uses for executing commands on Terminal tab on Pod page.

To be honest I don't know much about websockets.
But I have been able to create something that is working: https://gist.github.com/kadel/6322bf8a1ba8e7f3bb73

[dustymabe]

@kadel - great work

I was able to take your POC and modify it to actually do the tar stuff we want:

# https://pypi.python.org/pypi/websocket-client/
import websocket
import ssl
import sys

def on_message(ws, message):
    # Not sure why but needed to trim first byte 
    sys.stdout.write(message[1:])

#cmd = sys.argv[1:]
cmd = 'tar -cz /application-entity'.split()

args = {"token": "foobar",
        "namespace": "proj1",
        "pod": "sleephttpd",
        "container": "sleephttpd",
        "command": ''.join(['command={}&'.format(item) for item in cmd])}

url = "wss://api.engint.openshift.com:443/api/v1/namespaces/{namespace}/pods/{pod}/exec?access_token={token}&container={container}&{command}stdout=1&stderr=0&stdin=0&tty=0".format(**args)
ws = websocket.WebSocketApp(url, on_message=on_message)

# sslopt disables ssl/tls verification
ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})

One example of running it:

[root@f23 dir]# python exec.py | tar -tz
application-entity/
application-entity/Dockerfile
application-entity/Nulecule
application-entity/README.md
application-entity/artifacts/
application-entity/artifacts/docker/
application-entity/artifacts/docker/hello-apache-pod_run
application-entity/artifacts/kubernetes/
application-entity/artifacts/kubernetes/hello-apache-pod.json

[dustymabe]

@kadel @rtnpro - can you work together to make this into something we can use?