Migrate Emulator Pilot to native K8s

[mpeuster]

This is an intermediate step before the pilot is deployed on K8s using 5GTANGO SP, we first want to have a version that is directly deployed on K8s without 5GTANGO.

TODOs

See corresponding wiki entry

• adopt VNFs
  • prepare Dockerfiles
  • fix crashing CC processor (see Fix crashing CC processor #89)
• prepare K8s deployment descriptors
  • Deployments for starting the containers
  • Deployment of each CDU in a separate pod
  • Services for accessing the containers (from outside, eg, for eae)
  • Ingress for simplified external access? Or simply setting a fixed nodePort? Optional for now.
• prepare service descriptors and figure out interconnection of CDUs
  • Test connection from MDC to CC processor via CC broker using mosquitto CLI
  • Integrate and tests all CNFs of NS1 and NS2 to work together
• move DT outside of NS2 and k8s and connect to MDC from the outside
• combine all CDUs of one CNF in one deployment and pod that's reachable through one k8s service
• add new CC CDUs: Prometheus DB and exporter

[stefanbschneider]

Started working on deploying NS1 and NS2 on k8s: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Deploying-the-service-on-Kubernetes

To ensure all VNF containers keep running in k8s, they need a different start.sh than the vim-emu VNFs (that doesn't run the commands in background). I adjusted all relevant VNFs to use start.vimemu.sh for the :vimemu container and start.sh for :k8s (similar to Dockerfile.vimemu and Dockerfile).

Once all VNFs run on k8s, the next step is to configure the networking and connection of the VNFs within k8s.

[stefanbschneider]

Currently, all pods are deployed fine at the beginning, but then keep crashing every now and then:

$ kubectl get pods
NAME                                 READY   STATUS             RESTARTS   AGE
sm-ns1-deployment-5564965dcc-n5d57   2/3     CrashLoopBackOff   13         76m
sm-ns2-deployment-67c8f6574f-9zb82   2/2     Running            0          76m

It seems like this is the CC processor: terminated with exit code 137

[stefanbschneider]

Connecting cc processor to cc broker within NS1 worked: Both are running inside the same pod and the processor can simply subscribe to the broker at localhost:1883 as before

Connecting the MDC in NS2 to the CC broker in NS1 doesn't work yet because they are running in different pods. Next time, I'll have to follow the debug guide here to figure out what the problem is.
My current guess is that we need to change the service to type: ClusterIP (simply remove type because it's the default), but let's see.

Testing the connection works best by logging into the CC broker container and to the other VNF (eg, MDC) and using the mosquitto_sub CLI call to test subscribing to the broker. If this works, then the connection seems to work. Next would be pushing something from MDC via broker to CC processor.

[stefanbschneider]

Update/Todo: Put all containers/CDUs in separate pods such that they could be scaled independently. Also for SP to allow easy mapping of descriptors to pods, using VDU = pod.

For example: CC could have 3x Broker CDU and 5x Processor CDU and 1x DB CDU

Apparently, this means, I have to split the Kubernetes descriptors into separate deployments and services for each CDU: https://stackoverflow.com/a/43220561/2745116

[stefanbschneider]

Connection between containers works now (with microk8s DNS enabled). MQTT works also manually. Next:

• Test and debug NS1: Connect CC processor and broker
• NS2: Connect MDC to CC broker

[stefanbschneider]

Next up: Move digital twin outside of NS2 and even outside of Kubernetes deployment. Start it outside as normal docker container trying to access the MDC running inside the k8s deployment. Similar to a real machine that would use NS2 from the outside.

Discuss with Manuel how to start in privileged mode such that the Samba file share works.

On k8s side, use NodePort service for MDC to expose its Samba host at a fixed port of the cluster, where it can be accessed from the outside. Pass the cluster address and the exposed node port as environmental variables to the DT docker.

[stefanbschneider]

Started wiki page on how to deploy the DT as Docker container outside of K8s (and later also on how to connect it): https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Connecting-the-digital-twin

Problem: The current NodePort service maps the MDC's ports to other external ports for outside access (30000+). Not sure if this easily works with the DT. Better to somehow expose the MDCs normal ports to the outside. Have to google again how that works.

Note: The Docker run command doesn't work on Windows for some reason. But works on Ubuntu. This was due Windows line endings in the start.sh. When changing them to Unix line endings (in Notepad++) it works.

[mpeuster]

CC CDU03 (mqttexporter) and CC CDU04 (prometheus DB) are ready to be picked up for the K8s deployment #92

Still, I face some issues with DT, MDC in the vim-emu case. Might also happen in the K8s case #106

[stefanbschneider]

I figured out how to assign and expose fixed ports using k8s' NodePort: See updated wiki page.

Unfortunately, it still doesn't seem to work. The MDC still says "Error no EM63 response file was found! Retry!".

Not sure if this is because the k8s service didn't work properly or due to some other problem. Also not sure how to debug.

Wait: I forgot to set the Samba host IP when starting the DT! By default this points to a fixed IP in the emulator. It should be the IP of the cluster node, which exposes the MDC's ports (kubectl cluster-info).

[stefanbschneider]

Updated todos above according to call and updated wiki page: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Integration-with-SP

[stefanbschneider]

Using a NodePort service and starting minikube with a custom port range works fine. But this doesn't seem to work for microk8s...

Workaround is to install a load balancer:
Microk8s-guide-V2.pdf