New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nextepc in a Docker Container #16

Open
amilenovic opened this Issue Feb 5, 2018 · 15 comments

Comments

Projects
None yet
4 participants
@amilenovic

amilenovic commented Feb 5, 2018

I really like NextEPC design, and tried to dockerize it. Besides freediameter docker instruction, I couldn't find a Docker file for NextEPC.

I tried to use Ubuntu base image and applied same installation instructions as you described, but got following:

invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up mongodb (1:2.6.10-0ubuntu1) ...
Setting up nextepc-core:amd64 (0.3.2~xenial) ...
Setting up nextepc-mme:amd64 (0.3.2~xenial) ...
Setting up nextepc-sgw:amd64 (0.3.2~xenial) ...
Setting up nextepc-pgw:amd64 (0.3.2~xenial) ...
open: No such file or directory
Failed to bring up pgwtun.
dpkg: error processing package nextepc-pgw:amd64 (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up nextepc-hss:amd64 (0.3.2~xenial) ...
Setting up nextepc-pcrf:amd64 (0.3.2~xenial) ...
dpkg: dependency problems prevent configuration of nextepc:amd64:
 nextepc:amd64 depends on nextepc-pgw (= 0.3.2~xenial); however:
  Package nextepc-pgw:amd64 is not configured yet.

dpkg: error processing package nextepc:amd64 (--configure):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for systemd (229-4ubuntu21) ...
Errors were encountered while processing:
 nextepc-pgw:amd64
 nextepc:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

Then, I tried to copy /install dir into image, after building nextepc directly, but I got error on symbolic links. I tried to re-link them, but didn't work either. My knowledge in shared libs is not great and probably that is one of reason.

Do you have any tips/pointers on how I can create a container? Am I missing something obvious?

Thanks

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 6, 2018

Thank you for liking NextEPC.

Note that freeDiameter docker instruction is depreciated. At first, we was trying to build a debian package for freeDiameter. Unfortunately, freeDiameter seems to be no longer maintaining. So, we put this library into our source tree.

BTW, the problem is related to TAP/TUN driver in docker container. You may need to --cap-add=NET_ADMIN. Additionally, ipv6 forward should be enabled. You need to turn on sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0

For your reference,

  • On host machine.
docker run -p 4000:3000 --hostname nextepc -ti --name nextepc --privileged --cap
-add=SYS_ADMIN -e "container=docker" -v /sys/fs/cgroup:/sys/fs/cgroup -v $PWD:/mnt ubuntu /sbin/init
  • In the container
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0
sudo apt-get install nextepc

I'm hoping that you can use nextepc in docker environment.
Let me know if you have any problem.

@amilenovic

This comment has been minimized.

amilenovic commented Feb 7, 2018

Thank you for the guide.

Yes, I am now able to install nextepc in a container and can resolve TAP/TUN with the host (docker run) as you pointed out.

However, there is a difficulty of preparing a docker image. Since apt-get install nextepc bundles together installation of binaries and startup services, I am not able to create a Dockerfile having RUN apt-get install nextepc as simple instruction built into image. I have to run that on container startup.

This is not a major issue, but wonder is it feasible to separate binary installation from startup services? With that way you could have non-debian based container images also supported?

Please let me know what you think.

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 7, 2018

You made a good point!

For creating Dockerfile, we seem to separate startup services from the installation of binaries. The followings are my proposal.

  • Both Install and Startup
sudo apt-get install nextepc
  • Only Install
sudo apt-get install nextepc-org

I'm not sure which name is good for us? How about nextepc-org. Is there any suggestion?

Anyway, I'll try to make new debian package. I'm hoping that v0.3.3 supports this stuff.

Thank you for your comment.

@amilenovic

This comment has been minimized.

amilenovic commented Feb 7, 2018

That is a very good proposal!

I don't think I can help you with naming of packages. There must be some good naming practice on that, but I am not aware of.

Regardless, I look forward for v0.3.3 to give a try.

Thank you

acetcom added a commit that referenced this issue Feb 10, 2018

Debian package update for docker issue (#16)
* Previously, pgwtun is setup with ifupdown package.
* Now, we'll use the systemd-networkd.
@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 11, 2018

Version 0.3.3 has been released, but this issue has not been resolved yet.

First, I wanted to solve this problem without creating a new debian package(previously called nextepc-org). So, I changed the method of creating TUN device with using systemd-networkd. Nevertheless, the following two lines are preventing to create docker image.

See debian/nextepc-pgw.postinst

systemctl restart systemd-udevd
systemctl restart systemd-networkd

The two things are executed at the PGW installation phase to bring up TUN(pgwtun) device. So, the startup failed to create docker image.

I'm hoping to solve this problem using systemd framework. Is it possible to move them to PGW's systemd service file. See nextepc-pgw.service

I need to study a little more to move those two lines.

I would try to solve the problem next version.
Let me know if you can find another solution.

Thank you for raising this very nice issue!

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 13, 2018

We might be solving the problem using deb-systemd-invoke.

In docker container, /usr/sbin/policy-rc.d can prevent the command systemctl restart .... So, I've changed systemctl restart to deb-systemd-invoke. And then, I can make the docker image.

We released v0.3.3 again. It's just package revision. All source code is same.

Let me describe simple guide for docker user. Docker is just container. So, please remove all the installed NextEPC package including MongoDB and Web User Interface.

  • Write Dockerfile with some empty directory.
acetcom@nextepc:~$ mkdir -p docker
acetcom@nextepc:~$ cd docker
acetcom@nextepc:~/docker$ sudo sh -c 'cat << EOF > Dockerfile
FROM ubuntu:latest
RUN \
  apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv F26B05AF && \
  echo "deb http://ppa.launchpad.net/acetcom/nextepc/ubuntu xenial main" > /etc/apt/sources.list.d/nextepc.list && \
  apt-get update && \
  apt-get install -y nextepc
WORKDIR /root
EOF'
  • Create Docker Image with building Dockerfile
acetcom@nextepc:~/docker$ docker build -t nextepc .
  • Create Docker Container(Use systemd - /sbin/init, Docker Network Type is host)
acetcom@nextepc:~/docker$ docker run --net=host --hostname nextepc -ti --name nextepc --privileged --cap-add=SYS_ADMIN -e "container=docker" -v /sys/fs/cgroup:/sys/fs/cgroup -v $PWD:/mnt nextepc /sbin/init
  • Go container using /bin/bash
acetcom@nextepc:~/docker$ docker exec -it nextepc /bin/bash
  • In Container(Install WebUI)
root@nextepc:~# apt-get -y install curl sudo
root@nextepc:~# curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
root@nextepc:~# curl -sL http://nextepc.org/static/webui/install | sudo -E bash -
root@nextepc:~# systemctl start nextepc-webui
  • In Container(Configuration)
Edit /etc/nextepc/mme.conf
Edit /etc/nextepc/sgw.conf
root@nextepc:~# systemctl restart nextepc-mmed
root@nextepc:~# systemctl restart nextepc-sgwd

Add Subscriber using WebUI
  • In Host(Update iptables)
acetcom@nextepc:~/docker$ sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
acetcom@nextepc:~/docker$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
acetcom@nextepc:~/docker$ sudo iptables -I INPUT -i pgwtun -j ACCEPT

Now, turn on eNodeb and UE. In my case, ping is working.

Thanks!

@amilenovic

This comment has been minimized.

amilenovic commented Feb 15, 2018

Thanks for that. Building docker image works.

However, docker run with /sbin/init is a problem for me. I use Fedora VM where I need to be a privileged user by default and have to use sudo docker run. And when I supply /sbin/init my Fedora VM restarts.

I think I can get around this. But, I wonder do I need to have a service created inside running container? Can I simply just invoke CMD nextepc-epcd? If it fails, my container will exit, but my container runtime can handle such event. Even for development, this is more suitable.

The obvious issue with this approach are Mongo and WebUI which are now part of the same container. and have to be started alltogether. However, I believe that they can be 'containerized' in additional 2 containers. Probably one docker-compose file can be the simplest way to connect these 3 containers and have NextEPC up and running. Let me know what you think. I can offer help here.

Thanks again for the effort.

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 17, 2018

You're right. My initial proposal is generally not used in docker world. Of course, in a VM like fedora that requires root privileges, if we are using systemd(/sbin/init), the VM will be restarted.

To make this discussion a litter further, I've checked-in a docker-compose.yml for WebUI/MongoDB in git repository. It's okay up to here.

Next, we need to include MME/SGW/PGW/HSS/PCRF in docker-compose.yml.

As I know, we cannot expose SCTP port from docker container.(e.g "36412:36412/sctp"). Right? If it is true on docker latest version, eNodeB cannot connect to MME in docker container on default docker running network configuration. At this point, I'm hesitant how to improve docker-compose.yml.

Of course, regardless of systemd problem, we can fix this with --net=host. However, I'm looking for a better solution for NextEPC docker network setup.

What do you think is the best solution of NextEPC docker-compose.yml for us?

Thank you for your advice!

@laf0rge

This comment has been minimized.

Contributor

laf0rge commented Feb 17, 2018

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 17, 2018

Hi, Harald

docker-playground seems to be a really good example for NextEPC.
I wanted to use docker to set up test automation. Now, let me learn how to apply it.

Thank you very much for giving me such a good reference.

acetcom added a commit that referenced this issue Feb 20, 2018

@acetcom

This comment has been minimized.

Owner

acetcom commented Feb 22, 2018

We are really sorry that Docker does not support "36412:36412/sctp". There were a lot of plans, but now it does not seem to be suitable for use in a runtime environment.

So how could we use this? We first decided to use this for testing automation for various Linux distributions. It also supports several development environments for developers.

Let us explain an initial proposal below.

  • Test Automation

Run ./check.sh as shown below. This will create an image for all distributions supported by NextEPC. And then, it checks whether there are no problems with all distributions.
(Note that at first it takes a lot of time to build all docker images.)

acetcom@nextepc:~/git/nextepc/support/docker$ ./check.sh

This script runs internally as follows:

docker-compose build
docker-compose -f docker-compose.yml -f docker-compose.test.yml run test
...
DIST=fedora docker-compose build
DIST=fedora docker-compose -f docker-compose.yml -f docker-compose.test.yml run test
...
DIST=debian TAG=jessie docker-compose build
DIST=debian TAG=jessie docker-compose -f docker-compose.yml -f docker-compose.test.yml run test
...
  • Development Environment

The development environment we use in Docker is as follows.

acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose build dev
acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose run dev
Creating nextepc-mongodb ... done
Creating docker_base_1    ... done
acetcom@nextepc-dev:~$ 

The docker-compose build dev could be run only once at the beginning. Moreover, if you had run ./check.sh, all images were created by docker-compose build, so you only need to docker-compose run dev. If the MongoDB service is not running, it will be loaded automatically.

acetcom@nextepc:~/git/nextepc/support/docker$ DIST=centos docker-compose build dev
acetcom@nextepc:~/git/nextepc/support/docker$ DIST=centos docker-compose run dev
Creating nextepc-mongodb ... done
Creating docker_base_1    ... done
acetcom@nextepc-dev:~$ 

An example of creating a CentOS development environment is shown above.

  • WebUI & MongoDB for Host

Sometimes we will need MongoDB and WebUI. This is also true when running NextEPC in a Host environment. So far we had to install MongoDB and WebUI on the Host. Now, in the Docker environment, you can simply create as below.

Note that if you have MongoDB and WebUI running on the host, you should stop it.

acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose up -d
Creating nextepc-webui ... done
nextepc-mongodb is up-to-date
Starting docker_base_1 ... done
Creating docker_dev_1     ... done
Creating docker_build_1   ... done

Do you want to create MongoDB and WebUI as a Docker service? Or do you want to install it on the host and run it on your host? Both Host and Docker can access MongoDB and WebUI service if they were created as Docker. However, when MongoDB and WebUI is running on the Host, only the NextEPC daemon running on the Host can use them.

  • Runtime Environment

As mentioned, docker does not support SCTP. For TCP/UDP, docker uses DNAT internally. SCTP will do that too. But this is not fancy way. So, we select network mode as a host for sharing network between Docker and Host.

Note that if you launch this runtime environment, TUN device(pgwtun) is created on the host, not inside the Docker. And also, docker says that --network=host is only supported in Linux-Host. MacOS/Windows might not be working in this Runtime Environment.

acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose -f docker-compose.yml -f docker-compose.run.yml run run
acetcom@nextepc:~/git/nextepc/support/docker$ DIST=debian TAG=jessie docker-compose -f docker-compose.yml -f docker-compose.run.yml run run

If you want to update the configuration of runtime environment,

acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose -f docker-compose.yml -f docker-compose.run.yml run run /bin/bash
Starting nextepc-mongodb ... 
Starting docker_base_1 ... done
Starting docker_build_1 ... done
root@nextepc-run:~# apt-get install vim
root@nextepc-run:~# vi /etc/nextepc/nextepc.conf 
root@nextepc-run:~# exit
acetcom@nextepc:~/git/nextepc/support/docker$ docker-compose -f docker-compose.yml -f docker-compose.run.yml run run

It's just an early work. Ideas to improve are good for us.

Thanks!

@InfernoZeus

This comment has been minimized.

InfernoZeus commented Sep 17, 2018

I've just stumbled upon your project. FYI, the latest version of docker, 18.03, added support for SCTP port mappings.

@acetcom

This comment has been minimized.

Owner

acetcom commented Sep 17, 2018

Yes, SCTP port mapping is supported now. But sctp kernel module is not loaded in the docker for mac and windows.

If they support sctp kernel module, we will be re-structuring docker related stuff.

Thanks a lot!

@acetcom

This comment has been minimized.

Owner

acetcom commented Sep 20, 2018

As we mentioned earlier, sctp does not work properly on Mac/Windows.

Today, we tried to load the sctp kernel module into the docker environment using MacOSX as the Host. The results are not very good, but we would share the way we did below.

First, we need to figure out the version of linux kernel used by the docker.

$ docker run -it --rm alpine
 / # uname -a
Linux 2f222c9a9225 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 Linux

Linux Kernel Version is 4.9.93. Then, we need to create a Dockerfile to compile the Linux Kernel Module. Here is the Dockerfile we use below.

FROM alpine:latest
MAINTAINER Sukchan Lee <acetcom@gmail.com>

# install alpine packages
RUN apk add --no-cache --update \
    # build essentials
    abuild \
    bc \
    binutils \
    build-base \
    cmake \
    gcc \
    ncurses-dev \
    sed \
    # tools
    ca-certificates \
    wget

# download kernel sources
ARG KERNELVER=4.9.93
RUN wget -nv -P /srv https://www.kernel.org/pub/linux/kernel/v4.x/linux-$KERNELVER.tar.gz \
 && tar -C /srv -zxf /srv/linux-$KERNELVER.tar.gz \
 && rm -f /srv/linux-$KERNELVER.tar.gz

# build kernel modules
RUN cd /srv/linux-$KERNELVER \
 && make defconfig \
 && ([ ! -f /proc/1/root/proc/config.gz ] || zcat /proc/1/root/proc/config.gz > .config) \
 # enable modules
 && echo 'CONFIG_IP_SCTP=m' >> .config \
 # build modules
 && make oldconfig \
 && make modules_prepare \
 && make modules \
 && make modules_install \
 && make clean

WORKDIR /srv/linux-$KERNELVER

Now, let's make the docker image to build kernel module based on alpine. Do not forget that the KERNELVER environment variable must match the Docker Linux Kernel Version found above.

$  docker build --build-arg KERNELVER=4.9.93 -t alpine-kernel-modules .

If an error occurs, try again. We sometimes saw such an error.

The sctp.ko file will be created in the alpine-kernel-modules images. We will copy this into the Host machine in the following way.

 $ docker run -it --rm --privileged --volume ${HOME}:/mnt alpine-kernel-modules sh
 / # cp /lib/modules/4.9.93-linuxkit-aufs/kernel/net/sctp/sctp.ko /mnt

Get the NextEPC source code from github.

$ git clone https://github.com/acetcom/nextepc

Modify nextepc/support/docker/docker-compose.yml using the appropriate editor like vi.

diff --git a/support/docker/docker-compose.yml b/support/docker/docker-compose.yml
index 77404969..3c54c1f9 100644
--- a/support/docker/docker-compose.yml
+++ b/support/docker/docker-compose.yml
@@ -70,6 +70,7 @@ services:
     hostname: nextepc-dev
     user: ${USER}
     entrypoint: /bin/bash -c "/bin/bash -c \"$${@}\""
+    privileged: true
     command: |
         /bin/bash -c "
             sudo chmod 666 /dev/net/tun

Everything is ready. Now compile and run NextEPC as shown below.

$ cd nextepc/support/docker
$ docker-compose run dev
acetcom@nextepc-dev:~$ sudo apt-get install kmod
acetcom@nextepc-dev:~$ sudo insmod /mnt/sctp.ko
acetcom@nextepc-dev:~$ mkdir git
acetcom@nextepc-dev:~$ cd git/
acetcom@nextepc-dev:~/git$ git clone https://github.com/acetcom/nextepc
acetcom@nextepc-dev:~/git$ cd nextepc/
acetcom@nextepc-dev:~/git/nextepc$ autoreconf -if;./configure --prefix=`pwd`/install;make -j 2
acetcom@nextepc-dev:~/git/nextepc$ make install
acetcom@nextepc-dev:~/git/nextepc$ ./test/testepc
  PID[35478] : '/home/acetcom/git/nextepc/install/var/run/nextepc-epcd/pid'
  File Logging : '/home/acetcom/git/nextepc/install/var/log/nextepc/nextepc.log'
  MongoDB URI : 'mongodb://mongodb/nextepc'
  Configuration : '/home/acetcom/git/nextepc/install/etc/nextepc/nextepc.conf'
s1ap_message_test   : SUCCESS
nas_message_test    : SUCCESS
gtp_message_test    : SUCCESS
security_test       : SUCCESS
s1setup_test        : SUCCESS
attach_test         : SUCCESS
volte_test          : SUCCESS
handover_test       : SUCCESS
All tests passed.

However, the problem occurs next. When you exit the container that loaded the sctp module, the Docker server is killed. I cannot run rmmod or modprobe -rf before exiting. This part is really not good for us.

Enjoy!

@acetcom

This comment has been minimized.

Owner

acetcom commented Sep 29, 2018

Docker for MAC(Edge version) now supports the SCTP kernel module.
Docker people have stopped supporting AUFS.
So there is no dependency with AUFS kernel patch and SCTP is now available in D4M.

This is a really great news for us.
I believe that D4M(Stable version) will be updated sometime later.

You can find the SCTP kernel module in the following ways.

$ docker run -it --privileged --pid=host ubuntu nsenter -t 1 -m -u -n -i sh
/ # ls /lib/modules/4.9.125-linuxkit/kernel/net/sctp/ -al
total 1761
drwxr-xr-x    1 root     root          2048 Sep  7 08:21 .
drwxr-xr-x    1 root     root          2048 Sep  7 08:21 ..
-rw-r--r--    1 root     root       1721784 Sep  7 08:21 sctp.ko
-rw-r--r--    1 root     root         76840 Sep  7 08:21 sctp_diag.ko

In addition, the NextEPC development in MAC OS X environment is easy to use as shown below.
(MongoDB, Network Configuration is automatically set in the docker container.)

  • In MAC OS X Host
$ git clone https://github.com/acetcom/nextepc
$ cd nextepc/support/docker
$ docker-compose run dev
  • In Docker Container
acetcom@nextepc-dev:~$ mkdir git
acetcom@nextepc-dev:~$ cd git/
acetcom@nextepc-dev:~/git$ git clone https://github.com/acetcom/nextepc
acetcom@nextepc-dev:~/git$ cd nextepc/
acetcom@nextepc-dev:~/git/nextepc$ autoreconf -if;./configure --prefix=`pwd`/install;make -j 2
acetcom@nextepc-dev:~/git/nextepc$ make install
acetcom@nextepc-dev:~/git/nextepc$ ./nextepc-epcd
NextEPC daemon v0.3.10 - Sep 29 2018 13:50:55

  PID[35276] : '/home/acetcom/git/nextepc/install/var/run/nextepc-epcd/pid'
  File Logging : '/home/acetcom/git/nextepc/install/var/log/nextepc/nextepc.log'
  MongoDB URI : 'mongodb://mongodb/nextepc'
  Configuration : '/home/acetcom/git/nextepc/install/etc/nextepc/nextepc.conf'
[09/29 13:51:38.268] PCRF try to initialize
[09/29 13:51:38.312] PCRF initialize...done
[09/29 13:51:38.314] PGW try to initialize
...
[09/29 13:51:39.166] s1ap_server() [172.18.0.2]:36412
[09/29 13:51:39.166] MME initialize...done

[09/29 13:51:39.167] INFO: NextEPC daemon start (main.c:157)
(Ctrl-C)
^C[09/29 13:52:08.990] INFO: SIGINT received (main.c:51)
[09/29 13:52:08.991] INFO: NextEPC daemon terminating... (main.c:160)
...
[09/29 13:52:09.463] DB-Client terminate...done
[09/29 13:52:09.466] DB-Client terminate...done
acetcom@nextepc-dev:~/git/nextepc$ ./test/testepc
  PID[35413] : '/home/acetcom/git/nextepc/install/var/run/nextepc-epcd/pid'
  File Logging : '/home/acetcom/git/nextepc/install/var/log/nextepc/nextepc.log'
  MongoDB URI : 'mongodb://mongodb/nextepc'
  Configuration : '/home/acetcom/git/nextepc/install/etc/nextepc/nextepc.conf'
s1ap_message_test   : SUCCESS
nas_message_test    : SUCCESS
gtp_message_test    : SUCCESS
security_test       : SUCCESS
s1setup_test        : SUCCESS
attach_test         : SUCCESS
volte_test          : SUCCESS
handover_test       : SUCCESS
All tests passed.

Enjoy!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment