Docker

[anitsh]

Docker

Docker Daemon: A constant background process that helps to manage/create Docker images, containers, networks, and storage volumes.
Docker Engine REST API: An API used by applications to interact with the Docker daemon; it can be accessed by an HTTP client.
Docker CLI: A Docker command line client for interacting with the Docker daemon. a.k.a the Docker command.

If we think differently we could just connect some problems with Docker:
As we all know Docker runs on a single process it could result into single point of failure.
All the child processes are owned by this process.
At any point if Docker daemon fails, all the child process losses their track
and enters into orphaned state.
Security vulnerabilities.
All the steps needs to be performed by root for Docker operations.

---

To understand why the Docker Daemon is running with root access and how this can become a problem, we first have to understand the Docker architecture (at least on a high level).

Container images are specified with the Dockerfile. The Dockerfile details how to build an image based on your application and resources. Using Docker, we can use the build command to build our container image. Once you have the image of your Dockerfile, you can run it. Upon running the image, a container is created.

The problem with this is that you cannot use Docker directly on your workstation. Docker is composed of a variety of different tools. In most cases, you will only interact with the Docker CLI. However, running an application with Docker means that you have to run the Docker Daemon with root privileges. It actually binds to a Unix socket instead of a TCP port. By default, users can only access the Unix socket using sudo command, which is owned by the user root.

The Docker Daemon is responsible for the state of your containers and images, and facilitates any interaction with “the outside world.” The Docker CLI is merely used to translate commands into API calls that are sent to the Docker Daemon. This allows you to use a local or remote Docker Daemon.

Running the Docker Daemon locally, you risk that any process that breaks out of the Docker Container will have the same rights as the host operating system. This is why you should only trust your own images that you have written and understand.

Resource:

• https://docs.docker.com/get-started/overview
• https://docs.docker.com/engine/reference/commandline/build
• https://docs.docker.com/engine/reference/commandline/run/
• Learn Docker - https://birthday.play-with-docker.com
  • https://birthday.play-with-docker.com/cli-formating
  • https://birthday.play-with-docker.com/context
  • https://birthday.play-with-docker.com/jenkins-docker-hub
  • https://birthday.play-with-docker.com/run-as-user
• Containerd integration in Docker, Architecture
• https://medium.com/better-programming/about-var-run-docker-sock-3bfd276e12fd - /var/run/docker.sock
  It’s the Unix socket the Docker daemon listens on by default, and it can be used to communicate with the daemon from within a container.
• Docker Image
  • How is a docker image build?
  • https://www.thegeekdiary.com/how-to-create-docker-image-from-a-container-and-dockerfile
  • https://www.thegeekdiary.com/beginners-guide-to-the-docker-world
  • https://www.scalyr.com/blog/create-docker-image
  • https://forallsecure.com/blog/demystifying-a-docker-image
  • https://github.com/ForAllSecure/rootfs_builder
  • https://forallsecure.com/blog/demystifying-a-docker-image
  • https://github.com/docker-library/repo-info
  • kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster. https://github.com/GoogleContainerTools/kaniko
  • Fast and flexible Docker image building tool, works in unprivileged containerized environments like Mesos and Kubernetes. https://github.com/uber/makisu
• Docker Security
  • https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
  • https://dreamlab.net/en/blog/post/kernel-capabilities-in-docker-a-fine-grained-access-control-system
  • https://www.youtube.com/watch?v=GR5O3BSbkYM
  • https://www.zdnet.com/article/docker-malware-is-now-common-so-devs-need-to-take-docker-security-seriously/
  • https://blog.talosintelligence.com/2020/12/xanthe-docker-aware-miner.html
• Debug Docker
  • https://blog.container-solutions.com/what-you-need-to-know-about-debugging-docker-containers
• Docker and Kubernetes
  • https://www.digitalocean.com/community/curriculums/kubernetes-for-full-stack-developers
  • https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-on-digitalocean-kubernetes-using-helm
• Automate Docker installation/removal with Ansible
  • https://www.digitalocean.com/community/tutorials/how-to-use-ansible-to-install-and-set-up-docker-on-ubuntu-18-04
  • https://www.rechberger.io/tutorial-install-docker-using-ansible-on-a-remote-server
• https://github.com/joe-elliott/tracing-example

[anitsh]

https://birthday.play-with-docker.com/cli-formating

Docker has organized commands into a docker
docker --help List out the nouns
docker noun --help List out the verbs or actions that can be performed by noun.

There is a format flag, we can type any raw string to simply output it, and then instructions to be parsed are included within {{ }}.

Variables injected into our template are prefixed with a dot '.'.

--format '{{json .}}' //Provides all the options available/

--format '{{json .}}' | jq . Beautify the output to more readable form by passing the output to jq

Simple Formatting
--format '{{.VARIABLE_NAME}}' Access the variable in from the output

Table Formatting
If we want to display more columns, and with the table layout, we can use the table syntax with \t between each column:

docker container ls -a --format 'table {{.ID}}\t{{.Image}}\t{{.Command}}\t{{.Status}}'

We can add more complicated logic with the template. Lets take a look at the environment variable list for the last run container and print each of the elements of the PATH variable on separate lines.

docker container inspect --format '{{range .Config.Env}}{{with split . "="}}{{if eq (index . 0) "PATH"}}{{range ( split (index . 1) ":" ) }}{{printf "%s\n" .}}{{end}}{{end}}{{end}}{{end}}' $(docker container ls -lq)

[anitsh]

Went through this resource along with the tutorial

docker run busybox // Exists with empty output
docker run -it busybox sh //Run busy box and access sh shell
docker run busybox echo "hello from busybox"

With all the above commands, Docker client finds the image (busybox in this case), loads up the container and then runs a command in a new container each time.

docker rm $(docker ps -a -q -f status=exited) //Delete all the containers exited
docker container prune //Does the same

[anitsh]

Docker Context: allows you to connect to remote docker instances.

export TGT_HOST=10.0.0.4 // Set environment variable
ssh-keyscan -H $TGT_HOST >~/.ssh/known_hosts // Trust the host

Connecting to Nodes with DOCKER_HOST // PREVIOUS WAY TO DO
DOCKER_HOST=ssh://$TGT_HOST docker container ls
The DOCKER_HOST variable has various syntaxes, including tcp:// to connect to an export port, and unix:/// to connect to a local socket. The easiest method to configure securely uses ssh as we did above. Connecting via ssh requires:

• Access to login with ssh to the remote host.
• The ability to run docker commands on that host as your user (typically configured by adding your user to the docker group on the remote host).
• Docker client release 18.09 or newer on both the local and remote host

Using Docker Context //The new way to do with 19.03 docker release
unset DOCKER_HOST // Make sure to unset this environment variable

docker context create term2 --description "Terminal 2" --docker "host=ssh://$TGT_HOST" // Create new docker context

docker context use term2 // Switch and use new context named term2

docker --context default container ls // Run docker command in other contexts.

[anitsh]

docker exec -it <container name> /bin/bash // Get a bash shell in the container

docker exec -it <container name> <command> // Execute command in the container

Do not automatically restart the container when it exits. This restart policy is the default. Therefore both commands are the same.

docker run app
docker run --restart=no app

Only restart the container if it exits with a non-zero code.

docker run --restart=on-failure app

Same as above, but limit the number of restart attempts to 10.

docker run --restart=on-failure:10 app

Always restart the container, regardless of the exit code. On Docker daemon startup, only start the container if it was already running before.

docker run --restart=unless-stopped app

Always restart the container, regardless of the exit code. The container will also start when the Docker daemon starts, even if it was in a stopped state before.

docker run --restart=always app

Resources:

• https://maximorlov.com/pm2-to-docker-automatic-restarts
• https://maximorlov.com/from-pm2-to-docker-cluster-mode

[anitsh]

https://birthday.play-with-docker.com/jenkins-docker-hub

Ran docker container for .net core app.

Created a token in dokcer hub.
Registered the token in Jenkins.

git clone https://github.com/sixeyed/pi.git && cd pi && git checkout bday7
docker-compose -f jenkins.yml up -d
docker container run -d -p 80:80 pi:bday7 -m web

[anitsh]

https://birthday.play-with-docker.com/run-as-user

Summary:
Create a user inside the container image.
Tell Docker to use this user.
Create and configure permissions on any user writable directories.
Configure the application to write to user writable directories.
Take extra precautions for any host volumes.
Configure any listening ports to be above 1024 inside the container.
Use docker exec args to run commands as root, rather than sudo.

QnA for preview of the content

Q. What line in a Dockerfile changes the user Docker uses to run commands?
A. USER app:app

Q. What port restrictions exists when running commands as a non-root user?
A. Commands on the host cannot be published to ports below 1024.

Q. Where do named volumes get their file permissions?
A. They are initialized with the file owner and permission from the image.
A. Existing volumes maintain the state from the previous usage.

Main objective Converting Images to Run Without Root

Docker Hub has lots of popular images that are configured to run as root. This makes them very convenient for developers looking to get started quickly with the fewest complications. However for security, it’s recommended to run our containers as a non-root user. This exercise will walk you through some of the steps needed to run containers without root.

docker image pull nginx:1.16

docker image inspect nginx:1.16

    ....
    "User": "" // The default, when there is no user, is to **run as the root user**.
 }

Create new user in the Docker file.

FROM **nginx:1.16** 
RUN useradd -u 5000 app // `5000` is the user id, `app` is the user name
USER app:app // Tell docker to use the new user  

Fixing file permissions
Build the new image, inspect and run container:
docker build -t user/nginx:1.16-2 .
docker image inspect user/nginx:1.16-2
docker container run --rm user/nginx:1.16-2

Now configuration issues and permission issues will appear. By default nginx is the default user.

We need to update the nginx configuration of the container.

Extract the content from the container.

mkdir -p conf
docker container run user/nginx:1.16-2 tar -cC /etc/nginx . | tar -xC conf

This creates a conf directory that contains the nginx.conf and conf.d/default.conf.

Now update the config file.

• Edit the nginx.conf to remove or comment out the user nginx.
• Set paths for the following variables to be locations that the new user can write:
  pid
  client_body_temp_path
  fastcgi_temp_path
  proxy_temp_path
  scgi_temp_path
  uwsgi_temp_path

We will use /var/run/nginx/nginx.pid for the pid file, and /var/tmp/nginx/* for the other paths.
Updated nginx.conf file:

# user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
# Add the below pid
pid        /var/run/nginx/nginx.pid;


events {
    worker_connections  1024;
}


http {
    # add the below paths
    client_body_temp_path /var/tmp/nginx/client_body;
    fastcgi_temp_path /var/tmp/nginx/fastcgi_temp;
    proxy_temp_path /var/tmp/nginx/proxy_temp;
    scgi_temp_path /var/tmp/nginx/scgi_temp;
    uwsgi_temp_path /var/tmp/nginx/uwsgi_temp;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Updated Docker file:

  FROM nginx:1.16
  RUN useradd -u 5000 app \
  && mkdir -p /var/run/nginx /var/tmp/nginx \
  && chown -R app:app /usr/share/nginx /var/run/nginx /var/tmp/nginx
  COPY conf/nginx.conf /etc/nginx/nginx.conf
  USER app:app

Handling ports

Lets build the image with the updated configuration:

docker build -t user/nginx:1.16-3 .

And now try to run that image:

docker container run --rm user/nginx:1.16-3

This will give another permission error: nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

When applications aren’t root, they cannot listen on ports below 1024, so our web server listening on port 80 and 443 will not work. But, inside the container we can listen on a higher numbered port. And even better, with Docker we can still publish to a lower numbered port on the host, and map that high numbered port inside the container. To edit the port nginx listens on, edit the conf/conf.d/default.conf file and replace: listen 80; with: listen 8080;

Updated Docker file:

  FROM nginx:1.16
  RUN useradd -u 5000 app \
  && mkdir -p /var/run/nginx /var/tmp/nginx \
  && chown -R app:app /usr/share/nginx /var/run/nginx /var/tmp/nginx
  COPY conf/nginx.conf /etc/nginx/nginx.conf
  USER app:app
  COPY conf/conf.d/default.conf /etc/nginx/conf.d/

Rebuild image:
docker build -t user/nginx:1.16-4 .

Run that image detached, with the port mapping, and a container name:
docker container run -d -p 80:8080 --name nginx user/nginx:1.16-4

Using Volumes
Create a folder and new index.html file, and mount the folder

docker container stop nginx
docker container rm nginx
docker container run -d -p 80:8080 --name nginx -v "$(pwd)/html:/usr/share/nginx/html" user/nginx:1.16-4

We get a permission denied again. We can see that the files on the host and inside the container are owned by the same UID/GID and with the same permissions. There’s no mapping from the host user to the container user when running on Linux. There are a few ways to handle this:

Ensure the UID/GID of files on the host matches those of the container user.
Fix permissions inside the image, and only use named volumes which initialize from the image contents.
Avoid using host volumes on directories where the container will write.

Note that named volumes are only initialized when they are first created, so you only want to use these for persistent data, and not the contents of the image that you want to update with each new image.

Sudo Access
One last challenge users face when switching away from running everything as root is getting sudo access inside the container. Try to run an apt command and see what happens:

docker container exec -it nginx apt-get update

That fails without root access. If we try running sudo inside the container, what happens:

docker container exec -it nginx sudo apt-get update

Images are minimal, shipping only with the needed commands. And in containers sudo is not needed since it would be a security vulnerability (what’s the point of running as a user if that user can sudo to root) and there are better ways to get root inside of a container. The docker container exec command runs our command in the container namespace with the same settings like environment variables, working directory, and user, that the docker container run command uses to start the container. However, the docker container exec command gives options to override those settings, have a look at the help output to see how we can change the user:

docker container exec --help

Try running an apt-get update command inside the container as root instead of our app user.
Solution

docker container exec -it --user root nginx apt-get update

Extra Difference in run, exec and attach
docker run starting a new docker container.
docker exec running new things on a running container.
docker attach connecting a running container's main process( PID 1) standard input/output/error with requesting medium standard I/O.

When a container is started using /bin/bash then it becomes the containers PID 1 and docker attach is used to get inside PID 1 of a container. So docker attach < container-id > will take you inside the bash terminal as it's PID 1 as we mentioned while starting the container. Exiting out from the container will stop the container.
Whereas in docker exec command you can specify which shell you want to enter into. It will not take you to PID 1 of the container. It will create a new process for bash. docker exec -it < container-id > bash. Exiting out from the container will not stop the container.

• https://stackoverflow.com/questions/30960686/difference-between-docker-attach-and-docker-exec

[anitsh]

Remove Docker from Ubuntu

dpkg -l | grep -i docker : Give me a list of packages that contains the word “docker” in them

Stop and disable service

sudo systemctl stop/disable docker

sudo rm -rf /etc/systemd/system/docker.service.d

Remove all Docker packages.

sudo apt-get purge -y docker-engine docker docker.io docker-ce docker-ce-cli
sudo apt-get autoremove -y --purge docker-engine docker docker.io docker-ce
The “-y” flag here is to answer “yes” to the command prompt when it asks you whether to remove a package.

Remove all docker related files

sudo rm -rf /var/lib/docker /etc/docker
sudo rm /etc/apparmor.d/docker
sudo rm -rf /var/run/docker.sock

Remove group

sudo groupdel docker

Deactivate network interface and ethernet bridge, and delete

ifconfig docker0 down
brctl delbr docker0

Resource

• https://trello.com/c/c1RRgE2b/34-docker
• https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes

[anitsh]

Log

• https://www.mtgimage.org/how-to-view-docker-logs-in-linux

[anitsh]

Image Optimization

• Reduce the image size by using the flag --no-install-recommends in Dockerfiles - Article