"Docker is a platform for building, running, and shipping applications in a consistent manner, so if your application run flawlessly in your development machine, it can run and function the same way in other machines."
If you have faced with the problem of smoothly running your application in a development machine, but not working in a different machine, this might be stemming from:
- One or more files are missing
- Software version mismatch
- Different configuration settings, such as env variables and etc.
In these cases, Docker comes to the rescue.
With Docker, we can package our application with whatever configuration, files it needs and use it any machine we want
As it is seen here, we can package our applications with docker so that it uses specific versions of technologies needed. This will cause every machine to be using the same versions.
When you tell Docker to bring up your application with docker-compose up, Docker will automatically download and run these dependencies inside an isolated environment, called Container.
This is the beauty of Docker! This isolated environment allows multiple applications use different versions of software side by side.
As you can see here, while one application uses Node version 14, the other one uses Node version 9, and they can run side by side on the same machine without messing up with each other.
- Also, if we want to get rid of one application, i.e., we don't want to use it anymore, then we can erase that app and its dependencies with a single Docker command. This helps us not storing that application's dependencies in our machine, but instead in the docker container.
| Container | VM |
|---|---|
| An isolated environment for running an application | An abstraction of a machine or physical hardware, so that we can run several VMs in a physical machine |
Regarding software development, we can run our applications in isolation inside virtual machines.
So, on a single physical machine, we can run two applications in isolation on two different virtual machines with each app has dependencies they need.
All these are running on the same machine but with different isolated environments. That's one of the benefits of virtual machines.
- Each virtual machine needs a copy of an OS that needs to be licensed, patched, and monitored.
- Slow to start, because the entire OS needs to be booted up, just like a physical computer.
- Resource intensive(Each VM runs with depending on physical resources such as RAM, HDD, CPU and etc.)
- So if you run multiple VMs in a single computer, the computer resources must be portioned between these VMs.
- This limits the number of VMs that we can run on a physical machine...
- Containers doesn't have this limitation..
- Allow running multiple apps in isolation
- Lightweight(doesn't need a full OS)
- All containers on a single machine share the OS of the host.
- This means we need to license, patch, and monitor a single operating system.
- Also, since the OS is dependent on the host, the container will start up pretty quickly.
- Need less hardware resources.
- This means in a host, we can run 10s and 100s of containers side by side.
Docker uses a Client-Server architecture. So it has a client component that talks to the server component using a REST API.
The Server, also called Docker Engine, sits on the background and takes care of building and running docker containers.
Technically a container is just a process, like the other processes running on a computer.
Before, we said that containers share the OS of the host. But in fact, they share the Kernel of the host. Kernel is the core of the OS, it manages all aplications and hardware resources. Every OS has its own kernel, that's why we cannot run a Windows application on Mac or Linux, because under the hood this application talks to the Kernel of the underlying OS.
So,
- On a Linux machine, we can only run Linux containers.
- On a Windows machine, we can run both Linux and Windows containers, because Win10 is now shipped with a custom built Linux kernel.
- On a Mac machine, since Mac doesn't have a container supporting kernel, Docker runs a lightweight Linux VM to run the containers.
1- We take an application and dockerize it. This means to make a small change so that it can be run by Docker.
- How? We just add a
Dockerfileto it. Dockerfileis a plaintext file that includes instructions that Docker uses to package up this application into an image.- This image contains everything our application needs to run.
The image contains:
- A cut-down OS
- A runtime environment(e.g., Python)
- Application files
- Third-party Libraries
- Environment variables
We create a Dockerfile and give it to Docker for packaging our application into an image.
Once we have an image, we tell Docker to start a Container using that image. Through docker run ... we tell Docker to start that application inside a container, an isolated environment.
We can push this image into DockerHub. DockerHub to Docker is like Github to Git. It's a storage for Docker images anyone can use.
Once our application image is on DockerHub, we can put it on any machine that runs Docker
We are going to make a small demo. In this demo, we will dockerize our 'hello-docker' application.
1- Add Dockerfile inside your project. It's important that Dockerfile, with capital'D'.
Here, we have our main.py, which we want to run in a Docker container, and our Dockerfile.
2- Inside Dockerfile
We have:
FROM: FROM is used for setting a baseImage to use for subsequent instructions. FROM must be the first instruction in a Dockerfile. Also, a base image is an image we pull from DockerHub on which we build our Dockerfile. Here, we use python:alpine, which is a Python image that runs on a Linux container which hasalpinedistribution.COPY: COPY is used for copying files we want to the base image of choice. Here, using.is to say that we want to copy every file in the directory to/appdirectory. Important thing to note is that/appis located on the image, not on our local machine.WORKDIR: WORKDIR is used for setting up the working directory of the base image. Here, by setting it to/app, we declare that the commands we run will be run on this directory, unless otherwise stated.CMD: CMD stands for command. This is the same thing that we use to run anything on the terminal. With this logic, to run main.py, we just writeCMD python main.py.
3- Having written the Dockerfile, we can create an image by docker build -t hello-docker ..
4- We need to check whether our image is created successfully by docker image ls
We can also verify this from Docker Desktop app.
5- Test!
With the command docker run <image-id> we have
6- We need to push the image to DockerHub so that we can use the image on another machine.
create a repository..
Then, push..
7- Test through play-with-docker
Go here
Pull the Docker image with docker pull username/repository:tag.
And run docker run username/repository:tag
| Image | Container |
|---|---|
| Blueprint of the container | Instance of the image |
| Image is a logical entity | Container is a real world entity |
| Image is created only once | Containers are created any number of times using image. |
| Images are immutable | Containers changes only if old image is deleted and new is used to build the container. |
| Images does not require computing resource to work. | Containers requires computing resources to run as they run as Docker Virtual Machine. |
| To make a docker image, you have to write script in Dockerfile. | To make container from image, you have to run docker build <image-name> command |
| Docker Images are used to package up applications and pre-configured server environments. | Containers use server information and file system provided by image in order to operate. |
| Images can be shared on Docker Hub. | It makes no sense in sharing a running entity, always docker images are shared. |
| There is no such running state of Docker Image. | Containers uses RAM when created and in running state. |
We need to know about Linux CLI since Docker has its foundations in Linux, this is, Docker is built on basic Linux concepts.
Go here and pull Ubuntu distribution of Linux
After the image is pulled, execute docker run -it ubuntu. This will run the image interactively
Here, we successfully got to the shell. root@574ab7f5810e:/# means the following:
rootrepresents the currently logged in user.rootuser has the highest priviledges.574ab7f5810eis the name of the machine./represents where we are in the file system. This is the root directory. The root directory is the highest directory in the system.#means we have the highest priviledges because we logged in asroot. If we logged in as a normal user, we'd see$.
- Echo prints out the value of what is called
- whoami prints the current logged in user
- Echo $0 prints the location of the shell program
- history prints the history of commands executed
- ! runs the command listed in history
- Linux is case sensitive and uses
/for directories - pwd prints the working directory (print working directory)
In software development, we currently use package managers such as pip, npm and so on.
In Linux, we have apt (advanced package tool)
We can use apt to install packages(executable files)
Here, we just installed Python3
In Linux, everything is a file!
Port binding is an important concept. Since a running container can be regarded as another machine, its ports are different from host's ports.
Here, the first container is bound with host's 5000 port to its 5000 port.
The second ant third containers both have their 3000 port bound to host's 3000 and 3001 ports.
To put it more clearly, when you send a request to port 5000 from your computer, it will interact with port 5000 running container of Docker. Also, when you send a request to port 3000 of your computer, it will interact with port 3000 of container, and the same thing goes for port 3001 of your computer as well.
Let's say that you already started running an nginx container with docker run -p 5000:3000 nginx:1.20.1
with docker ps we can see that our container is up and running..
if you want to start another container with same host port though, it will print out error. (Port is already allocated)
When we bind host's 5001 port to container 3000, we can see that both containers are running without any problems.
"Docker creates its own isolated network in which Docker containers run."
So, when we deploy two containers in the same Docker network, They can talk to each other by just using the container name. Without localhost, port number, etc.. Just the container name. Because they are in the same network.
Applications running outside the Docker network has to talk to these containers using localhost:port-number
Later on, when we package our application into its own Docker image and run as a Docker container, 3 Docker containers will be talking to each other in Docker network, i.e., Node application, MongoDB, and MongoExpress..
On top of these, we will connect to this network from outside by a Web Browser, as follows:
Use docker network create <network-name>.
After running docker network create mongo-network and docker network ls, we have the following picture:
So, since we want to run our mongo container and mongo-express container in this network, we have to provide network option as we start containers..
docker run -p 27017:27017 -d \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=password \
--name mongodb --net mongo-network mongo
Here, -e is for setting environment variables.
We also need to docker run mongo-express
docker run -p 8081:8081 -d \
-e ME_CONFIG_MONGODB_SERVER=mongo-db \
-e ME_CONFIG_MONGODB_ADMINUSERNAME=admin \
-e ME_CONFIG_MONGODB_ADMINPASSWORD=password \
--net mongo-network --name mongo-express mongo-express
After running these, we can check the connection's success status by docker logs <container-name>
docker logs mongo-express gives us
In the beginning of the log, it says Mongo Express server listening at http://0.0.0.0:8081, so everything is okay.
This is what we have at localhost:8081
"Docker Compose is used for running Docker containers as an alternative to manually typing docker run ... from terminal."
So far, we have run two Docker containers, and we used the following codes on terminal to run them.
docker run -p 27017:27017 -d \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=password \
--namemongodb --net mongo-network mongo
docker run -p 8081:8081 -d \
-e ME_CONFIG_MONGODB_SERVER=mongo-db \
-e ME_CONFIG_MONGODB_ADMINUSERNAME=admin \
-e ME_CONFIG_MONGODB_ADMINPASSWORD=password \
--net mongo-network --name mongo-express mongo-express
This was super tedious and error-prone. It's always good to do things in a more structured manner. Docker Compose helps us doing that.
Docker Compose is a .yaml file.
This is the Docker Compose yaml structure. Here, we do a translation from docker run command to .yaml file.
version: '3'
services:
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=passwordHere,
- version: '3' -> Version of Docker Compose
- services: -> Container list is under services
- mongodb: -> Container name
- ports: -> HOST:CONTAINER
- environment: -> Environment variables
As you can see, we do not have to declare --network flag here. This is just so because Docker Compose runs containers in the same Docker network.
Together with mongo-express,
version: '3'
services:
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
mongo-express:
image: mongo-express
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=admin
- ME_CONFIG_MONGODB_ADMINPASSWORD=password
- ME_CONFIG_MONGODB_SERVER=mongodbTo remind once again, Docker Compose takes care of creating a common network.
docker-compose -f mongo.yaml up
In the following picture, you can see that Docker Compose is automatically creating a Docker network.
As you can see, in the first line it says Creating network "myapp_default" with the default driver.
Name of the network is: app_default
Note that everytime we remove or stop a container and create or restart it again, we lose the data and all the container configurations! This problem will be fixed with Volumes.
In order to stop both containers at the same time, we can use docker-compose -f mongo.yaml down.
"Dockerfile is a blueprint for creating Docker images."
We're gonna create a Docker image from our Node, JS application.
- The first line of every Dockerfile is
FROM <some-image>. - We have to base our image on a base image with
FROM. - Second, we want to configure our environment variables with
ENV. (Optional since we already set env vars in mongodb and mongo-express containers) - Third,
RUN->RUN mkdir -p /home/app - Fourth,
COPY->COPY . /home/app - Last one,
CMD->CMD ["node","server.js"]
Here, COPY and RUN are not interchangable since COPY copies files from the host machine to the container, but RUN runs commands inside a container.
Also, CMD is an entry point command, which means that it can be only one in a Dockerfile, on the other hand there can be multiple RUN commands.
Also, we are basing our image on a Node environment, which means that when the container is run, we don't have to install Node.
FROM node:13-alpine
RUN mkdir /home/app
COPY . /home/app
WORKDIR /home/app
CMD ["node","server.js"]Dockerfile name must be "Dockerfile"
docker build -t my-app:1.0 .
Here, -t is used for naming the image as my-app.
my-app:1.0's 1.0 is the tag of this image. It can be anything we want. For example, it can be my-app:version-1.
After completion, we can run docker images and see
my-app is created with tag 1.0.
Then, run the container of the image with docker run my-app:1.0.
Voila!
We can get inside the shell by docker exec -it <container id> /bin/sh
As you can see, everything we did in the Dockerfile is reflected here! The folders that we have is due to COPY and MKDIR commands we used.
1- Push the image you created to DockerHub.
I am going to use image burakhanaksoy/my-app:2.2.
2- Change Docker compose .yaml file as follows:
version: "3"
services:
my-app:
image: burakhanaksoy/my-app:2.2
ports:
- 3000:3000
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
mongo-express:
image: mongo-express
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=admin
- ME_CONFIG_MONGODB_ADMINPASSWORD=password
- ME_CONFIG_MONGODB_SERVER=mongodbHere, we just added this part:
my-app:
image: burakhanaksoy/my-app:2.2
ports:
- 3000:30003- Start multiple containers with Docker compose as follows:
docker-compose -f mongo.yaml up
4- Test!
Go to localhost:3000, and
Voila!
To run the app:
1 - Clone the repository and cd app
2- In the directory, run docker-compose -f mongo.yaml up
3- Test!
Go to localhost:3000
"Docker Volumes are used for data persistence in Docker."
There might be different scenarios in which we might have to use Docker Volumes. One of them is data persistence.
- As we experienced here, using mongodb, or any other db, as a container itself is not enough for keeping the state of the db.
- This is to say that everytime we restart db container, our data is gone!
Docker Volumes is used to help maintain the state (data) of the db.
Here, our container has its own virtual file system and whenever we stop/start the container, data in its virtual file system is gone and starts from a fresh state.
Through Docker Volumes we actually mount the physical file system of our host to the virtual file system inside the container.
Here, one thing is very important to note. File systems are replicated between host and container. This is to say that when some data change inside host file system, same change happends inside container file system, and vice-versa.
So even though we restart the container with a fresh state, it's file system is automatically cloned to the host's file system and this solves the problem.
docker run -v /home/mount/data:/var/lib/mysql/data- Here, the first one is the host file system directory, second one is the container file system directory.
- In this type, you can decide where on the host file system the reference is made
In this type, you only specify the container file directory. We don't specify which directory on the host should be mounted.
docker run -v /var/lib/mysql/data
For each container a folder is generated that gets mounted.
These types of volumes are called Anonymous Volumes because you don't have a reference to this automatically generated folder.
This is very similar to the Automatically Created Directory volume type, but it differs in that it specifies the name of the folder in the host file system.
docker run -v name:/var/lib/mysql/data
This type should be used because it's very beneficial and succinct.
Here we use a Named Volume and db-data is the reference name and /var/lib/mysql/data is the name of the path in the container.
You may have other container instructions in Docker compose .yaml file, but you need to write
volumes:
db-dataonce again at the end of the .yaml file.
here,
volumes:
db-datashould be at the same level as services.
1- Change the Docker Compose .yaml file.
version: "3"
services:
frontend:
image: burakhanaksoy/frontend:1.0
ports:
- 8080:8080
backend:
image: burakhanaksoy/backend:1.0
ports:
- 8000:8000
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
volumes:
- mongo-data:/data/db
mongo-express:
image: mongo-express
ports:
- 8081:8081
environment:
- ME_CONFIG_MONGODB_ADMINUSERNAME=admin
- ME_CONFIG_MONGODB_ADMINPASSWORD=password
- ME_CONFIG_MONGODB_SERVER=mongodb
volumes:
mongo-data:
driver: localHere, the last part
volumes:
mongo-data:
driver: localhere, driver: local is an additional information for Docker so that it creates a physical storage on a local file system.
mongo-data is the name reference for the volume.
And also inside
mongodb:
image: mongo
ports:
- 27017:27017
environment:
- MONGO_INITDB_ROOT_USERNAME=admin
- MONGO_INITDB_ROOT_PASSWORD=password
volumes:
- mongo-data:/data/dbthe last part
volumes:
- mongo-data:/data/dbhere /data/db is the path inside the mongodb container. It has to be the path where MongoDB persists its data.
In order to find the right filesystem path, in our case, for MongoDB, is /data/db, we can make a search on the Internet.
We can also check if this path exists by docker exec -it <container-id> /bin/sh
Note that each db has it's specific filesystem path, in other words path would likely to be different for MySQL or PostgreSQL. We should consult on the Internet.
For MySQL:var/lib/mysql
For PostgreSQL:var/lib/postgresql/data
2- Run docker-compose -f app.yaml up
3- Test!
Persists data into the DB and run docker-compose -f app.yaml up and docker-compose -f app.yaml down several times.
You'll see that our data won't disappear.
Awesome!
We finally made it! It was a nice 101. I want to thank Nana from TechWorld with Nana and Mosh Hamedani from Programming with Mosh for their amazing teaching skills and videos on YouTube. I watched their videos and prepared this document.
The videos I watched as I prepare this documentation is as follows: