Hey everyone,
I am on a jump box where I ssh into my EC2 instance that is running Amazon Linux with docker.
What I am needing to do is stop the docker service, run the docker daemon and apply some changes to the docker daemon from within the docker container itself.
What is happening is that after I stop the docker service I run dockerd and get the following error:
sudo dockerd --live-restore
INFO[0000] libcontainerd: new containerd process, pid: 6714
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
WARN[0001] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
ERRO[0001] [graphdriver] prior storage driver "devicemapper" failed: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed
FATA[0001] Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed
After scouring the internet I found that if I remove my /var/lib/docker directory and then reboot my instance it should work, and indeed it does resolve that error.
The issue is, once I reboot the docker service status shows docker as docker dead but subsys locked. Now I am able to run my dockerd command and it will run (although it will hang at the API call, but that is an issue for later)
Issue number 1:
- dockerd hangs on API Call (I will have an image of what happens below)
Issue number 2
- I am able to start docker after I get out of dockerd (Which was hanging at the API call), but I can NEVER stop docker. The moment I try to restart docker or stop docker it hangs and then the ec2 instance becomes corrupted and then recreates itself.
I need to be able to have functionality over the docker daemon, but the only way for me to do that is for me to delete the docker directory from within the container.... some help would be greatly appreciated.
Steps to reproduce:
- Create EC2 Instance that is running Amazon Linux with Docker
- Jump into running instance
- Stop docker
- Start daemon
- sudo dockerd
- You should see the error that I listed above
- Ctrl Z to get out of hanging docker daemon
- rm -rf /var/lib/docker
- sudo reboot
- Wait a few minutes for everything to reboot before jumping back in
- Run
sudo service docker status and you should see that docker is dead
- Run dockerd and see that it hangs once it reaches out to make API calls
- Repeat step 5
- Start docker
- sudo service docker start
- Stop or reset docker and this will break
This is the where it hangs after resolving the original error
sudo dockerd
INFO[0000] libcontainerd: new containerd process, pid: 3232
WARN[0000] containerd: low RLIMIT_NOFILE changing to max current=1024 max=4096
WARN[0001] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
WARN[0001] devmapper: Base device already exists and has filesystem xfs on it. User specified filesystem will be ignored.
INFO[0001] [graphdriver] using prior storage driver "devicemapper"
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support cgroup blkio weight
WARN[0001] Your kernel does not support cgroup blkio weight_device
WARN[0001] mountpoint for pids not found
INFO[0001] Loading containers: start.
INFO[0001] Firewalld running: false
INFO[0001] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[0001] Loading containers: done.
INFO[0001] Daemon has completed initialization
INFO[0001] Docker daemon commit=7392c3b/1.12.6 graphdriver=devicemapper version=1.12.6
INFO[0001] API listen on /var/run/docker.sock
Output of docker version:
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.3
Git commit: 7392c3b/1.12.6
Built: Tue Mar 7 20:34:04 2017
OS/Arch: linux/amd64
Hey everyone,
I am on a jump box where I ssh into my EC2 instance that is running Amazon Linux with docker.
What I am needing to do is stop the docker service, run the docker daemon and apply some changes to the docker daemon from within the docker container itself.
What is happening is that after I stop the docker service I run dockerd and get the following error:
After scouring the internet I found that if I remove my /var/lib/docker directory and then reboot my instance it should work, and indeed it does resolve that error.
The issue is, once I reboot the docker service status shows docker as
docker dead but subsys locked. Now I am able to run my dockerd command and it will run (although it will hang at the API call, but that is an issue for later)Issue number 1:
Issue number 2
I need to be able to have functionality over the docker daemon, but the only way for me to do that is for me to delete the docker directory from within the container.... some help would be greatly appreciated.
Steps to reproduce:
sudo service docker statusand you should see that docker is deadThis is the where it hangs after resolving the original error
Output of
docker version: