Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error response from daemon: can not get logs from container which is dead or marked for removal #91

Closed
Deepak-Enc opened this issue Jun 13, 2020 · 13 comments
Assignees

Comments

@Deepak-Enc
Copy link

@Deepak-Enc Deepak-Enc commented Jun 13, 2020

Docker image is running fine on my system but when run it on edge orchestration it shows me the errors:
Screenshot from 2020-06-13 10-40-25
Running perfect on my system:
Screenshot from 2020-06-13 10-42-01

Here is the command which I am running:
curl -X POST "192.168.43.11:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ "ServiceName": "age_detection", "ServiceInfo": [{ "ExecutionType": "container", "ExecCmd": [ "docker", "run", "-it","--rm","--device=/dev/video0","-e","DISPLAY=$DISPLAY ","-v","$XSOCK:$XSOCK","-v","$XAUTH:$XAUTH","-e","XAUTHORITY=$XAUTH","-e"," QT_X11_NO_MITSHM=1","animeshj123/age_det_image:02"]}], "StatusCallbackURI": "http://192.168.43.11:8888/api/v1/services/notification\"}"

@tdrozdovsky
Copy link
Contributor

@tdrozdovsky tdrozdovsky commented Jun 14, 2020

@Deepak-Enc I successfully reproduced your case with an error and also successfully launched the container using edge-orchestration.
Below is the command that I did to launch successfully

virtual-pc@virtualpc-VirtualBox:~$ curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"age_detection\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-it\",\"--rm\",\"--device=/dev/video0\", \"--net=host\",\"--ipc=host\", \"-e\",\"DISPLAY=$DISPLAY\",\"-v\",\"/tmp/.X11-unix:/tmp/.X11-unix\",\"-e\",\"QT_X11_NO_MITSHM=1\",\"animeshj123/age_det_image:02\"]}], \"StatusCallbackURI\": \"http://127.0.0.1:8888/api/v1/services/notification\"}"
{"Message":"ERROR_NONE","RemoteTargetInfo":{"ExecutionType":"container","Target":"10.0.2.15"},"ServiceName":"age_detection"}
  1. My X server settings are slightly different, but you can use your own if it works for you. The main thing to pay attention to quotation marks. In your example, they are not correct (I think this is a typo).
  2. The error that you encountered was a consequence of running the container twice. The program that works inside gives an error, because it can’t access the webcam.

    Before starting, make sure that such a container is not started before using the command
virtual-pc@virtualpc-VirtualBox:~$ docker ps -a
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
0d73793665d9        animeshj123/age_det_image:02   "python detect_age_c…"   16 minutes ago      Up 16 minutes                           crazy_visvesvaraya
f1a1dacb70fd        edge-orchestration:baobab      "sh run.sh"              16 minutes ago      Up 16 minutes                           edge-orchestration
virtual-pc@virtualpc-VirtualBox:~$ 

Stop the running container, and after that you can start using edge-orchestration

Please give feedback what is your result? I hope everything worked out for you. If you have questions, we will be happy to answer!

@Deepak-Enc
Copy link
Author

@Deepak-Enc Deepak-Enc commented Jun 15, 2020

Receiving the same error.
Screenshot from 2020-06-15 15-09-29

Screenshot from 2020-06-15 15-10-09

Screenshot from 2020-06-15 15-10-37

@tdrozdovsky
Copy link
Contributor

@tdrozdovsky tdrozdovsky commented Jun 15, 2020

You need to make sure that the program in the animeshj123/age_det_image:02 container starts correctly, as the container execution status : 1 line shows that an error has occurred in the program itself.
изображение

I suggest doing it. After that, I see that everything works correctly:

virtual-pc@virtualpc-VirtualBox:~$ sudo docker run --rm -ti --device=/dev/video0 --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" animeshj123/age_det_image:02
[sudo] password for virtual-pc: 
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] starting video stream...
^CTraceback (most recent call last):
  File "detect_age_cam.py", line 121, in <module>
    results = detect_and_predict_age(frame, faceNet, ageNet,
  File "detect_age_cam.py", line 29, in detect_and_predict_age
    detections = faceNet.forward()
KeyboardInterrupt
virtual-pc@virtualpc-VirtualBox:~$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS               NAMES
f1a1dacb70fd        edge-orchestration:baobab   "sh run.sh"         17 hours ago        Up 17 hours                             edge-orchestration
virtual-pc@virtualpc-VirtualBox:~$ curl -X POST "127.0.0.1:56001/api/v1/orchestration/services" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"ServiceName\": \"age_detection\", \"ServiceInfo\": [{ \"ExecutionType\": \"container\", \"ExecCmd\": [ \"docker\", \"run\", \"-it\",\"--rm\",\"--device=/dev/video0\", \"--net=host\",\"--ipc=host\", \"-e\",\"DISPLAY=$DISPLAY\",\"-v\",\"/tmp/.X11-unix:/tmp/.X11-unix\",\"-e\",\"QT_X11_NO_MITSHM=1\",\"animeshj123/age_det_image:02\"]}], \"StatusCallbackURI\": \"http://127.0.0.1:8888/api/v1/services/notification\"}"
{"Message":"ERROR_NONE","RemoteTargetInfo":{"ExecutionType":"container","Target":"10.0.2.15"},"ServiceName":"age_detection"}virtual-pc@virtualpc-VirtualBox:~$ 
virtual-pc@virtualpc-VirtualBox:~$ 

I also see such an error.
изображение
it can be fixed if restart docker (service docker restart)

@Deepak-Enc
Copy link
Author

@Deepak-Enc Deepak-Enc commented Jun 15, 2020

@tdrozdovsky
Copy link
Contributor

@tdrozdovsky tdrozdovsky commented Jun 15, 2020

Below is an example of how it may not work correctly

virtual-pc@virtualpc-VirtualBox:~$ docker run --rm -ti --device=/dev/video0 --net=host --ipc=host -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --env="QT_X11_NO_MITSHM=1" animeshj123/age_det_image:02
[sudo] password for virtual-pc: 
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] starting video stream...
[ WARN:0] global /io/opencv/modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video0): can't open camera by index
Traceback (most recent call last):
  File "detect_age_cam.py", line 117, in <module>
    frame = imutils.resize(frame, width=400)
  File "/usr/local/lib/python3.8/site-packages/imutils/convenience.py", line 69, in resize
    (h, w) = image.shape[:2]
AttributeError: 'NoneType' object has no attribute 'shape'

now look at the error code

virtual-pc@virtualpc-VirtualBox:~$ echo $?
1

the error is the same as the edge-orchestrator returns

I think it is necessary to look for reasons why the program does not work correctly (for example, there is no access to /dev/video0 or others)

@Deepak-Enc
Copy link
Author

@Deepak-Enc Deepak-Enc commented Jun 16, 2020

Kindly clear my doubt. It's working fine on my host machine that means there is no problem in docker container or docker file. There may be an issue with edge-orchestration because I am unable to detect in container or docker file.

@MoonkiHong
Copy link
Contributor

@MoonkiHong MoonkiHong commented Jun 16, 2020

@Deepak-Enc Please describe much more in details about what "an issue with edge-orchestration" is. From my understanding, your targeting container successfully runs on your local host through a docker container / docker file, but it fails when you try to run it on edge-orchestration. We have guided you several recommendation but still it is unclear to get through this issue. Are we correct?
@suresh-lc What is your opinion? It does not seem to be just an configuration issue.

@MoonkiHong MoonkiHong added the bug label Jun 16, 2020
@Deepak-Enc
Copy link
Author

@Deepak-Enc Deepak-Enc commented Jun 16, 2020

2020/06/16 04:27:56 containerexecutor.go:84: [containerexecutor] create container : c22c72a303
2020/06/16 04:28:01 containerexecutor.go:102: [containerexecutor] container execution status : 1
2020/06/16 04:28:01 containerexecutor.go:111: [containerexecutor] Error response from daemon: can not get logs from container which is dead or marked for removal
2020/06/16 04:28:01 orchestration_api.go:340: [orchestrationapi] service status changed [appNames:age_detection][status:]
2020/06/16 04:28:01 containerexecutor.go:122: [containerexecutor] Error response from daemon: removal of container c22c72a303bff50d18b4af5d5b7d644256cdd8fb2c7d2b08ccfa2ea225b3aa7f is already in progress

I am receiving these errors while running age-detection containers(real-time and Video) on edge-orchestration. It captures the first frame of application but then automatically container marked as a dead container.

@suresh-lc
Copy link
Contributor

@suresh-lc suresh-lc commented Jun 16, 2020

@Deepak-Enc : We could see from the logs the container is getting marked for removal after reading the first frame due to some error. But exactly why we are also trying to find. Can you share the complete error log file. And also if you could share the video of the steps you did, it would be great help.

@tdrozdovsky
Copy link
Contributor

@tdrozdovsky tdrozdovsky commented Jun 16, 2020

Below is a video of how a container works under edge-orchestration

Ubuntu_v16_04 [Running] - Oracle VM VirtualBox 6_16_2020 9_30_15 AM.zip

The log that you sent shows that the application in the container did not finish correctly. More data needed

@suresh-lc
Copy link
Contributor

@suresh-lc suresh-lc commented Jun 16, 2020

@Deepak-Enc : Can you take Baobab branch and try the scenario.
Looks container whitelisting is being not done for service started from edge orchestration.
err := verifier.GetInstance().ContainerIsInWhiteList(s.ParamStr[paramLen-1]).

Hence can you once try with Baobab branch and let us know the result.

@Deepak-Enc
Copy link
Author

@Deepak-Enc Deepak-Enc commented Jun 16, 2020

It's working on baobab. Thank You.

@MoonkiHong
Copy link
Contributor

@MoonkiHong MoonkiHong commented Jun 16, 2020

Issue Resolved. @suresh-lc @tdrozdovsky @Deepak-Enc Thank you for all your interests and passion!

@MoonkiHong MoonkiHong closed this Jun 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants