EmbeddedAnsibleWorker existing with "No such image: rabbitmq:3"

[psachin]

On gaprindashvili-1, when Embeded ansible role is enable I see below errors in evm.log. PR #16205 introduced DockerEmbeddedAnsible class to manages AWX containers using the docker-api gem which fails to start due to image unavailability

[----] E, [2018-02-16T14:34:38.053549 #1673:93be364] ERROR -- : [Docker::Error::NotFoundError]: No such image: rabbitmq:3
  Method:[block in method_missing]
[----] E, [2018-02-16T14:34:38.053760 #1673:93be364] ERROR -- : /usr/local/lib/ruby/gems/2.3.0/gems/docker-api-1.33.6/lib/docker/connection.rb:46:in `rescue in request'
/usr/local/lib/ruby/gems/2.3.0/gems/docker-api-1.33.6/lib/docker/connection.rb:38:in `request'
/usr/local/lib/ruby/gems/2.3.0/gems/docker-api-1.33.6/lib/docker/connection.rb:65:in `block (2 levels) in <class:Connection>'
/usr/local/lib/ruby/gems/2.3.0/gems/docker-api-1.33.6/lib/docker/image.rb:129:in `get'
/usr/local/lib/ruby/gems/2.3.0/gems/docker-api-1.33.6/lib/docker/image.rb:124:in `create'
/var/www/miq/vmdb/lib/embedded_ansible/docker_embedded_ansible.rb:142:in `run_container'
/var/www/miq/vmdb/lib/embedded_ansible/docker_embedded_ansible.rb:87:in `run_rabbitmq_container'
/var/www/miq/vmdb/lib/embedded_ansible/docker_embedded_ansible.rb:42:in `start'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:34:in `setup_ansible'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:12:in `do_before_work_loop'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:5:in `prepare'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:126:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:22:in `start_worker'
/var/www/miq/vmdb/app/models/embedded_ansible_worker.rb:20:in `block in start_monitor_thread'
[----] I, [2018-02-16T14:34:38.939790 #1902:8e1138]  INFO -- : MIQ(MiqPriorityWorker::Runner#message_sync_config) MIQ(MiqPriorityWorker::Runner) Synchronizing configuration...
[----] I, [2018-02-16T14:34:39.024870 #1902:8e1138]  INFO -- : MIQ(MiqPriorityWorker::Runner#sync_config) ID [223], PID [1902], GUID [b33f1233-5a62-4742-be89-5ac7b7666006], Zone [default], Active Roles [automate,database_operations,databa
se_owner,embedded_ansible,ems_inventory,ems_operations,event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Assigned Roles [automate,database_operations,database_owner,embedded_ansible,ems_inventory,ems_operations,
event,reporting,scheduler,smartstate,user_interface,web_services,websocket], Configuration:
[----] I, [2018-02-16T14:34:39.025242 #1902:8e1138]  INFO -- : 
---
:count: 2
:gc_interval: 900
:heartbeat_freq: 10
:heartbeat_timeout: 120
:memory_threshold: 629145600
:nice_delta: 1
:parent_time_threshold: 180
:poll: 1
:poll_escalate_max: 30
:poll_method: :normal
:restart_interval: 0
:starting_timeout: 600
:stopping_timeout: 600
:cpu_usage_threshold: 100
:dequeue_method: :drb
:queue_timeout: 600
[----] I, [2018-02-16T14:34:39.025291 #1902:8e1138]  INFO -- : ---
[----] I, [2018-02-16T14:34:39.025485 #1902:8e1138]  INFO -- : 
---
:guid: b33f1233-5a62-4742-be89-5ac7b7666006
[----] I, [2018-02-16T14:34:39.025564 #1902:8e1138]  INFO -- : MIQ(MiqPriorityWorker::Runner#message_sync_config) MIQ(MiqPriorityWorker::Runner) Synchronizing configuration complete...
[----] I, [2018-02-16T14:34:39.210283 #1673:93be364]  INFO -- : MIQ(EmbeddedAnsibleWorker#log_status) [Embedded Ansible Worker] Worker ID [257], PID [], GUID [26628f79-eccf-4a72-a1c1-09adfddf8064], Last Heartbeat [2018-02-16 19:34:14 UTC], Process Info: Memory Usage [], Memory Size [], Proportional Set Size: [], Unique Set Size: [], Memory % [], CPU Time [], CPU % [], Priority []
[----] E, [2018-02-16T14:34:39.210473 #1673:93be364] ERROR -- : MIQ(EmbeddedAnsibleWorker::Runner) ID [257] PID [1673] GUID [26628f79-eccf-4a72-a1c1-09adfddf8064] No such image: rabbitmq:3
 Worker exiting.

Evm status

[root@localhost vmdb]# rake evm:status --all
Checking EVM status...
 Zone    | Server | Status  | ID |  PID | SPID | URL                     | Started On           | Last Heartbeat       | MB Usage | Master? | Active Roles
---------+--------+---------+----+------+------+-------------------------+----------------------+----------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
 default | EVM    | started |  1 | 1673 | 1799 | druby://127.0.0.1:39586 | 2018-02-16T19:19:25Z | 2018-02-16T19:32:39Z |      203 | true    | automate:database_operations:database_owner:embedded_ansible:ems_inventory:ems_operations:event:reporting:scheduler:smartstate:user_interface:web_services:websocket

 Worker Type           | Status   |  ID |  PID | SPID | Server id | Queue Name / URL      | Started On           | Last Heartbeat       | MB Usage
-----------------------+----------+-----+------+------+-----------+-----------------------+----------------------+----------------------+----------
 EmbeddedAnsibleWorker | starting | 254 |      | 1862 |         1 |                       | 2018-02-16T19:32:39Z | 2018-02-16T19:32:39Z |
 MiqEventHandler       | started  | 219 | 1863 | 1875 |         1 | ems                   | 2018-02-16T19:19:27Z | 2018-02-16T19:32:40Z |      207
 MiqGenericWorker      | started  | 221 | 1884 | 1936 |         1 | generic               | 2018-02-16T19:19:27Z | 2018-02-16T19:32:39Z |      207
 MiqGenericWorker      | started  | 220 | 1872 | 1915 |         1 | generic               | 2018-02-16T19:19:27Z | 2018-02-16T19:32:37Z |      231
 MiqPriorityWorker     | started  | 222 | 1893 | 1941 |         1 | generic               | 2018-02-16T19:19:27Z | 2018-02-16T19:32:39Z |      205
 MiqPriorityWorker     | started  | 223 | 1902 | 1947 |         1 | generic               | 2018-02-16T19:19:27Z | 2018-02-16T19:32:40Z |      205
 MiqReportingWorker    | started  | 225 | 1927 | 1997 |         1 | reporting             | 2018-02-16T19:19:28Z | 2018-02-16T19:32:39Z |      202
 MiqReportingWorker    | started  | 224 | 1916 | 1966 |         1 | reporting             | 2018-02-16T19:19:28Z | 2018-02-16T19:32:38Z |      202
 MiqScheduleWorker     | started  | 226 | 1937 | 1967 |         1 |                       | 2018-02-16T19:19:28Z | 2018-02-16T19:32:33Z |      202
 MiqUiWorker           | started  | 227 | 1954 |      |         1 | http://127.0.0.1:3000 | 2018-02-16T19:19:28Z | 2018-02-16T19:32:32Z |      203
 MiqWebServiceWorker   | started  | 228 | 1968 |      |         1 | http://127.0.0.1:4000 | 2018-02-16T19:19:29Z | 2018-02-16T19:32:32Z |      203
 MiqWebsocketWorker    | started  | 229 | 1988 |      |         1 | http://127.0.0.1:5000 | 2018-02-16T19:19:29Z | 2018-02-16T19:32:32Z |      201

[psachin]

@carbonin

Can you please look into this issue?

[carbonin]

I can see this image on docker hub here https://hub.docker.com/r/library/rabbitmq/tags/3/

Can you docker pull rabbitmq:3 from that machine and post the output?
@psachin

[psachin]

@carbonin

I can see that the image is present

[root@localhost ~]# docker pull rabbitmq:3
Trying to pull repository docker.io/library/rabbitmq ... 
3: Pulling from docker.io/library/rabbitmq

Digest: sha256:c05acf3483ed8185529652cb9b344009651d4d0fe11be57142c0de260208285c

So I restarted evm and EmbeddedAnsible started. But I think this should be handled by UI(when user tries to disable/enable EmbeddedAnsible role)

I think we can close this issue. If anyone encounter similar issue, he should follow below steps

1. Verify EmbeddedAnsibleWorker status using cd vmdb; rake evm:status. It should be in starting or abort state
2. Verify is rabbitmq docker image is downloaded,

Example:

# docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
docker.io/ansible/awx_task   latest              8f67c38eafac        26 hours ago        1.098 GB
docker.io/ansible/awx_web    latest              44146e6af253        26 hours ago        1.071 GB
docker.io/rabbitmq           3                   02c3edbcf3d1        34 hours ago        126.9 MB
docker.io/memcached          alpine              1b97d9b2ea1a        4 days ago          7.87 MB

and the container is running

# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                NAMES
9f2294b2b050        ansible/awx_task:latest   "/tini -- /bin/sh -c "   29 minutes ago      Up 29 minutes       8052/tcp                             awx_task
c46320f09cc6        ansible/awx_web:latest    "/tini -- /bin/sh -c "   29 minutes ago      Up 29 minutes       0.0.0.0:54321->8052/tcp              awx_web
ad3084c91552        memcached:alpine          "docker-entrypoint.sh"   30 minutes ago      Up 30 minutes       11211/tcp                            memcached
a944e212c4b9        rabbitmq:3                "docker-entrypoint.sh"   30 minutes ago      Up 30 minutes       4369/tcp, 5671-5672/tcp, 25672/tcp   rabbitmq

3. If image is not downloaded, download using,

docker pull rabbitmq:3

4. Restart evm service,

systemctl restart evmserverd
# or
cd vmdb; rake evm:restart