duplicated service retirement request

[easyljy]

ManageIQ version: hammer-2

We saw duplicated service retirement requests, only one of them gets success, and other failed due to the service already in retiring status.

Root cause: we have 3 zones in one region, the retirement_check is raised based on zones. So it gets 3 times to check retirement on same service.

# /var/www/miq/vmdb/app/models/miq_schedule_worker/jobs.rb

31  def retirement_check
32    queue_work_on_each_zone(:class_name => 'RetirementManager', :method_name => 'check')
33  end

See some logs and screenshot

[----] I, [2019-03-13T03:39:59.399752 #26862:798f60]  INFO -- : Q-task_id([r10000000000120_service_retire_request_10000000000120]) MIQ(ServiceRetireRequest#create_request_tasks) Creating request task instances for: <Service Retire for: OpenStack-10000000000118 - >...
[----] I, [2019-03-13T03:40:01.557824 #6614:f7af58]  INFO -- : Q-task_id([r10000000000121_service_retire_request_10000000000121]) MIQ(ServiceRetireRequest#create_request_tasks) Creating request task instances for: <Service Retire for: OpenStack-10000000000118 - >...
[----] I, [2019-03-13T03:40:01.767459 #6606:f7af58]  INFO -- : Q-task_id([r10000000000122_service_retire_request_10000000000122]) MIQ(ServiceRetireRequest#create_request_tasks) Creating request task instances for: <Service Retire for: OpenStack-10000000000118 - >...

[Fryguy]

@tinaafitz Please review.

[tinaafitz]

@lfu Let's discuss this issue.