When I deployed the latest version (225.0.0) of cf-rabbitmq, post-deploy scripts failed (rabbitmq-server)

[eujungkim]

When I deployed v225.0.0 to bosh-lite, I got these errors.

1. bosh task error
   Error: Action Failed get_task: Task d8ab5d09-9d62-46f7-6bf6-68cf09a70401 result: 1 of 4 post-deploy scripts failed. Failed Jobs: rabbitmq-server. Successful Jobs: rabbitmq-statsdb-reset-cron-test, permissions-test, syslog-configuration-test.

2. error log in rabbitmq-server vm (when trying "rabbitmqcrl start_app")
   Error description:
   {could_not_start,rabbit,
   {error,
   {{shutdown,
   {failed_to_start_child,rabbit_memory_monitor,
   {badarg,
   [{lists,member,[disk,{error,bad_module}],[]},
   {rabbit_memory_monitor,init,1,
   [{file,"src/rabbit_memory_monitor.erl"},
   {line,121}]},
   {gen_server2,init_it,6,
   [{file,"src/gen_server2.erl"},{line,554}]},
   {proc_lib,init_p_do_apply,3,
   [{file,"proc_lib.erl"},{line,247}]}]}}},
   {child,undefined,rabbit_memory_monitor_sup,
   {rabbit_restartable_sup,start_link,
   [rabbit_memory_monitor_sup,
   {rabbit_memory_monitor,start_link,[]},
   false]},
   transient,infinity,supervisor,
   [rabbit_restartable_sup]}}}}

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[eujungkim]

Can you check this too? It seems similar issue.
docker-library/rabbitmq#147

[albertoleal]

Hi @eujungkim, thanks for reporting that.

If you adjust the disk alarm threshold -- rabbitmq-server.disk_alarm_threshold -- to be 50000000, are you able to deploy after that? [1]

[1] https://github.com/pivotal-cf/cf-rabbitmq-release/blob/master/jobs/rabbitmq-server/spec#L57

[eujungkim]

Thanks for your fast reply.
I adjusted the disk alarm threshold to be 50000000 and tried to deploy.
The deployment task still logged error like below, but there is no post-deploy error log in the rabbitmq-server vms.
I checked /var/vcap/sys/log/rabbitmq-server/post-deploy.stderr.log file of each rabbitmq-server vm and the file size was zero. And "rabbitmq-ctl status" in each vm displayed normal status, too.
So I ignored the error and proceeded and succeeded to create rabbitmq-service-broker and service anyway. And binding service to app was succeeded.

---

02:13:28 | Error: Action Failed get_task: Task 4e473323-a1b9-4734-50c3-b656aa4423f8 result: 1 of 4 post-deploy scripts failed. Failed Jobs: rabbitmq-server. Successful Jobs: rabbitmq-statsdb-reset-cron-test, permissions-test, syslog-configuration-test.

Started Fri Apr 21 02:10:19 UTC 2017
Finished Fri Apr 21 02:13:28 UTC 2017
Duration 00:03:09

Task 134 error

Capturing task '134' output:
Expected task '134' to succeed but was state is 'error'

[albertoleal]

Regarding post-deploy error: we've changed the way we're triggering the disk alarm [1]. The default behaviour now is relative to 40% of the memory.

Then our users have two options:

1. Increase persistent disk
2. Adjust disk alarm threshold to use free disk space (that's what I suggested you do do..)

But what you've reported is quite interesting! The job failed but apparently you could create/bind to a service. Would you mind sharing the deployment manifest you're using to deploy cf-rabbitmq?

[1] https://www.rabbitmq.com/disk-alarms.html

[eujungkim]

I attached my deployment manifest files. (I added ".txt" to filename for upload.)
Please let me know, if there are any mistakes.

cf-rabbitmq.yml.txt
lite-cloud-config.yml.txt

[albertoleal]

Hey @eujungkim -- I managed to deploy a slightly different version of the manifest you've provided me. I changed some urls and credentials, because I have diff ones, and the task succeeded :)

Task 2649 done

Started         2017-04-22 16:02:45 UTC
Finished        2017-04-22 16:11:43 UTC
Duration        00:08:58

Deployed `cf-rabbitmq' to `garden-bosh'

[eujungkim]

Thanks for your test.
Because the service works normally, I'll keep this deployment.
If the problem recurs next time, I'll let you know.