based on Change-Id: Ie759857fb94db9aa94aaeaeda2c6ab5bb159cc9e
All the work done for fuel-library packaging

Should be overriden by the change above after we switch
CI to package-based

implements blueprint: package-fuel-components

Change-Id: I48ed37a009b42f0a9a21cc869a869edb505b39c3

1) Package fuel library into three different
packages:
RPM: fuel-library6.1
ALL: fuel-ha-utils, fuel-misc

2) Install packages onto slave nodes
implements blueprint: package-fuel-components

Change-Id: Ie759857fb94db9aa94aaeaeda2c6ab5bb159cc9e

This commit makes post-start notify action to check
hostlist of nodes that should be joined to the cluster
to contain not only nodes that will be started but
also ones that are already started. This fixes
the case when Pacemaker sends notifies only for
the latest event and thus the node which is not
included into the start list will not join the
cluster. Also it checks whether the node is
already clustered and skips the join if it
is not needed.

Change-Id: Ibe8ecdcfe42c14228350b1eb3c9d08b1a64e117d
Closes-bug: #1455761

There is a mistake in OCF logic which tries
to start rabbitmq app without running beam
after Mnesia reset getting into the loop
which constantly fails until it times out

Change-Id: Id096961e206a083b51978fc5034f99d04715d7ea
Related-bug: #1436812

W/o this patch, the code in OCF script from
deployment/ dir will never get to the fuel-library
packages, which are building from files/ and /debian
dirs only.

The solution is:
1) sync the code diverged to the files/ and debian/
2) either to remove the source OCF file or to
update the way files being linked.

This patch fixes only the step 1 as there is not yet
decided how to deal with the step 2.

Related-bug: #1457441
Related-bug: #184966

Change-Id: Ied86640e8e853de99bcd26f1ae726fc8272b6db7
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, when rabbit app cannot start due to
corrupted mnesia state, the mnesia would be cleaned
not completely. This may prevent the rabbit app from
start and take the node out of the cluster permanently.

The solution is to remove all rabbit node related mnesia
files.

Closes-bug: #1457766

Change-Id: I680efbf573c22aa9a13d8429d985b5a57235b2bf
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

* When the rabbit node went down, its status remains 'running'
  in mnesia db for a while, so few retries (50 sec of total) are
  required in order to kick and forget this node from the cluster.
  This also requires +50 sec for actions stop & demote timeout.
* The rabbit master score in the CIB is retained after the current
  master moved manually. This is wrong and the score must be reset
  ASAP for post-demote and post-stop as well.
* The demoted node must be kicked from cluster by other nodes
  on post-demote processing.
* Post-demote should stop the rabbit app at the node being demoted as
  this node should be kicked from the cluster by other nodes.
  Instead, it stops the app at the *other* nodes and brings full
  cluster downtime.
* The check to join should be only done at the post-start and not at
  the post-promote, otherwise the node being promoted may think it
  is clustered with some node while the join check reports it as
  already clustered with another one.
  (the regression was caused by https://review.openstack.org/184671)
* Change `hostname` call to `crm_node -n` via $THIS_PCMK_NODE
  everywhere to ensure we are using correct pacemaker node name
* Handle empty values for OCF_RESKEY_CRM_meta_notify_* by reporting
  the resource as not running. This will rerun resource and restore
  its state, eventually.

Closes-bug: #1436812
Closes-bug: #1455761

Change-Id: Ib01c1731b4f06e6b643a4bca845828f7db507ad3
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, the failover time was longer than expected
as rabbit nodes was able to query corosync nodes left the
cluster and also try to join them by rabbit cluster ending
up being reset and rejoin alive nodes later.
1) Add functions:
  a) to get all alive nodes in the partition
  b) to get all nodes
This fixes get_monitor behaviour so that it ignores
attributes for dead nodes as crm_node behaviour
changed with upgrade of pacemaker. So rabbit nodes will
never try to join the dead ones.

2) Fix bash scopes for local variables
Minor change removing unexcpeted behavior when local variable
impacts global scope.

Related-bug: #1436812

Change-Id: I89b716b4cd007572bb6832365d4424669921f057
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, rabbitmqctl sometimes may hang failing
many commands. This is a problem as it brings the rabbit node
to unresponsive and broken state. This also may affect
entire cluster operations, for example, when the failed command is
the forget_cluster_node.

The solution is to check for the cases when the command rabbitmqctl
list_channels timed out and killed or termintated with exit codes
137 or 124 and return generic error.
There is also related confusing error message "get_status() returns generic
error" may be logged when the rabbit node is running out of the cluster
and fixed as well.

Closes-bug: #1459173

Change-Id: Ia52fc5f2ab7adb36252a7194f9209ab87ce487de
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

This commit checks whether there is a running
cluster of rabbitmq and if rabbitmq app is running
on the node and exits with non-zero code if
current node is not running rabbitmq, but should
do so

Change-Id: I2098405b39ade7325b94781aeb997de0937bdf4c
Closes-bug: #1458828

W/o this fix, the situation is possible when a
rabbit node would stuck in a start/stop loop failing
to join the cluster with an error:
"no_running_cluster_nodes, You cannot leave a cluster
if no online nodes are present."

This is an issue because the rabbit node should always
be able to join the cluster, if it was ordered to start
by pacemaker RA.

The solution is to force the mnesia reset, if the
rabbit node cannot join the cluster on post-start
notify. Note, that for the master starting, the node
wouldn't be reset. So, the mnesia will be kept intact
at least on the resource master.

Partial-bug: #1461509

Change-Id: I69bc13266a1dc784681b2677ae5616bfc28cf54f
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix the dead end situation is possible
when the rabbit node have no free memory resources left
and the cluster blocks all publishing, by design.
But the app thinks "let's wait for the publish block have
lifted" and cannot recover.

The workaround is to monitor results
of crucial rabbitmqctl commands and restart the rabbit node,
if queues/channels/alarms cannot be listed or if there are
memory alarms found.
This is the similar logic as we have for the cases when
rabbitmqctl list_channels hangs. But the channels check is also
fixed to verify if the exit code>0 when the rabbit app is
running.

Additional checks added to the monitor also require extending
the timeout window for the monitor action from 60 to 180 seconds.

Besides that, this patch makes the monitor action to gather the
rabbit status and runtime stats, like consumed memory by all
queues of total Mem+Swap, total messages in all queues and
average queue consumer utilization. This info should help to
troubleshoot failures better.

DocImpact: ops guide. If any rabbitmq node exceeded its memory
threshold the publish became blocked cluster-wide, by design.
For such cases, this rabbit node would be recovered from the
raised memory alert and immediately stopped to be restarted
later by the pacemaker. Otherwise, this blocked publishing state
might never have been lifted, if the pressure persists from the
OpenStack apps side.

Closes-bug: #1463433

Change-Id: I91dec2d30d77b166ff9fe88109f3acdd19ce9ff9
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, the list of file names not
accessible by rabbitmq user will be treated
as multiple arguments to the if command causing
it to throw the "too many arguments" error and
the chown command to be skipped.

This is the problem as it might prevent the rabbitmq
server from starting because of a bad files ownership.

The solution is to pass the list of files as a single
argument "${foo}".

Closes-bug: #1472175

Change-Id: I1d00ec3f31cd0f023bd58a4e11e5b31659977229
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix the situation is possible when
rabbit OCF returns OCF_NOT_RUNNING in the hope of
future restart of the resource by pacemaker.

But in fact, pacemaker will not trigger restart action
if monitor returns "not running". This is an issue
as we want resource restarted.

The solution is to return OCF_ERR_GENERIC instead of
OCF_NOT_RUNNING when we expect the resource to be restarted
(which is action stop plus action start).

Closes-bug: #1472230

Change-Id: I10c6e43d92cb23596636d86932674b36864d1595
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

This changes leverages the rabbitmq management plugin to dump
exchanges, queues, bindings, users, virtual hosts, permissions and
parameters from the running system. Specifically this change adds the
following:

* The dumping rabbitMQ definitions (users/vhosts/exchanges/etc) during
  the end of the deployment
* The possibility to restore definitions to the rabbitmq-server ocf
  script during rabbitMQ startup.
* Enabled rabbitmq admin plugin, but restricts it to localhost traffic.
  This reverts Ic01c26200f6019a8112b1c5fb04a282e64b3b3e6 but adds
  firewall rules to mitigate the issue.

DocImpact: The dump_rabbit_definitions task can be used to backup the
rabbitmq definitions and if custom definitions (users/vhosts/etc) are
created it must be run or the changes may be lost during the rabbitmq
failover via pacemaker.

Change-Id: I715f7c2ae527f7e105b9f6b7d82c443e8accf178
Closes-bug: #1383258
Related-bug: #1450443
Co-Authored-By: Alex Schultz <aschultz@mirantis.com>

Previously we were sending the json backup data on the command line
which fails when the dataset is large. This change updates the command
line options for curl to pass the filename directly and let it handle
the reading of the data.

Change-Id: I37f298279beca06df41fb08e1745602976c6a776
Closes-Bug: 1383258

It's really hard to debug, when get_status() returns $OCF_NOT_RUNNING
only and looses exit code and error output.

Added more logs to avoid of this situation.

Related-Bug: #1488999

Change-Id: Id0999235d7be688f55799e2952fe22e97b678ce7

W/o this patch, the race condition is possible
when there is no running rabbit nodes/resource
master. The rabbit nodes will start/stop in an
endless loop as a result introducing full downtime
for AMQP cluster and cloud control plane.

The solution is:
* On post-start/post-promote notify, do nothing, if
  either of the following is a true:
  - there is no rabbit resources running or no master
  - the list of rabbit resources being started/promoted
    reported empty
* For such cases, do not report resource failure and delegate
  recovery, if needed, to the "running out of the cluster"
  monitor's logic.
* Additionally, report about a last man standing when
  there is no running rabbit resources around.

Closes-bug: #1491306

Change-Id: If1c62fac26b63410636413c49fce55c35e53dc5f
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

The change makes OCF script ignore small number of timeouts of rabbitmqctl
for 'heavy' operations: list_channels, get_alarms and list_queues.
Number of tolerated timeouts in a row is configured through a new variable
'max_rabbitmqctl_timeouts'. By default it is set to 1, i.e. rabbitmqctl
timeouts are not tolerated at all.

Bug #1487517 is fixed by extracting declaration of local variables
'rc_alarms' and 'rc_queues' from assignment operations.


Text for Operations Guide:

If on node where RabbitMQ is deployed
other processes consume significant part of CPU, RabbitMQ starts
responding slow to queries by 'rabbitmqctl' utility. The utility is
used by RabbitMQ's OCF script to monitor state of the RabbitMQ.
When utility fails to return in pre-defined timeout, OCF script
considers RabbitMQ to be down and restarts it, which might lead to
a limited (several minutes) OpenStack downtime. Such restarts
are undesirable as they cause downtime without benefit. To
mitigate the issue, the OCF script might be told to tolerate
certain amount of rabbitmqctl timeouts in a row using the following
command:
  crm_resource --resource p_rabbitmq-server --set-parameter \
      max_rabbitmqctl_timeouts --parameter-value N

Here N should be replaced with the number of timeouts. For instance,
if it is set to 3, the OCF script will tolerate two rabbitmqctl
timeouts in a row, but fail if the third one occurs.

By default the parameter is set to 1, i.e. rabbitmqctl timeout is not
tolerated at all. The downside of increasing the parameter is that
if a real issue occurs which causes rabbitmqctl timeout, OCF script
will detect that only after N monitor runs and so the restart, which
might fix the issue, will be delayed.

To understand that RabbitMQ's restart was caused by rabbitmqctl timeout
you should examine lrmd.log of the corresponding controller on Fuel
master node in /var/log/docker-logs/remote/ directory. Here lines like
"the invoked command exited 137: /usr/sbin/rabbitmqctl list_channels ..."

indicate rabbitmqctl timeout. The next line will explain if it
caused restart or not. For example:
"rabbitmqctl timed out 2 of max. 3 time(s) in a row. Doing nothing for now."

DocImpact: user-guide, operations-guide

Closes-Bug: #1479815
Closes-Bug: #1487517
Change-Id: I9dec06fc08dbeefbc67249b9e9633c8aab5e09ca

Change get_status to return NOT_RUNNING when
beam is not_running. Otherwise, pacemaker
will get stuck during rabbitmq failover and
will not attempt to restart the failed resource

Change-Id: I926a3eafa9968abdf07baa5f2d5c22480300fb30
Closes-bug: #1484280

On notify, if we detect that we are a part of a cluster we still
need to start the RabbitMQ application, because it is always
down after action_start finishes.

Closes-Bug: #1496386
Change-Id: I307452b687a6100cc4489c8decebbc3dccdbc432

When the data returned from 'rabbitmqctl list_queues' grows a lot
and awk sums up all the rows especially for memory calculation it
returns the sum in scientific notation (example from bug
was .15997e+09), later when we want to calculate the memory in
MB instead of bytes, the bash division does not like this string.

We can just avoid the situation by doing the division into MB
in awk itself. Since we don't need the memory in bytes anyway.

Closes-Bug: #1503331
Change-Id: I38d25406b84d0f70ed62101d5fb5ba108bcab8bd

Added new OCF key stop_time (corresponding to start_time)
Added wait_sync function which tries until start_time/2
for queues on stopped/demoted node to reach synced state.

Added optional [-t timeout] to su_rabbit_cmd function to
provide arbitrary timeout

Change-Id: Iae2211b3d477a9603a58d5eacb12e0fba924861a
Closes-Bug: #1464637

Sync upstream changes back to Fuel downstream
Source https://github.com/rabbitmq/rabbitmq-server
version stable/fedfefebaa39a0aeb41cf9328ba44c3a458e4614

Related blueprint upstream-rabbit-ocf
Closes-bug: #1473015

Change-Id: Ie19c2f071c53b873a359c6c5134e9498c6391e66
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

... in the source distribution anymore

And fix local bashisms as a little bonus
Upstream patch rabbitmq/rabbitmq-server#374

Related-bug: #1464637

Change-Id: I13189de9f8abce23673c031d11132e495e1972e3
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

* Fix return code of the get_all_pacemaker_nodes() and
  get_alive_pacemaker_nodes_but() to be
  not provided as ignored anyway.
* Fix return code expectation of the fetched count attribute
  in the check_timeouts().
Upstream patch rabbitmq/rabbitmq-server#374

Closes-bug: #1506440

Change-Id: I44a6cff2ccba1ba53a18da90c9d74cbb6084ca0c
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

Update happens even during no-op commands like 'meta-data' or 'usage'.
During this update there is a short window for a race condition: a shell
redirection truncates the cookie file, and echo writes data there only
after a brief period of time. So erlang may read data from this empty
file and die with error "Too short cookie string".

Change-Id: I4c3201617669f3872145048b77337632cb93558c
Closes-Bug: #1512754

Looks like copy-paste has gone wrong.

Update happens even during no-op commands like 'meta-data' or 'usage'.
During this update there is a short window for a race condition: a shell
redirection truncates the cookie file, and echo writes data there only
after a brief period of time. So erlang may read data from this empty
file and die with the error "Too short cookie string".

…dempotent-cookie

Don't update cookie on every run of HA OCF script

RabbitMQ itself was already listening on the correct IP
for controllers, but epmd and management plugin listened
everywhere (although management was covered by firewall
rules).

This covers all RabbitMQ server connection binding so that
all connections are done on the same IP address (with the
unfortunate side effect of blocking localhost connections).

Removed unused parameter rabbitmq_host from
nailgun::rabbitmq.

Change-Id: I9bfb8bc85fcd6d4711c4ca9d79745ad2ce7e673a
Closes-Bug: #1501731

Working with RMQ definitions via management plugin
requires knowing the IP address where it listens.

host_ip parameter will default to 127.0.0.1, but is
configurable.

Add two flags:
 * enable_rpc_ha which enables queue mirroring for RPC queues
 * enable_notifications_ha which enables queue mirroring for
   Ceilometer queues

Since the feature is experimental, both flags are set to true by
default to preserve current behaviour.

The change is implemented in several steps:
 * the upstream script changed so that it allows to extend the
   list of parameters and uses a policy file to define RabbitMQ
   policies.
 * we add our own version of OCF script which wraps around the
   upstream one. It defines a new enable_rpc_ha and
   enable_notifications_ha parameter and passes their value to the
   upstream script.
 * we add our policy file, where we use the introduced parameters
   to decide which policies we should set.

So we will have two OCF scripts for RabbitMQ in our deployment:
 * rabbitmq-server-upstream - the upstream version
 * rabbitmq-server - our extention, which will be used in the
   environment

The upstream version of the script is pushed to the upstream
along with empty policy file, so that other users can define their
own policies or extend the script if needed. Here are the
corresponding pull requests:
  rabbitmq/rabbitmq-server#480
  rabbitmq/rabbitmq-server#482
(both are already merged)

Text for Operations Guide

It is possible to significantly reduce load which OpenStack puts on
RabbitMQ by disabling queue mirroring. This could be done separately
for RPC queues and Ceilometer ones. To disable mirroring for RPC
queues, execute the following command on one of the controllers:

    crm_resource --resource p_rabbitmq-server --set-parameter \
        enable_rpc_ha --parameter-value false

To disable mirroring for Ceilometer queues, execute the following
command on one of the controllers:

    crm_resource --resource p_rabbitmq-server --set-parameter \
        enable_notifications_ha --parameter-value false

In order for any of the changes to take effect, RabbitMQ service
should be restarted. To do that, first execute

    pcs resource disable master_p_rabbitmq-server

Then monitor RabbitMQ state using command

    pcs resource

until it shows that all RabbitMQ nodes are stopped. Once they are,
execute the following command to start RabbitMQ:

    pcs resource enable master_p_rabbitmq-server

Beware: during restart all messages accumulated in RabbitMQ will be
lost. Also, OpenStack will stop functioning until RabbitMQ is up
again, so plan accordingly.

Note that it is not yet well tested how this configuration affects
failover when some cluster nodes go down. Hence it is experimental,
use at your own risk!

DocImpact:  ops-guide

Implements: blueprint rabbitmq-disable-mirroring-for-rpc
Change-Id: I80ae231ca64e2a903b0968d36ba0e85ca9cc9891

This change fixes the copy-paste gone wrong and pulls in the rabbitmq
upstream commit of c85fdd0f5c54f312fc2147dad2b956961aae3f12.

Closes-Bug: #1526062
Change-Id: I49e45cd893af8c65ed5ddd3efb834e38737a69a2

* Fix the get_status() unexpectedly reports generic error
  instead of "not running"
* Add proc_stop and proc_kill functions
  (TODO these shall go as external common ocf heplers, eventually)
* Rework stop_server_process()
  - make it to return SUCCESS/ERROR as expected
  - grant the "rabbitmqctl stop" a graceful termintation window and only
    then ensure the beam process termination and pidfile removal as well
  - return the actual status with get_status()
* Rework kill_rmq_and_remove_pid()
  - use proc_stop to try to kill by pgrp with -TERM, then -KILL, or
    by the beam process name match, if there is no PID.
  - make it to returns SUCCESS/ERROR
* Fix action_stop()
  - fail early by the stop_server_process() results without additional
    rabbitmqctl invocations in the get_status() call
  - rework hard-coded sleep 10 to use the gracefull stop windows in the
    stop_server_process() instead
  - ensure the rabbit-start-time removal from CIB before to try to stop
    the server process
  - issue the "stop: action end" log record before the actual end
* Add comments and make logs to be more informational

Related Fuel bug https://bugs.launchpad.net/fuel/+bug/1529897

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>
Co-authored-by: Alex Schultz <aschultz@mirantis.com>

* Add ocf_run wrappers and info log messages for CIB attribute events
* Move "fast" CIB attribute updates before "heavy" operations like
  start/stop/wait to ensure CIB consistent even if the timeouts
  exceeded for the ops
* Delete master and start time attributes from CIB on action_start
  to ensure the correct rabbit nodes uptime evaluation for new
  master elections for corresponding pacemaker resources
* For post-demote notify and action_demote() delete the master
  attribute from CIB as well.
* For post-start notify, update the start time in the CIB even when
  the node is already clustered. Otherwise it would remain running
  in cluster w/o the start time registered, which affects the new
  master elections badly.
* fix wrong log message when joining by a node

Related Fuel bug https://bugs.launchpad.net/fuel/+bug/1530150
https://bugs.launchpad.net/fuel/+bug/1530296

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

Closes-bug: #1530296

Change-Id: Id2258da4f272dc8eca92130d45ecb69a16ed7c35
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

The sleep in not needed according to the
https://www.rabbitmq.com/man/rabbitmqctl.1.man.html
"If a pid_file is specified, also waits for the process
specified there to terminate."

Related Fuel bug https://launchpad.net/bugs/1529897
Related PR
rabbitmq/rabbitmq-server#523

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

Related Fuel bug:
https://launchpad.net/bugs/1529897

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, the rabbit OCF cannot make
proc_stop to try to kill the pid-less beam process
by its name matching because the proc_kill()'s
1st parameter cannot be passed empty.

The fix is to use the "none" value then the pid-less
process must be matched by the service_name instead.

Also, fix the proc_kill to deal with Multi process
pid files as well (there are many pids, a space separated).

Related Fuel bugs:
https://launchpad.net/bugs/1529897
https://launchpad.net/bugs/1532723

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

* Fix get status() to catch beam state and output errors
* Fix action_stop() to force name-based mathcing then no
pidfile and the beam's unresponsive
* Fix proc_stop to use name based matching if no pidfile
found
* Fix proc_stop to retry sending the signal when using the name
based match as well

W/o this patch, the situation is possible when:
- beam's running and cannot process signals, but is reported "not running"
by the get_status(), while in fact it shall be reported as generic error
- which_applications() returned error, while its output is still
being parsed for the "what" match, while it shall not.
- action stop and proc_stop gives up then there is no pidfile and the beam's
running unresponsive.

The solution is to make get_status to return generic error and action
stop to use the rabbit process name matching for killing it.

Related Fuel bug:
https://bugs.launchpad.net/fuel/+bug/1529897

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

W/o this fix, the situation is possible when:
- beam's running and cannot process signals, but is reported "not running"
by the get_status(), while in fact it shall be reported as generic error
- which_applications() returned error, while its output is still
being parsed for the "what" match, while it shall not.
- action stop and proc_stop gives up then there is no pidfile and the beam's
running unresponsive.

The solution is to make get_status to return generic error and action
stop to use the rabbit process name matching for killing it. These and
other related fixes listed below (tl;dr)

* Fix get_status, action_stop, proc_stop then beam's unresponsive
  (ie. fails to process signals or does it very slowly)
  - Fix get status() to catch beam state and output errors
  - Fix action_stop() to force name-based mathcing then no
    pidfile and the beam's unresponsive
  - Fix proc_stop to use name based matching if no pidfile
    found
  - Fix proc_stop to retry sending the signal when using the name
    based match as well
* Fix the get_status() unexpectedly reports generic error
  instead of "not running"
* Add reworked proc_stop and proc_kill functions from the
  ocf-fuel-funcs
* Rework stop_server_process()
  - make it to return SUCCESS/ERROR as expected
  - grant the "rabbitmqctl stop" a graceful termintation window and only
    then ensure the beam process termination and pidfile removal as well
  - return the actual status with get_status()
* Rework kill_rmq_and_remove_pid()
  - use proc_stop to try to kill by pgrp with -TERM, then -KILL, or
    by the beam process name match, if there is no PID.
  - make it to returns SUCCESS/ERROR
* Fix action_stop()
  - fail early by the stop_server_process() results without additional
    rabbitmqctl invocations in the get_status() call
  - rework hard-coded sleep 10 to use the gracefull stop windows in the
    stop_server_process() instead
  - ensure the rabbit-start-time removal from CIB before to try to stop
    the server process
  - issue the "stop: action end" log record before the actual end
* Add comments, adjust logs levels and make them to be more informational

Upstream PRs
rabbitmq/rabbitmq-server#523
rabbitmq/rabbitmq-server#532
rabbitmq/rabbitmq-server#538
rabbitmq/rabbitmq-server#540

Closes-bug: #1529897

Change-Id: I1c382e3cf004630847b6626fabaecaa0094ee271
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

* Add ocf_run wrappers and info log messages for CIB attribute events
* Move "fast" CIB attribute updates before "heavy" operations like
  start/stop/wait to ensure CIB consistent even if the timeouts
  exceeded for the ops
* Delete master and start time attributes from CIB on action_start
  to ensure the correct rabbit nodes uptime evaluation for new
  master elections for corresponding pacemaker resources
* For post-demote notify and action_demote() delete the master
  attribute from CIB as well.
* For post-start notify, update the start time in the CIB even when
  the node is already clustered. Otherwise it would remain running
  in cluster w/o the start time registered, which affects the new
  master elections badly.

Upstream RR rabbitmq/rabbitmq-server#524
Closes-bug: #1530150

Change-Id: I9db3c819031cef620377b4fee08ea92e90b11c70
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

When monitor detected the node as OCF_RUNNING_MASTER, this may be
lost while the monitor checks in progress.
* Rework the prev_rc by the rc_check to fix this.
* Also add info log if detected as running master.
* Break the monitor check loop early, if it shall be exiting to be
  restarted by pacemaker.
* Do not recheck the master status and do not update the master score,
  if the node was already detected by monitor as OCF_RUNNING_MASTER.
  By that point, the running and healthy master shall not be checked
  against other nodes uptime as it is pointless and only takes more
  time and resources for the action monitor to finish.
* Fail early, if monitor detected the node as OCF_RUNNING_MASTER, but
  the rabbit beam process is not running
* For OCF_CHECK_LEVEL>20, exclude the current node from the check
  loop as we already checked it before

Closes-bug: #1531838

Change-Id: I319db307c73ef24d829be44eeb63d1f52f4180fa
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

When monitor detected the node as OCF_RUNNING_MASTER, this may be
lost while the monitor checks in progress.
* Rework the prev_rc by the rc_check to fix this.
* Also add info log if detected as running master.
* Break the monitor check loop early, if it shall be exiting to be
  restarted by pacemaker.
* Do not recheck the master status and do not update the master score,
  if the node was already detected by monitor as OCF_RUNNING_MASTER.
  By that point, the running and healthy master shall not be checked
  against other nodes uptime as it is pointless and only takes more
  time and resources for the action monitor to finish.
* Fail early, if monitor detected the node as OCF_RUNNING_MASTER, but
  the rabbit beam process is not running
* For OCF_CHECK_LEVEL>20, exclude the current node from the check
  loop as we already checked it before

Related Fuel bug:
https://launchpad.net/bugs/1531838

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

RabbitMQ will resolve <prefix>-<fqdn> hostnames to valid mgmt/messaging IP

Change-Id: Ifc2af16b08663655d365587ea6f45c87bfc68698
Depends-On: I9813fa8c20d47e0ef1e251fe5ac8d01d08fe7703
Closes-bug: #1528707

It would allow to instantiate multiple rabbit clusters constructed
from prefix-based instances of rabbit nodes.

Doing otherwise might not trigger the restart while it is clearly
needed.

Doing otherwise might not trigger the restart while it is clearly
needed.

Upstream PR: rabbitmq/rabbitmq-server#560

Change-Id: I480ebaddc98fa0784098efbf0c5ab8c512c8661d
Closes-Bug: #1513421

Currently time-out when running 'rabbitmqctl list_channels' is treated
as a sign that current node is unhealthy. But it could not be the
case, as the hanging channel could be actually on some other
node. Given that currently we have more than one bug related to
'list_channels', it makes sense to improve diagnostics here.

This patch doesn't change any behaviour, only improves logging after
time-out happens. If time-outs continue to occur (even with latest
rabbitmq versions or with backported fixes), we could switch to this
improved list_channels and kill rabbitmq only if stuck channels are
located on current node. But I hope that all related rabbitmq bugs
were already closed.

timeout(1) manpage mentions 124 as another valid return code from, in addition to 128 + signal-number.

…ist-channels-diagnostics

Improve OCF script diagnostics for timed-out 'list_channels'

Upstream: rabbitmq/rabbitmq-server#571

Shell was sometimes complaining at line 1447 due to empty `rc_check`

Change-Id: I9411fbc41f8ebf6ac41504ff7456ee7952485564
Partial-Bug: #1531838

Upstream PR: rabbitmq/rabbitmq-server#563

Currently time-out when running 'rabbitmqctl list_channels' is treated
as a sign that current node is unhealthy. But it could not be the
case, as the hanging channel could be actually on some other
node. Given that currently we have seen more than one bug related to
'list_channels', it makes sense to improve diagnostics here.

This patch doesn't change any behaviour, only improves logging after
time-out happens. If time-outs continue to occur (even with latest
rabbitmq versions or with backported fixes), we could switch to this
improved list_channels and kill rabbitmq only if stuck channels are
located on current node. But I hope that all related rabbitmq bugs
were already closed.

Change-Id: I4746d3a4e85dc2a51af581034ae09a1cf0eefce2
Partial-Bug: #1515223
Partial-Bug: #1513511

curl is used by OCF script for fetching definitions (queues etc.), but
results of that invocation is shown as garbage in pacemaker logs -
progress indicator doesn't make any sense in logs.

According to curl manpage the following combination of options should be
used "--silent --show-error" - this will suppress only progress
indicator, errors will still be shown.

Also other short curl options are replaced with their long counterparts
- for improved readability.

Fix multiple nodes may be reported in logs as the running master

Related Fuel bug https://bugs.launchpad.net/bugs/1540936

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

The running master is always inside of its own cluster.
Fix the cluster membership check when a node is the master.

Fix multiple nodes may be reported in logs as the running master

Closes-bug: #1540936

Change-Id: Ic2dfe7b2ba657b9bf06d97f49ddb4b69f2f4e063
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

Move check if we are current cluster master to earlier place in code.
That way we will avoid unnecessary operations for master case.

The action_stop may sometimes stop the rabbitmq-server gracefully
by the PID, but leave unresponsive beam.smp processes running and
spoiling rabbits. Those shall be stopped as well. The solution is:
- make proc_stop() to accept a pid=none to use a name matching instead
- make kill_rmq_and_remove_pid() to stop by the beam process matching as well
- fix stop_server_process() to ensure there is no beam process left running

Closes-bug: #1541029

Change-Id: Ib9669d15bb714be8a88fd65d7f1815173da788d3
Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

The action_stop may sometimes stop the rabbitmq-server gracefully
by the PID, but leave unresponsive beam.smp processes running and
spoiling rabbits. Those shall be stopped as well. The solution is:
- make proc_stop() to accept a pid=none to use a name matching instead
- make kill_rmq_and_remove_pid() to stop by the beam process matching as well
- fix stop_server_process() to ensure there is no beam process left running

Related Fuel bug: https://launchpad.net/bugs/1541029

Signed-off-by: Bogdan Dobrelya <bdobrelia@mirantis.com>

Doing otherwise causes node to restart when get_monitor is called
within action_promote - it does not find a master and assumes that
it is running out of cluster.

Also, code is refactored a little bit - a new function returning
current master is created and is used in the changed code.

Closes-Bug: #1543154
Change-Id: If14fcfc915d76c9580be0a097b250d79cf953b9e

Without the break we always wait for 50 seconds, even if we don't need
to wait at all.

Change-Id: Ib361fbac714d61056f4b9d71f23bb74af33abf77

 + extracted function checking if we are in the same cluster with
   given node

 + made post-promote ignore promotion of self. Previously it was
   done inside jjj_join, but now we need to do that before the
   new check.

 + now we write "post-promote end" log entry at the very
   end of post-promote, not somewhere in the middle.

Closes-Bug: #1544036
Change-Id: Id28d6c94abe5d96452f7ecba2b3fe022f40afa0d

Without the break we always wait for 50 seconds, even if we don't need
to wait at all.

There are three types of rabbitmq attributes for pacemaker nodes:
	-'rabbit-master'
	-'rabbit-start-time'
	- timeouts:
		-'rabbit_list_channels_timeouts'
		-'rabbit_get_alarms_timeouts'
		-'rabbit_list_queues_timeouts'

Attributes with names 'rabbit-master' and 'rabbit-start-time' should be
public because we monitor this attributes in cycle for all nodes in our
script.

All timeouts attributes were changed to private to avoid unnecessary
transitions.

Also, --lifetime and --node options were removed for attrd_updater as
'lifetime' for this command is always 'reboot' and 'node' default value
is local one.

Private attributes usage in rabbitmq script

There are three types of rabbitmq attributes for pacemaker nodes:
	-'rabbit-master'
	-'rabbit-start-time'
	- timeouts:
		-'rabbit_list_channels_timeouts'
		-'rabbit_get_alarms_timeouts'
		-'rabbit_list_queues_timeouts'

Attributes with names 'rabbit-master' and 'rabbit-start-time' should be
public because we monitor this attributes in cycle for all nodes in our
script.

All timeouts attributes were changed to private to avoid unnecessary
transitions.

Also, --lifetime and --node options were removed for attrd_updater as
'lifetime' for this command is always 'reboot' and 'node' default value
is local one.

Closes-bug: #1524672
Change-Id: Ie45ae3a82b8daa35dbdd977dc894877160af457b

We still see that rabbitmqctl list_channels times out from time
to time, though the RabbitMQ cluster is absolutely healthy in any
other aspect.

Setting max_rabbitmqctl_timeouts to 3 seems to be a sane default
to help avoid unnecessary restarts.

Several OCF calls might run simultaneously. For example, it often
happens that two monitor calls intersect. Logging current process id
for each line helps distinguish logs of different calls.

Also aligned get_status() logging with format used in all other
parts of the script.

Doing otherwise causes node to restart when get_monitor is called
within action_promote - it does not find a master and assumes that
it is running out of cluster.

Also, code is refactored a little bit - a new function returning
current master is created and is used in the changed code.

We still see that rabbitmqctl list_channels times out from time
to time, though the RabbitMQ cluster is absolutely healthy in any
other aspect.

Setting max_rabbitmqctl_timeouts to 3 seems to be a sane default
to help avoid unnecessary restarts.

Upstream PR: rabbitmq/rabbitmq-server#650

Closes-Bug: #1550293
Change-Id: I6b0686ef66ba3966e03c8706594f473e9ab01145

 + extracted function checking if we are in the same cluster with
   given node

 + made post-promote ignore promotion of self. Previously it was
   done inside jjj_join, but now we need to do that before the
   new check.

 + now we write "post-promote end" log entry at the very
   end of post-promote, not somewhere in the middle.

Upstream PR: rabbitmq/rabbitmq-server#597

curl is used by OCF script for fetching definitions (queues etc.), but
results of that invocation is shown as garbage in pacemaker logs -
progress indicator doesn't make any sense in logs.

According to curl manpage the following combination of options should be
used "--silent --show-error" - this will suppress only progress
indicator, errors will still be shown.

Also other short curl options are replaced with their long counterparts
- for improved readability.

Change-Id: I5ae35b3f76dc33be68c79f5dc983f0c779529fb9
Closes-Bug: #1540831

Several OCF calls might run simultaneously. For example, it often
happens that two monitor calls intersect. Logging current process id
for each line helps distinguish logs of different calls.

Also aligned get_status() logging with format used in all other
parts of the script.

Upstream PR: rabbitmq/rabbitmq-server#653

Closes-Bug: 1553089
Change-Id: Icbaeb560021f70ef13e062cb79fe2cba84e33dce

[OCF HA] Log process id in RabbitMQ OCF script

This reverts commit 686bed1b4f090d7f6fd368b94a5ced12c8e28744, reversing
changes made to d42a753d75dc419c123de257a974ca9c175789f7.

Change-Id: I56ce3671558cf12ab7ce7d616e14cf27f3adb5f1
Closes-bug: #1556123

This reverts commit 4aeaa79bc566c81bc7f5c20d7afbe39c32771aba.

Related Fuel bug https://bugs.launchpad.net/fuel/+bug/1556123