helm: use PVC for rabbitmq data

[audrium]

closes reanahub/reana-message-broker#41

To test:

• Redeploy the cluster with NFS provisioner and REANA_MAX_CONCURRENT_BATCH_WORKFLOWS: 1
• Check the /var/reana locally, rabbitmq folder shouldn't be there:
• Check if rabbitmq files are in NFS PVC:

reana ❯ kubectl exec -it reana-dev-storage-nfs-server-provisioner-0  -- bash
bash-5.0# cd /export/pvc-77e7cbb4-7dce-4a97-b7f0-6305a6a7c684/
bash-5.0# ls
config  rabbitmq  users  uwsgi

• Check RabbitMQ pod logs (note there is a line about db init):

reana ❯ kubectl logs reana-message-broker-0 | grep database
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@localhost
2021-10-22 11:51:35.616 [info] <0.274.0> Running boot step database defined by app rabbit
2021-10-22 11:51:35.616 [info] <0.274.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@localhost is empty. Assuming we need to join an existing cluster or initialise from scratch...
2021-10-22 11:51:36.026 [info] <0.274.0> Running boot step database_sync defined by app rabbit
2021-10-22 11:51:36.158 [info] <0.828.0> Statistics database started.

• Add some sleep time (e.g. sleep 120) to the helloworld serial demo
• Launch 3 workflows
• Open RabbitMQ UI: http://localhost:31672/#/queues/%2F/workflow-submission (test,1234)
• Inspect that 2 workflows are in the queue (1 ready, 1 unack)
• Delete RabbitMQ pod: kubectl delete pod reana-message-broker-0
• Queue should be persisted
• Workflows should finish
• Check RabbitMQ pod logs (note there is no line about db init):

reana ❯ kubectl logs reana-message-broker-0 | grep database
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@localhost
2021-10-22 12:15:17.399 [info] <0.274.0> Running boot step database defined by app rabbit
2021-10-22 12:15:17.427 [info] <0.274.0> Running boot step database_sync defined by app rabbit
2021-10-22 12:15:17.611 [info] <0.836.0> Statistics database started.

When RabbitMQ was using Deployment instead of StatefulSet there was an error after RabbitMQ pod deletion:

2021-10-22 14:11:22.082 [error] <0.505.0> Failed to start message store of type msg_store_transient for vhost '/': {{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.80454629>,ok}]},transient,600000,worker,[rabbit_msg_store]}}
2021-10-22 14:11:22.082 [info] <0.509.0> [{initial_call,{rabbit_msg_store,init,['Argument__1']}},{pid,<0.509.0>},{registered_name,[]},{error_info,{exit,{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,608}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.504.0>,<0.503.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.274.0>]},{message_queue_len,0},{messages,[]},{links,[<0.504.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,1598},{stack_size,28},{reductions,2068}], []
2021-10-22 14:11:22.082 [error] <0.509.0> CRASH REPORT Process <0.509.0> with 0 neighbours exited with reason: no match of right hand value {error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}} in rabbit_msg_store:init/1 line 724 in gen_server2:init_it/6 line 608
2021-10-22 14:11:22.083 [error] <0.505.0> Unable to recover vhost <<"/">> data. Reason {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.80454629>,ok}]},transient,600000,worker,[rabbit_msg_store]}}}
 Stacktrace [{rabbit_variable_queue,do_start_msg_store,4,[{file,"src/rabbit_variable_queue.erl"},{line,503}]},{rabbit_variable_queue,start_msg_store,3,[{file,"src/rabbit_variable_queue.erl"},{line,488}]},{rabbit_variable_queue,start,2,[{file,"src/rabbit_variable_queue.erl"},{line,479}]},{rabbit_priority_queue,start,2,[{file,"src/rabbit_priority_queue.erl"},{line,85}]},{rabbit_amqqueue,recover_classic_queues,2,[{file,"src/rabbit_amqqueue.erl"},{line,129}]},{rabbit_amqqueue,recover,1,[{file,"src/rabbit_amqqueue.erl"},{line,120}]},{rabbit_vhost,recover,1,[{file,"src/rabbit_vhost.erl"},{line,52}]},{rabbit_vhost_process,init,1,[{file,"src/rabbit_vhost_process.erl"},{line,47}]}]
2021-10-22 14:11:22.083 [info] <0.503.0> supervisor: {<0.503.0>,rabbit_vhost_sup_wrapper}, errorContext: start_error, reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.80454629>,ok}]},transient,600000,worker,[rabbit_msg_store]}}}, offender: [{pid,undefined},{id,rabbit_vhost_process},{mfargs,{rabbit_vhost_process,start_link,[<<"/">>]}},{restart_type,permanent},{significant,false},{shutdown,30000},{child_type,worker}]
2021-10-22 14:11:22.083 [warning] <0.274.0> Unable to initialize vhost data store for vhost '/'. The vhost will be stopped for this node.  Reason: {shutdown,{failed_to_start_child,rabbit_vhost_process,{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.80454629>,ok}]},transient,600000,worker,[rabbit_msg_store]}}}}}
2021-10-22 14:11:22.083 [info] <0.274.0> Running boot step empty_db_check defined by app rabbit
2021-10-22 14:11:22.083 [info] <0.274.0> Will not seed default virtual host and user: have definitions to load...
2021-10-22 14:11:22.083 [error] <0.503.0> Supervisor {<0.503.0>,rabbit_vhost_sup_wrapper} had child rabbit_vhost_process started with rabbit_vhost_process:start_link(<<"/">>) at undefined exit with reason {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhos...",...]},...}}} in context start_error
2021-10-22 14:11:22.083 [info] <0.274.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-10-22 14:11:22.083 [info] <0.274.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-10-22 14:11:22.084 [info] <0.505.0> [{initial_call,{rabbit_vhost_process,init,['Argument__1']}},{pid,<0.505.0>},{registered_name,[]},{error_info,{exit,{error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L",undefined,{#Fun<rabbit_variable_queue.0.80454629>,ok}]},transient,600000,worker,[rabbit_msg_store]}}},[{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,600}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.503.0>,rabbit_vhost_sup_sup,rabbit_sup,<0.274.0>]},{message_queue_len,0},{messages,[]},{links,[<0.503.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,10958},{stack_size,28},{reductions,53715}], []
2021-10-22 14:11:22.084 [info] <0.274.0> Running boot step background_gc defined by app rabbit
2021-10-22 14:11:22.084 [error] <0.505.0> CRASH REPORT Process <0.505.0> with 0 neighbours exited with reason: {error,{{{badmatch,{error,{"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient",eexist}}},[{rabbit_msg_store,init,1,[{file,"src/rabbit_msg_store.erl"},{line,724}]},{gen_server2,init_it,6,[{file,"src/gen_server2.erl"},{line,565}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]},{child,undefined,msg_store_transient,{rabbit_msg_store,start_link,[msg_store_transient,"/var/lib/rabbitmq/mnesia/rabbit@localhost/msg_stores/vhos...",...]},...}}} in gen_server2:init_it/6 line 600

[audrium]

Did some tests on QA to compare rabbitmq deployment with PVC versus hostpath:

• 5 bsm search workflows:
  
• 50 helloworld workflows:
  
  Results seems to be identical, so I guess there is no significant performance downgrade by using PVC with rabbitmq

[tiborsimko]

Tested with the standard 1k pMSSM benchmark, and the slowdown was only minimal. See also my comment to #579 about possible dedicated node in the future.