Per-vhost message store

[michaelklishin]

This idea has been floating around since at least late 2012: we should use separate message stores for different vhosts, e.g. two (for persistent, transient messages) per vhost.

Some of the benefits are:

• Better isolation of physical resources between hosts (e.g. in the spirit of Dealing with Noisy Neighbors on brokers with multiple vhosts #498, Per-vhost connection limit #500, Per-vhost limit on the number of queues #501)
• Improved availability should a message store terminate or significantly fall behind
• (At least potentially) Less contention, more parallelism during normal operation of a multi-tenant node
• Opens the door to moving different vhosts to different disks

Downsides:

• Virtual host initialisation (or restart) will be more expensive. Most users don't seem to use a lot of virtual hosts, so this is a reasonable trade-off.

[michaelklishin]

effort-high because migration of existing data can take a while to cover well with tests. Otherwise this is effort-medium.

[hairyhum]

What if we enable per vhost message stores for new vhosts only, and recommend to shovel messages for migration?

[michaelklishin]

I'm afraid that's not gonna fly with hosted RabbitMQ providers, and at least some Cloud Foundry users. It would be great to automatically migrate messages on boot (if we can).

[hairyhum]

Initial research:
Thanks to rabbit_msg_store architecture enabling per-vhost grouping is pretty easy by starting message stores in subdirectories and controlling them with supervisor.
Migration of old data can still be a problem, since there is no vhost information in message records.

[hairyhum]

Migration mechanism proposal:

move_messages_to_vhost_store() ->
    Queues = get_durable_queues(),
    OldStore = run_old_persistent_store(),
    Migrations = spawn_for_each(fun(Queue) -> 
                                    migrate_queue(Queue, OldStore) 
                                end, Queues),
    wait_for_completion(Migrations),
    delete_old_store(OldStore).

migrate_queue(Queue, OldStore) ->
    OldStoreClient = get_client(OldStore),
    NewStore       = ensure_new_store_exist(get_vhost(Queue)),
    NewStoreClient = get_client(NewStore),
    walk_queue_index(
        fun(MessageIdInStore) ->
            Msg = get_msg_from_store(OldStoreClient),
            put_message_to_store(Msg, NewStoreClient)
        end,
        Queue).

[michaelklishin]

@hairyhum given that upgrades perform a backup of the node data dir, that sounds reasonable.