Fixed #383, #380, #379, #378

[grtjn]

• Fixed Reassigning replicas doesn't spread replicas #383: Using multiple replica-names for one forests will make them spread over different following hosts, not all on the same.
• Fixed Ignore replica forests when bootstrapping to standalone #380: You need multiple hosts for forest replication. If you bootstrap against a single host, forest-replication is silently ignored (under assumption you are doing a test deployment on localhost or dev)
• Fixed Replication forests not created once forests exists #379: If you later decide you want forest replication, the code to add them to existing forests was skipped. Necessary for adding rep to system databases.
• Fixed Forest replication doesn't work unless forests-per-host #378: Forest replication was only applied if <forests-per-host> was used. Now also works without.

[grtjn]

Sorry for piling up changes for 4 tickets, but they were very closely related. I can provide a sample ml-config to test this..

[rlouapre]

+1
Any idea when this PR will be merged

[grtjn]

@rlouapre You can pull this locally using upgrade: ml upgrade --fork=grtjn --branch=forest-rep-improvements

[dmcassel]

@rlouapre, it hasn't been merged because neither @paxtonhare nor I have had a chance to test it. Have you tried out these changes?

[grtjn]

You can relatively easy test this and #366 as well by following these steps:

• create a cluster of at least 2 nodes, you can use mlvagrant for this
• create a roxy or slush project, and configure it to point to the cluster
• run: ./ml upgrade --fork=grtjn --branch=366-replicate-internals
• edit ml-config.xml, and change the assignments section into:

  <assignments xmlns="http://marklogic.com/xdmp/assignments" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://marklogic.com/xdmp/assignments assignments.xsd">
    <assignment>
      <forest-name>@ml.content-db</forest-name>
      <replica-names>
        <replica-name>@ml.content-db-rep1</replica-name>
      </replica-names>
      @ml.forest-data-dir-xml
    </assignment>
    <assignment>
      <forest-name>@ml.content-db-rep1</forest-name>
      @ml.forest-data-dir-xml
    </assignment>
    @ml.test-content-db-assignment
    @ml.test-modules-db-assignment
    @ml.rest-modules-db-assignment
    <assignment>
      <forest-name>@ml.modules-db</forest-name>
      <replica-names>
        <replica-name>@ml.modules-db-rep1</replica-name>
      </replica-names>
    </assignment>
    <assignment>
      <forest-name>@ml.modules-db-rep1</forest-name>
    </assignment>
    @ml.schemas-assignment
    @ml.triggers-assignment
  </assignments>

• run: ./ml local bootstrap
• run: ./ml local bootstrap --replicate-internals

The end result should be that all forests have a local replica failover.

[rlouapre]

Just see this PR has been merged.

Tested it successfully on a simple 3 nodes cluster.

A couple of questions:

• I need to create a replica-name for each node of the cluster. Would it be possible to discover all nodes of the cluster and create replica forest on each node except the current node?

<forest-name>@ml.content-db</forest-name>
      <replica-names>
        <replica-name>@ml.content-db-rep1</replica-name>
        <replica-name>@ml.content-db-rep2</replica-name>
      </replica-names>

• I assume Roxy deployment script will not changed existing forests setup. Would it be possible to change this behavior so that Roxy script managed the replica forest settings? Scenario: new nodes are added to the cluster re-run bootstrap will create forest and replicas on new nodes.

[grtjn]

@rlouapre Keep in mind that for databases with a forests-per-host setting, a forest is created on each host of the target group, and for each forest a replica would be created. It doesn't make sense to create n replicas for each forest. That would completely kill the performance of the cluster. Each replica adds 100% IO for that forest.

Yes, Roxy bootstrap is designed to keep things in place as much as possible, and only add changes. If you started with 3 nodes, did a bootstrap, and then add node 4, you can rerun bootstrap to add more forests. I haven't tried how well that works with replica forests. Rep forests might get redistributed, not sure how well that works..

If this still leaves questions or issues, maybe open new tickets..