Document anonymous replica

[TarantoolBot]

There is a new type of replica in tarantool, anonymous one. Anonymous
replica is read-only (but you still can write to temporary and
replica-local spaces), and it isn't present in _cluster table.

Since anonymous replica isn't registered in _cluster table, there is no
limitation for anonymous replica count in a replicaset. You can have as
many of them as you want.

In order to make a replica anonymous, you have to pass an option
replication_anon=true to box.cfg. You also have to set 'read_only'
to true.

Let's go through anonymous replica bootstrap.
Suppose we have a master configured with

box.cfg{listen=3301}

And created a local space called "loc"

box.schema.space.create('loc', {is_local=true})
box.space.loc:create_index("pk")

Now, to configure an anonymous replica, we have to issue box.cfg,
as usual.

box.cfg{replication_anon=true, read_only=true, replication=3301}

As mentioned above, replication_anon may be set to true only together
with read_only
The instance will fetch masters snapshot and proceed to following its
changes. It will not receive an id so its id will remain zero.

tarantool> box.info.id
---
- 0
...

tarantool> box.info.replication
---
- 1:
    id: 1
    uuid: 3c84f8d9-e34d-4651-969c-3d0ed214c60f
    lsn: 4
    upstream:
      status: follow
      idle: 0.6912029999985
      peer:
      lag: 0.00014615058898926
...

Now we can use the replica.
For example, we may do inserts into the local space:

tarantool> for i = 1,10 do
         > box.space.loc:insert{i}
         > end
---
...

Note, that while the instance is anonymous, it will increase the 0-th
component of its vclock:

tarantool> box.info.vclock
---
- {0: 10, 1: 4}
...

Let's now promote the replica to a normal one:

tarantool> box.cfg{replication_anon=false}
2019-12-13 20:34:37.423 [71329] main I> assigned id 2 to replica 6a9c2ed2-b9e1-4c57-a0e8-51a46def7661
2019-12-13 20:34:37.424 [71329] main/102/interactive I> set 'replication_anon' configuration option to false
---
...

tarantool> 2019-12-13 20:34:37.424 [71329] main/117/applier/ I> subscribed
2019-12-13 20:34:37.424 [71329] main/117/applier/ I> remote vclock {1: 5} local vclock {0: 10, 1: 5}
2019-12-13 20:34:37.425 [71329] main/118/applierw/ C> leaving orphan mode

The replica just received id 2. We can make it read-write now.

box.cfg{read_only=false}
2019-12-13 20:35:46.392 [71329] main/102/interactive I> set 'read_only' configuration option to false
---
...

tarantool> box.schema.space.create('test')
---
- engine: memtx
  before_replace: 'function: 0x01109f9dc8'
  on_replace: 'function: 0x01109f9d90'
  ck_constraint: []
  field_count: 0
  temporary: false
  index: []
  is_local: false
  enabled: false
  name: test
  id: 513
- created
...

tarantool> box.info.vclock
---
- {0: 10, 1: 5, 2: 2}
...

Now replica tracks its changes in 2nd vclock component, as expected.
It can also become replication master from now on.

Side notes:

• You cannot replicate from an anonymous instance.
• To promote an anonymous instance to a regular one,
  you first have to start it as anonymous, and only
  then issue box.cfg{replication_anon=false}
• In order for the deanonymization to succeed, the
  instance must replicate from some read-write instance,
  otherwise noone will be able to add it to _cluster table.
  Requested by @sergepetrenko in tarantool/tarantool@e17beed.