Broker can receive checkpoint create command from the gateway

[deepthidevaki]

Depends on #9719

Broker can receive a checkpoint request from the gateway.

• Decide if command-api will receive the command or we create a new end-point.
• Writes the command to the logstream
• Similar to other commands
  • should send the response back after it is processed by the StreamProcessor
  • should go through the backpressure.

[deepthidevaki]

I discussed with @oleschoenburg the following options.

1. Use CommandApiHandler to receive take-backup command and write to the logstream. But define a new BackupApiHandler to process other backup requests like status query and delete.

• Pro:
  • Take-backup is processed like any other command. It is written to the logstream and the response is send by the StreamProcessor
  • We can apply backpressure if needed.
• Cons:
  • Backup requests are handled in two different places

2. Define a new BackupApiHandler to receive all backup commands including take-backup. We will define a new RequestType.BACKUP. In this option, there are several way to handle the take-backup request.
   a. BackupApiHandler receive take-backup command, write the checkpoint record to the logstream and immediately send success response to the gateway, without waiting for it to be processed/committed. Gateway/User then must query the status of the backup and retry if needed (in case the backup was not triggered, because the command was lost due to a leader change).
   b. BackupApiHandler receives take-backup command, write the checkpoint record to the logstream, wait until the record is processed and send success response to the gateway. Here BackupApiHandler should somehow wait for the record is to be processed, but it wouldn't know what is the result of the processing. So it can only send an empty success response.
   c. BackupApiHandler receives take-backup command, write the checkpoint record to the logstream, and do not send the response. The response will be send by the streamprocessor after the record is processed.

We decided to go with 2c because:

1. All backup requests are handled at a single point.
2. By letting StreamProcessor sending the response backup, we follow the usual command processing pattern.
3. CheckpointRecordProcessor can then send either a success response (backup is triggered) or a response that indicates a checkpoint already exists, so no backup will be triggered.
4. Other backup requests will be responded by the BackupApiHandler itself, but each request can define its own response types.