PendingAddOp keeps refCnt onthe toSend buffer longer than needed.

[jvrao]

BUG REPORT

1. Please describe the issue you observed:
   PendingAddOp:maybeRecycle()->recycle() keeps the buffer until writeComplete() is called
   for each bookie write. We need to keep this buffer only until it is successfully transferred by netty.
   In the current code, the write is retired only if disableEnsembleChangeFeature is enabled. Otherwise, there is no point in keeping this buffer around.

Netty direct buffers are too scarce in the heavy workload scenarios. Keeping these buffers around longer than needed negatively affects the throughput on the client.

[sijie]

@jvrao don't we have to keep the buffer around for retries? any idea that you are thinking of to improve here?

[jvrao]

@sijie toSend buffer is not needed for retries as we discussed on slack.