Apply backpressure in misc places to prevent resource exhaustion issues

[hiddenalpha]

Relates to SDCISA-15833, Probably fixes #170 .

Usage example (replace dots with your respective quota):

  RedisQues.builder()
      .withRedisMonitoringReqQuota(...)
      .withCheckQueueRequestsQuota(...)
      .withQueueStatsRequestQuota(...)
      .withGetQueuesItemsCountRedisRequestQuota(...)
      .build();

If nothing is configured, RedisQues will use INT_MAX to emulate the behavior as it was before.

[dominik-cnx]

Curious to know: Do the clients using redisques typically expect such feedback. I mean can they handle it fine and continue processing / recover fine.

[dominik-cnx]

Update: I think we end up in the error case below in the Gateleen MonitoringHandler. It is currently considered en "error", we would need to relax it to a "warn", hmm, maybe OK, do you plan to implement some slowdown around here?

public void updateLastUsedQueueSizeInformation(final String queue) {
    log.trace("About to update last used Queue size counter");
    vertx.eventBus().request(getRedisquesAddress(), buildGetQueueItemsCountOperation(queue), (Handler<AsyncResult<Message<JsonObject>>>) reply -> {
        if (reply.succeeded() && OK.equals(reply.result().body().getString(STATUS))) {
            final long count = reply.result().body().getLong(VALUE);
            vertx.eventBus().publish(getMonitoringAddress(), new JsonObject().put(METRIC_NAME, prefix + LAST_USED_QUEUE_SIZE_METRIC).put(METRIC_ACTION, "update").put("n", count));
        } else {
            log.error("Error gathering queue size for queue '{}'", queue);
        }
    });
}

[hiddenalpha]

As far as I can tell, those are not the same operations. Seems as there was some trouble naming things. As those operation names sound confusingly similar. I don't understand how they actually differ.

Up to now the only downstream I found was QueueStatsService.

Can you confirm/disconfirm?

[dominik-cnx]

I haven't tried to understand how it all works, but in this context I would except that we get some HeavyweightException (or maybe StacktraceException) compared to the ThriftyExceptionFactory that would then give as some LightweightException (StacktraceLessException). It would make things easier to understand I believe. But is this then not convenient to work with?