[DLQ Improvements] Introduce a new metric to track how many events go to the DLQ

[webmat]

There are clearly 4 different outcomes for events in this code block, yet only 3 metrics are being incremented.

When an event is meant to the DLQ, :non_retryable_failures is incremented. This behaviour is fine is the DLQ is not actually enabled. If the DLQ is enabled, however, I think we should increment a new metric instead, so users can track this explicitly.

[andsel]

The original code link should be:

logstash-output-elasticsearch/lib/logstash/outputs/elasticsearch/common.rb

Lines 208 to 224 in 9e10ad7

	if DOC_SUCCESS_CODES.include?(status)
	@document_level_metrics.increment(:successes)
	next
	elsif DOC_CONFLICT_CODE == status
	@document_level_metrics.increment(:non_retryable_failures)
	@logger.warn "Failed action.", status: status, action: action, response: response if !failure_type_logging_whitelist.include?(failure["type"])
	next
	elsif DOC_DLQ_CODES.include?(status)
	handle_dlq_status("Could not index event to Elasticsearch.", action, status, response)
	@document_level_metrics.increment(:non_retryable_failures)
	next
	else
	# only log what the user whitelisted
	@document_level_metrics.increment(:retryable_failures)
	@logger.info "retrying failed action with response code: #{status} (#{failure})" if !failure_type_logging_whitelist.include?(failure["type"])
	actions_to_retry << action
	end