Refactor Elasticsearch metrics

[alexshtin]

What changed?
Refactor Elasticsearch related metrics. Renamed them to have elasticsearch prefix and arranged other definitions.
Also moved ack channel creation into processor.go.

Why?
Unify metrics.

How did you test it?
Modified existing and added new unit tests. Local canary run.

Potential risks
Few Elasticsearch metrics are renamed and dashboards need to be updated.

Is hotfix candidate?
No.

[wxing1292]

any possibility this channel is already acked or what if Done is called > 1 times

[wxing1292]

above code seem to have possibility to get the the channel > 1 times, so a.ackChInternal <- ack should be

select {
case a.ackChInternal <- ack:
default:
    // noop
}

[alexshtin]

This is just a refactoring. This code was there just wasn't extracted as a method. There are 2 calls to Done() in code above: normal case (success/error) or duplicate visibilityTaskKey. Both of them a guaranteed to be called only once by mapToAckChan and in case of duplicate the item get overwritten and never used again.

[wxing1292]

which case will have visibilityTaskKey == ""

[alexshtin]

When extractVisibilityTaskKey wasn't able to extract it. It doesn't return error because it already logged it and reported metrics. So here I just ignore the item.

[paulnpdev]

I know everyone feels code should be self-documenting (and it should whenever possible) but personally I think that when an experienced reviewer familiar with the code has a question about what's going on, that's a good sign that a comment might be worthwhile. Just saying :)

[wxing1292]

maybe return an error?

[alexshtin]

I think it is ok. It should never ever happend. Actually event removing ,ok part would be a better option and just let it panic with default go message.