Add instrumentation and logging to BQ sink.

[lavkesh]

Acceptance criteria:

• Analysis of metrics for BQ sink.
• Implementation.

Discussion:

Insert time.
Counter for success/failures of insert messages.
No of Error messages(deserialisation and repose errors from bigquery) or dlqed.
Log the offsetinfo when the error happens.
table/dataset creation logging and metrics.
stencil Proto update logging and metrics. (log exceptions etc)
Think about completeness/freshness/deduplication.

[lavkesh]

Core Metrics firehose analysis.

Option 1:

sink_messages_total(type)

• Counter to indicate how many messages were pass through sink
• The values of type can be(total, success, failures)
• sum(sink_messages_total(type=total)) = sum(sink_messages_total(type=success|failure))

global_messages_total(type)

• counter to indicate how many unique messages were pass through firehose.
• type can be (consumer, sink, dlq, ignored,filtered)
• sum(global_messages_total(type=consumer)) = sum(global_messages_total(type=sink|dlq|ignored))

retry_messages_total(type,error_type)

• counter to indicate how many messages were processed by SinkWithRetryDecorator
• type can be (total, success, failures)
• error_type are defined in ErrorTypes class.

dlq_messages_total(type,error_type)

• counter to indicate how many messages were processed by SinkWithDLQ
• type can be (total, success, failures)

error_messages_total(errortype)

• counter to indicate how many messages has errors after processed by sink
• errortypes are defined in ErrorTypes class

Option 2:
input_message_total(scope)

• counter to indicate input messages to different scopes.
• scope= consumer,sink,retry,dlq

success_message_total(scope)

• counter to indicate success messages for different scopes
• scope= consumer,sink,retry,dlq

failed_message_total(socpe)

• counter to indicate failed messages for different scopes
• scope= consumer,sink,retry,dlq

[lavkesh]

Insert time-> BQ insert time. [Sink metrics]
Counter for success/failures of insert messages. [ Sink metrics]
No of Error messages(deserialisation and repose errors from bigquery) or dlqed.[ Sink metrics]
Log the offsetinfo when the error happens[log bigquery errors while parsing, and print toptic, offset , partition in abstract sink]
table/dataset creation logging and metrics.[BigqueryOperationTotal, api_name=, table=,dataset=] [time_taken_operation]
stencil Proto update logging. (log exceptions etc)

Think about completeness/freshness/deduplication.