Add processing timings for each step in traces pipeline

[srikanthccv]

No description provided.

[ankitnayan]

Could we figure out anything from this?

[srikanthccv]

We now have clarity on the time it takes in each step https://signoz-team.slack.com/archives/C02SGR46RRR/p1714625576474649. We looked into whether async inserts help, the wait_for_async_insert setting decides whether we wait till ack. If we do not wait for ack we risk losing data because we won't be able to retry. If we wait for ACK, it doesn't make any difference from the write duration.

We need to parallelize these 4 calls

signoz-otel-collector/exporter/clickhousetracesexporter/writer.go

Lines 410 to 436 in 275a1e6

	func (w *SpanWriter) WriteBatchOfSpans(ctx context.Context, batch []*Span) error {
	if w.spansTable != "" {
	if err := w.writeModelBatch(ctx, batch); err != nil {
	w.logger.Error("Could not write a batch of spans to model table: ", zap.Error(err))
	return err
	}
	}
	if w.indexTable != "" {
	if err := w.writeIndexBatch(ctx, batch); err != nil {
	w.logger.Error("Could not write a batch of spans to index table: ", zap.Error(err))
	return err
	}
	}
	if w.errorTable != "" {
	if err := w.writeErrorBatch(ctx, batch); err != nil {
	w.logger.Error("Could not write a batch of spans to error table: ", zap.Error(err))
	return err
	}
	}
	if w.attributeTable != "" && w.attributeKeyTable != "" {
	if err := w.writeTagBatch(ctx, batch); err != nil {
	w.logger.Error("Could not write a batch of spans to tag/tagKey tables: ", zap.Error(err))
	return err
	}
	}
	return nil
	}

and writeIndexBatch can further be broken to send in multiple batches.

[ankitnayan]

If we wait for ACK, it doesn't make any difference from the write duration.

Still it would lead to better batching when multiple otel-collectors are writing to clickhouse?

SigNoz/signoz#5013

[srikanthccv]

Yes, and you already mentioned the main drawback of the async in the linked issue.