Add logs and metrics for append segments upgraded by a concurrent replace task

[kfaraz]

Changes

• Add metrics segment/upgraded/count and segment/upgradedRealtime/count
  emitted with dimensions taskId, dataSource, taskType, groupId, interval, etc.
• Add new logs and cleanup existing ones
• Minor method renames

[AmatyaAvadhanula]

Thanks @kfaraz. These metrics will be very helpful in monitoring concurrent append and replace.
I have a minor comment about changing realtime to pending in the metric name and description.
Also, do you think that we should also add a metric in SegmentTransactionalAppendAction whenever an upgrade segments entry is created because of a segment append within an interval locked by a REPLACE lock?

[AmatyaAvadhanula]

Please rename this to pendingSegment/upgraded/count or something similar.
Upgraded pending segments need not correspond only to realtime tasks.

[AmatyaAvadhanula]

Just upgradedSegments seems sufficient.

[AmatyaAvadhanula]

... upgraded [%d] segments ...

[gargvishesh]

Think it is within the line limit.

Suggested change

	final List<List<DataSegment>> segmentsLists = Lists.partition(
	new ArrayList<>(segments),
	SEGMENT_INSERT_BATCH_SIZE
	);
	final List<List<DataSegment>> segmentsLists = Lists.partition(new ArrayList<>(segments), SEGMENT_INSERT_BATCH_SIZE);

[kfaraz]

The soft limit for better readability is actually 80 chars.
120 chars is the hard limit which we should not exceed.

---

Obviously, all of this is subjective and depends on personal coding style.
But generally, I too avoid breaking lines in method/constructor calls if

• they can be fit "comfortably" in a single line
• they are not difficult to follow

The end goal is to have easily readable code.

So I would prefer

performAction(arg1, arg2, arg3, arg4, arg5);

over

performAction(
    arg1,
    arg2,
    arg3,
    arg4,
    arg5
);    

but NOT

performAction(computeFirstArg(), object.getSecondArg(), arg3, arg4, thirdArgIfNonNull());

over

performAction(
    computeFirstArg(),
    object.getSecondArg(),
    arg3,
    arg4,
    thirdArgIfNonNull()
);

For the case at hand, we have two options:

Option 1:

final List<List<DataSegment>> segmentsLists = Lists.partition(
        new ArrayList<>(segments),
        SEGMENT_INSERT_BATCH_SIZE
);

Option 2:

final List<List<DataSegment>> segmentsLists = Lists.partition(new ArrayList<>(segments), SEGMENT_INSERT_BATCH_SIZE);

I feel Option 1 here reads better for 2 reasons:

• Option 2 is definitely too long, it's right at the 120 char limit.
• Option 2 is a little more complex cognitively, it has too much stuff happening in a single line.

Hope you find that helpful (and not too verbose!). 🙂

[kfaraz]

Thanks for the suggestions, @AmatyaAvadhanula ! Let me try to rename the added metrics and add another one for the transactional append action.

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.