[server][dvc] Handle exceptions while fetching offsets

[majisourav99]

Problem Statement

During ingestions, inside processConsumerRecord most of the code path is under a try-catch block and exceptioned are handled, but later code path does not handle exceptions which leads the store ingestion task to unsubscribe and sit idle.

Solution

This PR handles the exceptions that could be thrown outside internalProcessConsumerRecord and allows the push to fail early that waiting for the 24hr timeout to make it failed.

Code changes

• Added new code behind a config. If so list the config names and their default values in the PR description.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[sushantmane]

@majisourav99 - this PR has already been approved. Just checking, are you planning to make any additional changes? Or is it no longer needed?

[ZacAttack]

Ditto what @sushantmane said. Let's move it or lose it @majisourav99

[majisourav99]

Ditto what @sushantmane said. Let's move it or lose it @majisourav99

Let me close this. IIRC I fixed the issue in some other places.