Fix for concurrency problem in Elasticsearch and fix for excluding invalid mountnames. Related issues are 1472, 901 and 1455

[KarthikeyanV-TechM]

Details of changes:

1. Added a per-mountname lock directly inside regardDeviceAlarm. This way, if Kafka delivers hundreds of updates for the same device at once, they’ll queue up and be processed sequentially per device (no version conflicts in Elasticsearch).
2. All messages for the same mountname will be queued in sequence.
3. Different devices (mountname values) are processed in parallel (no global bottleneck).
4. Prevents version_conflict_engine_exception completely and data losses in ES.
5. Added a new utility called alarmLogTracker.js which tracks and logs all alarm-notification related updates into a log file alarm-notification.log
6. This log file will retain a logs of size upto 200MB for 10days.
7. Fix for excluding invalid mountnames from /v1/provide-list-of-cached-devices API.
8. Related issues are 1472, 901 and 1455

[PrathibaJee]

Hi karthikeyan,
Thanks for working out in detail.
The proposed solution looks good. After going through the changes I have some doubts regarding (considering that the prod environment produce millions of alarms to MWDI consumer).

1. memory leak &
2. error isolation

feel free to either fix or comment further. Many thanks.

[PrathibaJee]

good to release the lock in the finally block.

[KarthikeyanV-TechM]

Done.

[PrathibaJee]

Kindly remove the diamond symbol. Thanks.

[KarthikeyanV-TechM]

Done.

[PrathibaJee]

If this is specific to local environment. Please remove. Thanks.

[KarthikeyanV-TechM]

Done.

[PrathibaJee]

We're storing a Promise and a queue count for each mountname (id) in locks and queueCounts, but never removing them after processing is complete.
As MWDI already have memory consumption problems, suggesting to clean up both maps when the queue count drops to zero. Thanks.

finally {
  release();
  const newCount = queueCounts.get(id) - 1;
  if (newCount <= 0) {
    queueCounts.delete(id);
    locks.delete(id);
  } else {
    queueCounts.set(id, newCount);
  }
}

[KarthikeyanV-TechM]

Done.

[PrathibaJee]

Check if fn() hangs or throws an error, it could block the queue for that id.
If such scenarios happen , add a timeout wrapper or error isolation, something like this,

const timeout = (ms) => new Promise((_, reject) => setTimeout(() => reject(new Error("Timeout")), ms));
const result = await Promise.race([fn(), timeout(10000)]); // 10s timeout

[KarthikeyanV-TechM]

Done.