Last Finalized Snapshot Detection and Alert System

[Seth-Schmidt]

Is your feature request related to a problem?
There is no system setup to notify when a master node has not been processing snapshots or has gone down.

Describe the solution you'd like
The /health endpoint of the core-api should periodically query the protocol state contract for the current epoch and compare the epochId against active projects to detect how many epochs have passed since the last finalization. If any sampled projects have not finalized within a reasonable time (20-30 epochs?), then pooler will send a Slack alert notifying of a failure to finalize.

Describe alternatives you've considered
It might be worth exploring an external monitoring service that tracks the status independently since finalization requires submissions from multiple master nodes. However, it would be possible to get additional information for specific nodes if the monitoring is done internally.

[anomit]

The associated PR has been reviewed by me. Ideally would need another review from one of the @PowerLoom/backend-engineering team members before merging. Moving completion date to this week, Tuesday EOD.

[Seth-Schmidt]

Moving completion date to allow for final review of changes mentioned here: #77 (review)

[SwaroopH]

@Seth-Schmidt update this please ^

[Seth-Schmidt]

merged to nms_master: #77

closing