Monitoring sentinel when "Transitions are disabled" is difficult

[mrjones-plip]

Describe the bug
On a production instance we're seeing the /api/v2/monitoring return an ever growing value for sentinel backlog. The sentinel container logs are showing that the backlog is up to do date though.

To Reproduce
Steps to reproduce the behavior:

1. Deploy an instance with CHT 4.1.0
2. Have activities that require sentinel transitions
3. Ensure transitions are actually getting processed but /api/v2/monitoring shows backlog going up

Expected behavior
/api/v2/monitoring and actual back log are in sync

Logs

2023-03-31 14:48:11 DEBUG: Initiating all tasks
2023-03-31 14:48:11 INFO: Task dueTasks started
2023-03-31 14:48:11 INFO: Task reminders started
2023-03-31 14:48:11 INFO: Task replications started
2023-03-31 14:48:11 INFO: Task outbound started
2023-03-31 14:48:11 INFO: Task purging started
2023-03-31 14:48:11 INFO: Task backgroundCleanup started
2023-03-31 14:48:11 INFO: Task reminders completed
2023-03-31 14:48:11 INFO: Task replications completed
2023-03-31 14:48:11 INFO: Task outbound completed
2023-03-31 14:48:11 INFO: Task purging completed
2023-03-31 14:48:11 INFO: Task dueTasks completed
2023-03-31 14:48:11 INFO: Background cleanup batch: 95881 -> 95881 (0)
2023-03-31 14:48:11 INFO: Task backgroundCleanup completed

Environment

• Instance: MoH Mali CHW
• Browser: NA
• Client platform: NA
• App: sentinel
• Version: 4.1.0

Additional context
App Monitoring showed (private GH issue) that this happened EXACTLY after we upgrade from 4.0.1 to 4.1.0 on Mar 9th:

Possibly related to the last time this bui happened in #7113 ?

[kennsippell]

2023-04-03 22:30:21 ERROR: Failed loading transition "muting" 
2023-04-03 22:30:21 ERROR: Error: Configuration error. Config must define have a 'muting.mute_forms' array defined.
    at Object.init (/home/kenn/cht-core/shared-libs/transitions/src/transitions/muting.js:181:13)
    at Object.loadTransition [as _loadTransition] (/home/kenn/cht-core/shared-libs/transitions/src/transitions/index.js:184:16)
    at /home/kenn/cht-core/shared-libs/transitions/src/transitions/index.js:149:12
    at Array.forEach (<anonymous>)
    at Object.loadTransitions (/home/kenn/cht-core/shared-libs/transitions/src/transitions/index.js:139:25)
    at Object.loadTransitions (/home/kenn/cht-core/sentinel/src/transitions.js:10:20)
    at /home/kenn/cht-core/sentinel/src/config.js:64:32
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
2023-04-03 22:30:21 WARN: Disabled transition "mark_for_outbound" 
2023-04-03 22:30:21 INFO: Loading transition "create_user_for_contacts" 
2023-04-03 22:30:21 ERROR: Transitions are disabled until the above configuration errors are fixed

This can be fixed in app configuration by disabling the muting transition. There are no muting forms for this project.

ERROR: Transitions are disabled until the above configuration errors are fixed

Is sentinel doing nothing in this state? What is the purpose of having sentinel run but do nothing like this? The log output makes things look pretty healthy. Would a better pattern be to fail fast? Or could we at least re-print this error each 5 minutes so this isn't a needle in a haystack?

[dianabarsan]

Is sentinel doing nothing in this state?

This is correct, Sentinel is doing no transition processing in this state. The backlog value is true.
The scheduler log that @mrjones-plip linked is related to a completely separate part of Sentinel, which is working correctly even with invalid transition config.

[dianabarsan]

Or could we at least re-print this error each 5 minutes so this isn't a needle in a haystack?

I think this is a great suggestion. Having a way to quickly check whether sentinel is indeed processing transitions over documents is very important.

My preferred solution for this is monitoring service logs, which is on the Infrastructure Focus Group OKRs as a goal for the near future.

[kennsippell]

Another option is a boolean flag for is sentinel processing docs in monitoring api?

monitoring service logs, which is on the Infrastructure Focus Group OKRs as a goal for the near future

Are there more details on what this mean? Even high level vision would be helpful.

[dianabarsan]

boolean flag for is sentinel processing docs in monitoring api?

That works too. Adding a health check to Sentinel has been something we've discussed several times in the past. Right now, Sentinel doesn't communicate its state except for the two "seq" docs, which are already used by the monitoring API.

Are there more details on what this mean?

So far, nothing is settled. High level vision would be that all apps logs (api, sentinel, haproxy, nginx, couchdb, etc) are scanned for errors and other significant entries, with filtered entries being funneled into a digestible dashboard of sorts.

[mrjones-plip]

Thanks for the clean up of title and labels of this ticket @kennsippell - much appreciated!

[kennsippell]

Ready for AT on 8162-logs-errors-when-trans-disabled

To test this - update an app_settings.json with an unknown or crashing transition. Look at sentinel logs and verify there is a descriptive error. Wait 5 minutes and verify the descriptive error is reprinted.

[mrjones-plip]

@kennsippell - to be a bit more explicit for QA on how to test, "unknown or crashing transition" could be achieved with smang in app_settings.json , yeah?

  "transitions": {
    "smang": true
  }

[ngaruko]

How hard would it be to add a small e2e tests to transitions tests @kennsippell ? It would go along way towards our new quality assistance regime.

[ngaruko]

Testing details

Config: Default
Environment: Local
Platform: WebApp
Browser: Chrome

---

Test scenario

1. Add 'unkown transition' in app_settings.tasks.rules as in this comment and upload settings
2. Check error in sentinel logs

Fixed on 8162-logs-errors-when-trans-disabled >> Log in feedback doc

Sentinel logs

---

Test passed successfully. 🎉

[kennsippell]

Merged. Spoke with @ngaruko about the e2e test suggestion. He said he'd look at existing tests and file a followup ticket as needed.