Get of encounters - external API is very slow

[mahalakshme]

Issue:

On doing the below:

zgrep '/api/encounters' chs.log.2024-03-14.*
zgrep '/api/encounters' chs.log.2024-03-13.*

found the below logs:
taking 2 mins when page number is higher:

Lesser time when page no is lower:

But unable to reproduce the issue via POSTman: taking less than 40 secs even when page no is higher

AC:

• Identify rootcause of the issue
  ---- Is it because prod was slow around that time
  ---- similar to sync offset problem - higher page no contributing to slow sync - Reduce offsets when syncing avni-client#524
  ---- or some issue in code - contributing to more response time - generally
  ---- or something else?
• What are the possible solutions for the above identified rootcause

[petmongrels]

notes

• increasing page number is slowing the response to some extent (not true when looking at full numbers)
• using encounter type improves the response times significantly (remains true)
• using page size is improving the response time (check for concept cache hits) - before code change

To try

• We are trying to get all OBS resolved from UUID to string in single db call. Catching here is unlikely to help because the cache-key is unlikely to repeat often. We can trying caching all concepts and then to in-memory lookup in the scope of one external API request.

notes after code change

• response times is better by 1.5 to 2 times is we have calls that hit the cache more reliably that is find by UUID
• most of the time is spend on the server processing and JSON serialisation doesn't take that much time comparatively

response times comparison

https://docs.google.com/spreadsheets/d/1B9gChZnzRkhJhxRurcO-111FKYrs2Ax7A9GMGS2r-O4/edit#gid=0

[AchalaBelokar]

• tested with rwbniti22
• tested with ai@gvam_uat mentioned in the sheet
• tested with maha@calcutta_kids

[petmongrels]

There have no requests from Niti in last month. Checked for other organisations like RWB which was ok. but is not comparable.