form submission data optimization

[raihan-mpower]

optimize the amount of data being pulled from server in login , right now it pulls all data from server and process the form submission.

[ndisha]

@raihan-mpower @dimasciput @koros
@maimoonak detailed the different approaches that we have for synching in OpenSRP and specifically for the data optimization on first time log in, she wrote:

Every device to server sync includes download of all FormSubmissions that do not exist on the device for that HealthWorker from server. If this is first login or new device it leads to unlimited download with each and every data that may never be needed on mobile-client.

*Problem: * There are three different potential problems in this approach
1- Bulk data download from start of program to date with every minute detail
2- The data on download goes under form files loading and reading, JSON parsing, table creations and data push in multiple tables i.e. avoidable heavy IO operations. The approach is inherent into app and trying changing the process would lead to drastic changes in mobile-client code
3- HealthWorker has access to his/her beneficiaries only

*Solution: *
1- As @mberg suggested that the approach would be changed and in future mobile-client would sync data directly from server (sqlite to couch). This would be supported by the change suggested in an earlier point where we would have all data as Client, and Event and Obs (standard, central unlikely to change model). This would allow mobile-client send a filtered request for data download. The filter could be:

all FormSubmissions for forms specified (do we need it)
all FormSubmissions for HealthWorker specified HealthWorkers in specified date range (do we need it)
all Clients only
all Clients filtered by event types
all Clients registered in specified date range
all Events filtered by event types
all Events in specified date range
all FormSubmission or Events of specified Location (do we need it)
Events would optionally have Obs data
2- The change would lead to massive refactoring on mobile-client and @raihan-mpower and @dimasciput can better comment on it, but from server side all above filters would be available and mobile-client can then go into this transition smoothly. New FormSubmissions on mobile-client would submit data to server as it is right now. Server would create Client and Events out of this and any other fresh device can download data as filtered Client and Event or even FormSubmissions.

3- The problem would be solved by using the approach above as HealthWorker would be able to download data of his/her own choice using different filters.

HealthWorker would also have a search and claim Client feature where he/she would able to search Clients/Events based on identifiers, some attributes and locations (@raihan-mpower and @dimasciput).
Data privacy needs to be discussed on how much details to be allowed sent to mobile-client based on HealthWorker Role and Location.
This would also allow external apps to sync and query data by following a standard, simple and unlikely to change Documentation.

@raihan-mpower @dimasciput kindly weigh in on this and let us know which approach would best work, could we maybe have a pre-configured filter that loads only specific data (e.g. data for the ANMs location) or we could have the user be given the option to configure the data to download on log in (still with an option to skip) and configure later, whichever works for her?

[sohelsarder]

3- HealthWorker has access to his/her beneficiaries only is not potential problem, that data hiding is some projects strict requirement also, so that issue we need to keep in consider while planning for added more flexibilities over it. I think the use case is for some projects don't want to share one providers data to another but for other cases that sharing might require.