No sensor data is being collected when using a self-hosted server

[avaidyam]

Collaborator: MindLAMP 2 (Google Play Store Version 1.0). I'm also seeing this behavior when building from the Android source code. I self-deployed MindLAMP and have set up both server and database. When logging in as a participant on my Pixel 3A (Android 11), I can see the notification pop up that sensor data collection is happening from time to time, but when I request sensor events for that participant through the API, I get a list that only includes LAMP dashboard analytics events.

@amaljofyzco Do you know if this/related sensor issues are fixed in v1.1 since you pushed that last week?

---

This issue has been reported multiple times across both iOS and Android devices.

[amaljofyzco]

Adithya, I hope Collaborator is not using the play-store - beta build. Now beta version is 1.1.

[avaidyam]

Does the beta fix these issues? It works on my device but the sensor collection service notification disappears after some time. Does that mean there is no data being collected?

[amaljofyzco]

Aditya, Can you share the device information. We are not facing such issues, if possible I will share a build will log file so we can track it.

[avaidyam]

The collaborator is running his own separate infrastructure so I do not have access to that info on his end. However, I am testing on both a Pixel 2 XL (Android 10) and a Pixel 4 XL (Android 11).

[amaljofyzco]

Ok Adithya, we haven't tested in android 11. I can share a test build with log file enabled is that okay.

[avaidyam]

Yes, that's fine. A few questions, though:

1. How would you need to receive the log file from the device?
2. And is there a way to automatically share the logs by default for all devices (i.e. no special builds for diagnostics purposes)?

[amaljofyzco]

1 - Goto internal-storage | Android | data | digital.lamp.mindlamp | files | LampLog.txt
2 - Yes, for now we can enable this.

[avaidyam]

@amaljofyzco If you enable it, though, it should not impact the performance of the app or sensor collection. We should find a way to always save logs with confidential information stripped out and remotely request them for diagnostic purposes. Do you have any ideas on an implementation for this?

[michaelmenon]

@avaidyam Can we update beta to production and test that and see if we get these issues ? We don't want to spend time on it if its a build issue. In order to avoid confusion it would be better if we can just update the production with the current beta and test it again. As beta has fixes data collection which is not there in production.

[avaidyam]

It's already set to v1.1 and the issues I am reporting come from v1.1 as well.

Furthermore, the collaborator writes:

I'm also seeing this behavior when building from the Android source code.

So I think this is an ongoing issue if he's seeing it from directly building the repository too.

[michaelmenon]

Hi @amaljofyzco can you confirm the 1.1 is up to date with what is updated to beta ?

[michaelmenon]

@amaljofyzco there is a way to send log files to developer account from a separate thread in Android. We can implement that and update the build asap. I have shared the info with you in chat.

[michaelmenon]

@amaljofyzco suggested we can integrate firebase analytics to collect the logs from the app. We will update the build with that integrated asap to figure out what's going on with Android 11

[michaelmenon]

why was it closed :)

[avaidyam]

Let's keep this issue open until that integration is complete and the collaborator reports to us that the issue is fixed on their end.

[amaljofyzco]

Aditya, could you add me as a member in your firebase account (lampv2test@gmail.com). So that I can access it for setting up the analytics.

[amaljofyzco]

why was it closed :)

why was it closed :)

sorry by mistake

[avaidyam]

@amaljofyzco I've added you to the Firebase account temporarily - let me know what changes you've made when you're done so we can document them too!

[amaljofyzco]

Aditya, I have enabled the Crashlytics and Firebase Events. I will be check-in the build.
But I can't see the analytics because of permission.

[avaidyam]

@amaljofyzco Can you remind me why we are enabling analytics (this requires enabling Google Analytics) for the app? I thought Crashalytics was what we wanted?

[amaljofy]

Adithya, I also added the event-logs in the app. Like when we are sending the sensor-data we will post an event.
I have added the package for analytics and added in the code.

Crashlytics also implemented, I will be updating the build after testing.

Thanks

[avaidyam]

Can you clarify exactly which new event logs you have added?

[amaljofyzco]

Adithya, The event - logs added are follows.

• Service_Started
• Service_Stopped
• API_Send_${sensorEventDataList.size}
• Fit_Success //this event fire when google-fit auth success.
• Fit_ERROR //this event fire when google-fit auth failed.

Crashlytics also enabled.

[avaidyam]

By event logs are you suggesting these are SensorEvent objects? Or are you sending them to the logging gateway?

[amaljofyzco]

No SensorEvent object is passing to logs. Only the size of sensor object list is passing as a string.

[amaljofyzco]

I have updated my pixel 3xl to Android 11, a small Fix done inside our framework code, which doesn't support in 11OS.

[avaidyam]

@amaljofy So the items you mentioned are being logged, not uploaded to the LAMP API? If so, that's helpful - thanks!

[amaljofyzco]

Correct, I will be pushing to beta shortly.

[avaidyam]

That's fantastic - thanks @amaljofy! I'll have my team also use the beta and investigate issues on our end, and if we find anything we'll respond here.

[amaljofyzco]

Code check-in done. App updated to playstore-beta, now its Being reviewed.

[avaidyam]

@amaljofy @michaelmenon Another recent report:

Collaborator:
We do not appear to be receiving sensor data from any iOS devices. I am
using the python version of the API to query our database, and whilst I
can see sensor data coming from the Android devices, the only data I
have from the iOS devices is the login information to the dashboard, and
their active data from the surveys. I've attached a screenshot of the
data I can see coming through from 1 of our 3 connected iOS devices,
when I use the LAMP.SensorEvent.all_by_particpant(‘id’) method.
I can see on our dashboard, that these 3 devices all have their devices
connected, and have recently accessed the dashboard, although it is also
saying no data has ever been received from their device.
I have checked with them that they have given LAMP access to all the
sensors on their device, and got them to send screen shots of their
settings on their phone to confirm. And I have also checked with them
that they had not exited LAMP, and that it was still running in the
background.

I think this is turning into a very urgent issue that we need to resolve quickly.

[jijopulikkottil]

"We do not appear to be receiving sensor data from any iOS devices"

We will test the app-store version with iOS 14 device and confirm. Hope the settings are like below:

Edit:
Downloaded app from app-store on iOS14 device and is working.
Could you please ask to do like:

1. Remove app from background (force quit the app)
2. Open the app again and check its showing Login screen or Home screen!

[avaidyam]

@jijopulikkottil Let me invite the collaborator to respond here! //cc @nerfanian @zone86

[michaelmenon]

@avaidyam did we get any reply from the collaborator on the iOS issue they reported.. Also were you able to test the Android beta which @amaljofy pushed.

[avaidyam]

@jijopulikkottil @michaelmenon I’ve just shared a collaborator’s iOS device logs privately with you on Basecamp. Please take a look and let me know if that issue has been fixed.

[jijopulikkottil]

We are doing some optimisations on our code.. its still testing mode. Once we pushed the code, hope we can provide them the testflight beta build to test sensor data syncing issue.
Hope I can push the code within two days.

[avaidyam]

@kkadapa Were you noticing this issue on iOS or Android or both?

[kkadapa]

Hi @avaidyam I see this issue currently only for iOS as mentioned here https://mindlamp.discourse.group/t/question-regarding-the-couchdb-step-count-bug/262

[kkadapa]

Hi @avaidyam I installed TestFlight version 2021.1.21(237) and allowed to sync my Steps but I still don't see Steps data in Prevent tab as you can see below, please let me know if you need any additional details

[avaidyam]

@kkadapa You will not be able to see raw step count data in the dashboard; you have to use the API to access it for now.

[avaidyam]

@kkadapa I've investigated this issue further to ensure we're able to solve the problem. After logging in to the latest production release of the mindLAMP app on an iPhone 6s using the credentials you specified in our email conversation, I performed a man-in-the-middle attack to intercept and inspect the incoming and outgoing data between the app and your self-hosted server.

This issue does not seem to be with the app or the server code itself. You/your organization will need to reconfigure the load balancer currently serving the API Server as it currently has a maximum payload size limit that is far too low (< 1 MB). All requests from the app are being returned with a 413 Payload Too Large HTTP error. To confirm this, I extracted the relevant API call and generated fake data for testing purposes.

Below is the command I executed to confirm this issue (replace ID with the correct Participant ID):

curl \
  -X POST \
  https://mindlampdev.eastus2.cloudapp.azure.com/participant/ID/sensor_event \
  -H 'Authorization: Basic ID@lamp.com:ID' \
  -H 'Content-Type: application/json' \
  -d @fakedata.txt

Below is the fake data I generated and used:
fakedata.txt

The response from your load balancer was:

<html>                                                            
<head><title>413 Request Entity Too Large</title></head>          
<body bgcolor="white">                                            
<center><h1>413 Request Entity Too Large</h1></center>            
<hr><center>nginx/1.14.0 (Ubuntu)</center>                        
</body>                                                           
</html>  

---

@jijopulikkottil @amaljofy @michaelmenon
One issue I was able to identify during this process is that the lamp.accelerometer.motion sensor only produces a key magnetic (with x, y, and z) instead of the 4 other ones required, but that will be discussed as part of a different issue.