Large number of submissions results in too many open files error #518
Labels
Comments
yanokwa
added
bug
good first issue
|
This could be as simple as closing the response object in a finally block. |
|
@yanokwa I would like to fix it !! |
|
@lakshyagupta21 Please go ahead. |
|
@yanokwa I tried running app for the above odk directory but didn't get anything related to above error. And all my files are getting uploaded but somewhere in last i am getting sql error in uploadingComplete method in InstanceUploaderActivity. I request you to please check one more time by reinstalling the app and replacing the previous odk directory by the above directory. I am using Marshmallow for testing , will you please tell me about the device on which you are testing. I will try in that device too. Is there any limit on maximum request that we can make to https://nafundi-test.appspot.com/ because i wasn't able to access this link yesterday. |
|
If i see about response code in logcat i am getting |
|
@lakshyagupta21 I've added more quota to https://nafundi-test.appspot.com so it shouldn't go down. First, file an issue with the "Expression tree is too large" bug so we have a record. Then please try to reproduce the issue on a Pixel running API 25 in the Android emulator. |
|
@lakshyagupta21 I've also added a new ODK folder in the original post that doesn't have the unique instanceID. When you submit, you should now see multiple submissions on the server. Be sure to delete the test form (and thus the submissions) and wait a few minutes between each run. |
|
@yanokwa There is one more point that i want to discuss. Actually now a days service is being used for all the backend related things like file upload, file download, etc. So what if we write services for uploading files as when user tries to upload any number of files he has to wait till all his files first gets uploaded on server, then only he is allowed to move to other activities of Application. I mean there is no functionality to run the uploading process to go in background so that user can fill,edit or send other forms too. While testing it took me around 30 minutes - 1 hour to upload all 2000 files. And while uploading say 1998 number of files the dialog box is showing status like this "1 out of 999 files" and after completing all 999 it again starts with 1 for the rest of 999 files. The expected behaviour should be like "1 out of 1998". |
|
@lakshyagupta21 ah that status is a bug. I think it's publishing based on the query number, not the current position of the uploading process. I missed that one when I update the uploader to allow more than 1000 instances. |
|
I think you can create a separate issue for that one and create a PR to fix that status counter. Regarding the background service to do the upload, I think it would be a nice refactor to do the uploading asynchronously (and should go into separate issue as well). |
|
@lakshyagupta21 i created a separate bug for the dialog box issue #585. |
|
@nribeka Thanks !! |
|
@yanokwa I created the emulator having PIXEL running API 25 and tried running the app on it. While running it gets stuck somewhere in the middle of uploading files. It doesn't show anything in logcat also i tried it 3-4 times but couldn't produce the above error. I tried in Nexus 5 running API 23 in that it was working fine, what should i do ? |
|
@nribeka I need help in this issue. |
|
Hi @lakshyagupta21, I'm sorry about the delay on this issue! I've tested this on a Pixel emulator running API 25 and for 1998 files, it fails at 1466. Given that this is so painful to test on a slow connection, I think @nribeka should give it go. And to make the test go faster, @nribeka, I'd recommend using a local VM instead of running it against App Engine. https://github.com/nafundi/odk-aggregate-box makes it easy to get Aggregate up and running. |
|
@yanokwa will look at this 👍 |
|
I don't know if you're seeing this @yanokwa, but i'm getting a lot of exception when uploading. |
|
It might be an issue with you guys killing my App Engine instance! Try with a local box: https://github.com/nafundi/odk-aggregate-box |
|
I'm in locked network, so can't use my phone. Tried using vagrant, and the emulator won't like the virtualbox running in the same instance. Have you run into this issue? This exception is different though. So I looked into the code. When ResponseMessageParser gets created, it's also reading the response entity as a whole and then close the stream. But the following line of code will try to read this closed stream. |
|
I used to have the problem of Virtualbox and the emulator not running simultaneously. The solution is to upgrade to the latest version of HAXM. I think 6.1.1 is when they fixed the problem. Android Studio doesn't make it easy for you to upgrade HAXM, you might have to install it manually. |
|
Tested locally and 9k + 2k instances were uploaded successfully. Maybe we need to close this? I notice there are 2 issues that we need to address from this testing:
|
|
@nribeka Can you please file an issue on the observer in MainMenuActivity? |
|
I think there is a bug and I can repro it on my Pixel emulator running API 25 and submitting to nafundi-test. I do not get the behavior when submitting to a local server. I do not get it with a Nexus 5 emulator running API 22 and submitting to nafundi-test. What emulator did you use, @nribeka? Run 1 Run 2 |
yanokwa
added
needs reproduction
and removed
good first issue
|
Testing on a Moto G4 with Android 7.0 submitting to nafundi-test also gets errors. |
|
Hello @nribeka, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 10 days. You can reclaim this issue or claim any other issue by commenting Thanks for your contributions, and hope to see you again soon! |
|
It's the same issue: #2626 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If you try to submit, say 1998 files, ODK Collect v1.4.16 will crash with the following stack trace.
If you need a test server, use https://nafundi-test.appspot.com. If you need 2000 submissions, use this odk folder.
While testing, you might need to delete the form from the server (it's the best way to delete all submissions). You can find the same blank form from
/odk/forms/basic.xml.The text was updated successfully, but these errors were encountered: