Crash when trying to export data #120
Comments
|
The export data function is extremely slow and it looks like it is eating loads of memory - there must be a better way to do it. The relevant code is here: https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/master/app/src/main/java/uk/org/openseizuredetector/ExportDataActivity.java#L268 I think the thing to do is not use LogManager. cursor2json but instead receive the cursor and write the csv file directly to avoid the overhead of turning it into json and parsing back? |
|
This is quite a serious bug as it prevents users obtaining their data easily, so I'll do a quick fix as V4.1.11 rather than waiting for V4.2. |
|
Topic
https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
Is giving handles to it. Android has async callback to give feedback back
to the UI
Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
… This is quite a serious bug as it prevents users obtaining their data
easily, so I'll do a quick fix as V4.1.11 rather than waiting for V4.2.
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***
com>
|
|
Thanks, I'll have a look at that. Ours will be a bit more complicated
because I'd like to parse the dataJSON string into the csv, but hopefully
that solution might avoid the memory consumption problem.
Graham
…On Mon, 21 Aug 2023, 16:14 AroonPro, ***@***.***> wrote:
Topic
https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
Is giving handles to it. Android has async callback to give feedback back
to the UI
Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
> This is quite a serious bug as it prevents users obtaining their data
> easily, so I'll do a quick fix as V4.1.11 rather than waiting for V4.2.
>
> —
> Reply to this email directly, view it on GitHub
> <
#120 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***
> com>
>
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACLSY652NAE5R7EF43A5QDXWN3NZANCNFSM6AAAAAA3YMFKK4>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
|
a4b6a43 should fix the out of memory crash - we only process the data one row at a time rather than all at once. |
|
https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/V4.2.x/app/src/main/java/uk/org/openseizuredetector/LogManager.java#L503
HashMap is recreated inside an active loop. This will only be cleared when
leaving the context.
Op ma 21 aug. 2023 17:32 schreef Graham Jones ***@***.***>:
… Thanks, I'll have a look at that. Ours will be a bit more complicated
because I'd like to parse the dataJSON string into the csv, but hopefully
that solution might avoid the memory consumption problem.
Graham
On Mon, 21 Aug 2023, 16:14 AroonPro, ***@***.***> wrote:
> Topic
>
>
https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
>
> Is giving handles to it. Android has async callback to give feedback
back
> to the UI
>
> Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
>
> > This is quite a serious bug as it prevents users obtaining their data
> > easily, so I'll do a quick fix as V4.1.11 rather than waiting for
V4.2.
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#120 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
>
> > .
> > You are receiving this because you are subscribed to this
thread.Message
> > ID: ***@***.***
> > com>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#120 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AACLSY652NAE5R7EF43A5QDXWN3NZANCNFSM6AAAAAA3YMFKK4>
> .
> You are receiving this because you were assigned.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AV6J745US463XDO66KT2FATXWN5STANCNFSM6AAAAAA3YMFKK4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
The same counts for
https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/V4.2.x/app/src/main/java/uk/org/openseizuredetector/ExportDataActivity.java#L283-L287
Creating variables inside lead to memory build-up. Easy fix
Op di 22 aug. 2023 06:08 schreef Bram Regtien ***@***.***>:
…
https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/V4.2.x/app/src/main/java/uk/org/openseizuredetector/LogManager.java#L503
HashMap is recreated inside an active loop. This will only be cleared when
leaving the context.
Op ma 21 aug. 2023 17:32 schreef Graham Jones ***@***.***>:
> Thanks, I'll have a look at that. Ours will be a bit more complicated
> because I'd like to parse the dataJSON string into the csv, but hopefully
> that solution might avoid the memory consumption problem.
>
> Graham
>
> On Mon, 21 Aug 2023, 16:14 AroonPro, ***@***.***> wrote:
>
> > Topic
> >
> >
> https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
> >
> > Is giving handles to it. Android has async callback to give feedback
> back
> > to the UI
> >
> > Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
> >
> > > This is quite a serious bug as it prevents users obtaining their data
> > > easily, so I'll do a quick fix as V4.1.11 rather than waiting for
> V4.2.
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
> #120 (comment)>,
>
> >
> > > or unsubscribe
> > > <
> >
> https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
>
> >
> > > .
> > > You are receiving this because you are subscribed to this
> thread.Message
> > > ID: ***@***.***
> > > com>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
> #120 (comment)>,
>
> > or unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AACLSY652NAE5R7EF43A5QDXWN3NZANCNFSM6AAAAAA3YMFKK4>
>
> > .
> > You are receiving this because you were assigned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <#120 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AV6J745US463XDO66KT2FATXWN5STANCNFSM6AAAAAA3YMFKK4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
|
|
Ah, that was probably it then. I get too careless with Java by trusting
the garbage collector. If it was C I'd be much more careful!
I don't use that now- we go straight from the DB cursor to the csv file:
master...V4.1.x#diff-e10be5e2def6ca2fd5f4ad55b8a98b82485ad04706f9531644a0c0b6cc831b89R495
…On Tue, 22 Aug 2023, 05:08 AroonPro, ***@***.***> wrote:
https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/V4.2.x/app/src/main/java/uk/org/openseizuredetector/LogManager.java#L503
HashMap is recreated inside an active loop. This will only be cleared when
leaving the context.
Op ma 21 aug. 2023 17:32 schreef Graham Jones ***@***.***>:
> Thanks, I'll have a look at that. Ours will be a bit more complicated
> because I'd like to parse the dataJSON string into the csv, but
hopefully
> that solution might avoid the memory consumption problem.
>
> Graham
>
> On Mon, 21 Aug 2023, 16:14 AroonPro, ***@***.***> wrote:
>
> > Topic
> >
> >
>
https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
> >
> > Is giving handles to it. Android has async callback to give feedback
> back
> > to the UI
> >
> > Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
> >
> > > This is quite a serious bug as it prevents users obtaining their
data
> > > easily, so I'll do a quick fix as V4.1.11 rather than waiting for
> V4.2.
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
>
#120 (comment)>,
>
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
>
> >
> > > .
> > > You are receiving this because you are subscribed to this
> thread.Message
> > > ID: ***@***.***
> > > com>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#120 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AACLSY652NAE5R7EF43A5QDXWN3NZANCNFSM6AAAAAA3YMFKK4>
>
> > .
> > You are receiving this because you were assigned.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#120 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AV6J745US463XDO66KT2FATXWN5STANCNFSM6AAAAAA3YMFKK4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACLSY52CMDUZSUFNOF2723XWQWD3ANCNFSM6AAAAAA3YMFKK4>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***
com>
|
|
Graham,
That looks great.
Can you also take care of timeconversions?
master...V4.1.x#diff-e10be5e2def6ca2fd5f4ad55b8a98b82485ad04706f9531644a0c0b6cc831b89R495
We can use my new osdutil.convertTimeUnit
Op di 22 aug. 2023 08:18 schreef Graham Jones ***@***.***>:
… Ah, that was probably it then. I get too careless with Java by trusting
the garbage collector. If it was C I'd be much more careful!
I don't use that now- we go straight from the DB cursor to the csv file:
master...V4.1.x#diff-e10be5e2def6ca2fd5f4ad55b8a98b82485ad04706f9531644a0c0b6cc831b89R495
On Tue, 22 Aug 2023, 05:08 AroonPro, ***@***.***> wrote:
>
>
https://github.com/OpenSeizureDetector/Android_Pebble_SD/blob/V4.2.x/app/src/main/java/uk/org/openseizuredetector/LogManager.java#L503
>
> HashMap is recreated inside an active loop. This will only be cleared
when
> leaving the context.
>
> Op ma 21 aug. 2023 17:32 schreef Graham Jones ***@***.***>:
>
> > Thanks, I'll have a look at that. Ours will be a bit more complicated
> > because I'd like to parse the dataJSON string into the csv, but
> hopefully
> > that solution might avoid the memory consumption problem.
> >
> > Graham
> >
> > On Mon, 21 Aug 2023, 16:14 AroonPro, ***@***.***> wrote:
> >
> > > Topic
> > >
> > >
> >
>
https://stackoverflow.com/questions/31367270/exporting-sqlite-database-to-csv-file-in-android
> > >
> > > Is giving handles to it. Android has async callback to give feedback
> > back
> > > to the UI
> > >
> > > Op ma 21 aug. 2023 15:27 schreef Graham Jones ***@***.***>:
> > >
> > > > This is quite a serious bug as it prevents users obtaining their
> data
> > > > easily, so I'll do a quick fix as V4.1.11 rather than waiting for
> > V4.2.
> > > >
> > > > —
> > > > Reply to this email directly, view it on GitHub
> > > > <
> > >
> >
>
#120 (comment)>,
>
> >
> > >
> > > > or unsubscribe
> > > > <
> > >
> >
>
https://github.com/notifications/unsubscribe-auth/AV6J745443LP63P2PKVP3TTXWNO37ANCNFSM6AAAAAA3YMFKK4>
>
> >
> > >
> > > > .
> > > > You are receiving this because you are subscribed to this
> > thread.Message
> > > > ID: ***@***.***
> > > > com>
> > > >
> > >
> > > —
> > > Reply to this email directly, view it on GitHub
> > > <
> >
>
#120 (comment)>,
>
> >
> > > or unsubscribe
> > > <
> >
>
https://github.com/notifications/unsubscribe-auth/AACLSY652NAE5R7EF43A5QDXWN3NZANCNFSM6AAAAAA3YMFKK4>
>
> >
> > > .
> > > You are receiving this because you were assigned.Message ID:
> > > ***@***.***>
> > >
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#120 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/AV6J745US463XDO66KT2FATXWN5STANCNFSM6AAAAAA3YMFKK4>
>
> > .
> > You are receiving this because you commented.Message ID:
> > ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#120 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AACLSY52CMDUZSUFNOF2723XWQWD3ANCNFSM6AAAAAA3YMFKK4>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***
> com>
>
—
Reply to this email directly, view it on GitHub
<#120 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AV6J7437DKJWH3COJDMO26TXWRFKHANCNFSM6AAAAAA3YMFKK4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using the data export function to export 12 hours of data resulted in the following crash. It works ok for 2 hours of data.
***** DEVICE INFO
Brand: samsung
Device: j5y17lte
Model: SM-J530F
Manufacturer: samsung
Product: j5y17ltexx
SDK: 28
Release: 9
***** APP INFO
Version: 4.1.10
Installed On: 2022-04-14 16:19:04
Updated On: 2023-07-26 07:56:13
Current Date: 2023-08-21 08:53:57
***** ERROR LOG
java.lang.OutOfMemoryError: Failed to allocate a 150994952 byte allocation with 25165824 free bytes and 61MB until OOM, max allowed footprint 162135992, growth limit 201326592
at java.util.Arrays.copyOf(Arrays.java:3260)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:660)
at java.lang.StringBuilder.append(StringBuilder.java:203)
at org.json.JSONStringer.string(JSONStringer.java:344)
at org.json.JSONStringer.value(JSONStringer.java:252)
at org.json.JSONObject.writeTo(JSONObject.java:723)
at org.json.JSONStringer.value(JSONStringer.java:237)
at org.json.JSONArray.writeTo(JSONArray.java:613)
at org.json.JSONArray.toString(JSONArray.java:585)
at uk.org.openseizuredetector.LogManager.cursor2Json(LogManager.java:220)
at uk.org.openseizuredetector.LogManager.lambda$getDatapointsByDate$0$uk-org-openseizuredetector-LogManager(LogManager.java:473)
at uk.org.openseizuredetector.LogManager$$ExternalSyntheticLambda5.accept(Unknown Source:4)
at uk.org.openseizuredetector.LogManager$SelectQueryTask.onPostExecute(LogManager.java:754)
at uk.org.openseizuredetector.LogManager$SelectQueryTask.onPostExecute(LogManager.java:701)
at android.os.AsyncTask.finish(AsyncTask.java:695)
at android.os.AsyncTask.access$600(AsyncTask.java:180)
at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:712)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:216)
at android.app.ActivityThread.main(ActivityThread.java:7266)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:494)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:975)
The text was updated successfully, but these errors were encountered: