Opening new My Tracks very slow with larger number of tracks

[pebogufi]

Description

With new My tracks my phone S10 is very slow, but not my Tab with Android 10.
I use two devices, which are nearly complete identicaly in OsmAnd folders and files.
+Samsung S 10, Android 12, installed on internal, Android/media
+Samsung Tab S5e, Android 10, installed on Sd card, Android/obb
The S10 is normally much faster, example: install of osmand
S10 needs 5-8 sec,
Tab needs more than 20 sec.

BUT opening my tracks needs (total files in tracks subfolders ~500)
S10 5-7 sec
Tab ~1 sec.
I suppose the problem is the file access restrictions in Android 12 which uses extended file acces when creating statistics (tracks, distance, uphill, downhill, duration, size) which are at bottom line in tracks page.

proposed change

Please add an option/checkbox to decide whether this automatic calculation of statstics should be done. I think this will not only help a lot of users with new Android versions > 10, but make them really happy !

Problem 2
Opening My tracks / "Visible on map" with S10, Android 12 needs so much time, that i often get a message from system, whether to continue or abort the app.
With Tab it is also very slow, 5-10 sec, but no system warning. Visible files with both devices max 5, total files in tracks subfolders ~500.

Environment
OsmAnd Version: OsmAnd~ 4.6.0#537m, veröffentlicht: 2023-08-02
Android version: 12
Device model: Samsung S 10

[pebogufi]

With S10:
After removing all but 50 files in tracks subfolders and no tracks visible the function My tracks / "Visible on map" needs 3-5 sec to show only one line "currently recording track" while recording is off.

[sonora]

It's already been noted a number of times, e.g. here #17257, and I believe often before the statistics display was added. Also, I see the same effect under Android 11, even when I use a build which targets Android 10, and/or using the "All files access" Android permission, which circumvents the new file access mechanisms.

So chances are the root cause for this may be in how we do things different in our code now vs. the previous (collapsable list) MyPlaces>Tracks screen (which was comparably instantaneous), and not necessarily in the stat calculation or in the new google storage access limitations...

[pebogufi]

17257 -> vshcherb closed this as completed on May 29.
:-(

[Corwin-Kh]

This issue should be checked on latest build as it could be already fixed. Aside from this issue, there was another one that is related to SmartFolders and affected opening My places -> tracks. It is fixed in #2159

[pebogufi]

tracks. It is fixed in #2159

Link reports error 404, not found

After using again my folders with 767 gpx files
opening track folder needs 10 sec, that is no improvement to the initial report.
Tested with S10 (as used above), OsmAnd~ 4.6.0#930m, veröffentlicht: 2023-10-27

[Corwin-Kh]

sorry for link - turn put it's closed. 1994f23
this one should work. The case i'm talking about is fixed in this pull request. It is already in last builds.
So, if it is still present then problem somewhere else. we'll try to find out.

[sonora]

Unfortunately, I have the same experience, there is no improvement with latest builds. It is still very slow to open either the My Tracks or the Visible Tracks screen. Something we do there seems to be extraordinarily resource intensive. (Interestingly, while it is always slow, times can vary from 3 to 15 sec, Android 11, ~3500 tracks in the folder).

What's worse is #18077, which is a showstopper for me using our 4.6.x releases at all, I am forced to stay with 4.5 for all my work.

[vshcherb]

Refactoring is needed with GPX Database.

1. We need to generate all long SQL queries from Enum of GPX characterestics and GPX setting.
2. We shouldn't read file system list and only get track items GPXDatabase. GPXDatabase should be updated in background first time My Places open or Configure Map > Tracks. On Start we should just read GPXDatabase.
3. Filters code should be simplified as we shouldn't have separate classes for filters - they don't require any specific functions. Instead we should have Conversion function (1 per unit), GPXDatabase should provide generic API to retrieve min/max field from database.
4. GPXDatabase should have automatic regeneration code for characteristic when new characterestic is added or when schema is changed. For that purpose we can store timestamp of generation. Scheme will be defined simply as date (in Android there is API to check existing column).

[sonora]

I wonder how we did it with the old, expandable list view. That one was comparatively lightning- fast...

[sonora]

Fun fact: When you create a smart folder with filter condition "Visible on map", that one opens essentially instantaneously. But opening the built-in "Visible tracks" screen creates the long wait, every time.

[pebogufi]

@sonora VERY fun fact!
Thanks a lot for that hint, it is great.
I can confirm, that it opens instantaneously and
build in quite slow every time.

[sonora]

I tested a new build with the GPX db mprovement merge of 2 weeks ago. The effect was that upon the first app start I received a black, empty map screen with no visible activity, I believe for 2, perhaps 3 minutes. Knowing that the GPX database was probably being updated, I waited things out, and eventually the map appeared and the app functioned normally.

Then testing MyPlaces/Tracks and Visible Tracks, there is no decisive speed-up. It still takes about 12 seconds watching the spinner every time I open one of the 2 screens, or go from one to the other.

So also this supports my observations that we did not introduce this bug by adding the statistics functionality (bug was here definitely before that), and perhaps anything else which can be fixed in the db, but we added it when we changed from the prior collapsible view to the new paged view.

So it looks like we still need an analysis which process or lookup we call every time when opening one of the 2 screens and which scales so badly with increasing number of tracks on the device?

(Android 11, OsmAnd build targeting Android 10 and having 'All files access' permission, so no file access limitation in play for this test at all.)

[sonora]

PS: When I start the app specifically to check out a track, i.e. when after a fresh app start and immediately after the map becomes visible (but presumably the app is still busy with a lot of other initialization tasks), I pull up the My Tracks screen, that takes up to 40 seconds of watching the spinner.

[Chumva]

Work is still in progress, probably this week there will be UI fixes that should improve speed with my places - tracks screen

[sonora]

@vshcherb Here are some benchmark figures for the time it takes to display the Tracks screen (or again for the Visible tracks screen):

• 3500 tracks: 12 - 14 sec
• 2500 tracks: 7 - 8 sec
• 1500 tracls: 3 sec

So the time to display the screen seems to grow overproportionately with the number of total tracks present in OsmAnd's track folder. This is e.g. typical of sorting mechanisms. But in any case the code strategy seems flawed in the following aspects:

(1) The time to open the Tracks screen should not depend on the total number of tracks in all folders, but merely on the number of items to be shown on the screen to be opened. (In my case simply 70 subfolders and zero tracks on the top level screen)

(2) Similarly, there should not have to be another wait time (and every time) when going from the MyPlaces/Tracks screen to the "Visible Tracks" screen. It seems that latest at this time there should already be a table of all visible tracks present in memory, rather than going through the long wait time again (it seems like the complete flat list of all tracks is searched on every occasion?)

(3) I suggest to disable the manual pull-down list-refresh functionality on these screens. The list of tracks should not change while users are on these screens, so a pull-down refresh makes no sense (but causes 12 sec delays for me time and time again when I scroll to the top of the list).

Initially I thought we may simply have to introduce some caching here of whatever the code produces during these long wait times, so they are not needed upon every (even repeated) opening of the screens. But now I am fairly convinced the code strategy is flawed: Whatever the app keeps doing on these occasions should not be needed via such costly operations at all, for the simple purpose of producing a mostly rather short inventory list on a dialog.

-> I would appreciate if whoever implemented the "New Tracks screen" code could take a look at these findings, and at least explain what the code does during these wait periods, and why that would be required upon every opening of the screen or changing between them?

[sonora]

@Chumva I believe that this is the culprit:

OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/TrackFoldersHelper.java

Line 124 in a763ecc

public void reloadTracks() {

and it leads to this

OsmAnd/OsmAnd/src/net/osmand/plus/configmap/tracks/TrackFolderLoaderTask.java

Line 82 in a763ecc

File[] files = GPXFolderUtils.listFilesSorted(sortByMode, folderFile);

where the problem may be: We always sort the entire list of all files recursively (regardless of which folder they are in), which is a very costly operation, and probably totally unnecessary? Because almost never do we display the entire flat list, so the sorting should not be applied in the beginning to everything when cataloging all file/folder items, but in the end and only to the much fewer files / folders which are to be shown on the screen's current view?

[sonora]

PS: A considerable portion of the wait time (about half) is spent for this line, even if I have no Smart Folders created;

OsmAnd/OsmAnd/src/net/osmand/plus/configmap/tracks/TrackFolderLoaderTask.java

Line 95 in 5ca0048

smartFolderHelper.addTrackItemToSmartFolder(item);

[sonora]

This obvious fix at least brings some noticeable improvement whenever no smart folders are present: #19228.

In my setup this PR speeds things up from 12-14 sec to 5-9 sec. There is more potential, but merging this is a good start.

[sonora]

More low hanging fruits:

• Reloading all tracks when on the MyPlaces/Tracks screen tapping"Visible on map" seems obsolete

• And also: Isn't this sorting code https://github.com/osmandapp/OsmAnd/blob/5ca0048428c1d16821eae8b089fde852265fdecb/OsmAnd/src/net/osmand/plus/track/helpers/GPXFolderUtils.java#L30C1-L45C2 superseded by TracksAdapter.java? (At least for the purpose of populating the new Tracks screen, but we seem to execute it there.)

[vshcherb]

(1) The time to open the Tracks screen should not depend on the total number of tracks in all folders, but merely on the number of items to be shown on the screen to be opened. (In my case simply 70 subfolders and zero tracks on the top level screen)

It's not correct ... Time depends on the number of tracks that's totally fine. Loading & sorting each time your open a screen is fine as well to create a structure especially on new load.

What's not fine that 3500 is a very small number! Loading 3500 records from db and sorting 3500 numbers is about 0.1 seconds should be.

270 t/s - 3500 tracks: 12 - 14 sec
300 t/s - 2500 tracks: 7 - 8 sec
~500 t/s - 1500 tracks: 3 sec

It looks totally fine that it's linear dependency but 0.2 millisec per track is too much. Did you try release versions ? https://download.osmand.net/releases/net.osmand-4.7.2.apk

[sonora]

@vshcherb

• Can we please merge this PR Speed up tracks screen when no Smart Folders are used (partial fix for #17805) #19228? It makes sense in any case, speeds things up by ~50% (if you don#t have a smart folder), saves CPU cycles and battery power.
• Sorting every time: Well, the sorting is important for what you see on the screen, not what's in the background list... 😉 I almost never look at the "All" list, the only one which would require sorting of all 3500 line items.
• Time stamps: see e.g this log file (this already includes the improvements by Speed up tracks screen when no Smart Folders are used (partial fix for #17805) #19228?): logcat.log.txt. It implies like 4 seconds, e.g.

03-03 18:13:52.584   810  2715 I net.osmand: TrackFolderLoaderTask Start loading tracks in tracks {pool-20-thread-1}
03-03 18:13:56.668   810  2715 I net.osmand: TrackFolderLoaderTask Finished loading tracks. took 4085ms {pool--20-thread-1}

But the actual time I am waiting for the list to appear on screens seems even longer (often like twice that), not sure where the extra time is spent. (PS: I find that using a pull-down refresh on the Tracks screen often is quicker than when I re-enter the screen from the menu, also that seems weird.) an, of course, as stated above, I believe there is no need to again re-load the very same the tracks list when changing to the "Visible on maps" screen.

And yes, you are right, reading a file list table of 3500 files should be peanuts, that's what I am saying all along ... ;)

[vshcherb]

1. That's up to @Chumva I think

Tonight (or later this week) we decided finally switch nightly builds to release builds. We just tested that startup time is 5 times faster on release builds

[sonora]

@Chumva Some more code questions:

• OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/filters/SmartFolderHelper.kt

  Line 248 in 39185d4

  return HashSet(allAvailableTrackItems)

  • Should simply return allAvailableTrackItems, it already is class HashSet?

• Remove

  OsmAnd/OsmAnd/src/net/osmand/plus/myplaces/tracks/filters/SmartFolderHelper.kt

  Line 37 in 39185d4

  private val LOG = PlatformUtil.getLog(SmartFolderHelper::class.java)

  and the 3 occurrences of LOG.Debug, this speeds things up considerably (can even back out my interim fix then)

[sonora]

@Chumva I believe we can now merge #19239 , unless you see anything I overlook. This beats the previous fix, the time needed now dropped to 4 - 6 sec for my setup, and I see no drawback. For up to ~1500 tracks I am now at 1 sec.

It looks like most of the now remaining time is used on reading the GPX database and creating the stats.

[vshcherb]

Today was new build released that's not debug and signed as release apk... So everything needs to be reinstalled actually. I tested it still doesn't work as fast as release but this one became 20-50% faster.
So I think to let you test the new builds and close the issue. Actually the only extra test we could do again is to load 3000 tracks as @sonora mentionned

[vshcherb]

Actually I was wrong ... Now is build officially released, so it's ready to test.

[vshcherb]

yes I can confirm current nightly build 3x-5x faster and a lot faster actually

[sonora]

Ok, thanks, I am out on a trip and will not be able to re-install my environment for a test, hope someone else can help out here.

Looks like you have not merged #19239 yet, it's definitely worth it in any case, though.

[vshcherb]

Yes assigned to review & merge

[sonora]

@vshcherb Just out of curiosity: Using e.g. assembleAndroidFullOpenglFatRelease should produce a "fast" build, no matter if we sign it with our debug or our release key, or am I missing anything here?

EDIT: see #18168 (comment)

[sonora]

Small success story here: After quite some work on streamlining the code, removing recursions, and removing surplus track and folder analysis calculations, @Chumva and I were able to reduce the time it takes to load the 'MyPlaces>Tracks' screen by an order of magnitude. On my setup from once ~40 sec to now <3 sec.

And also the logcats now reflect significant improvement:

05-16 01:34:39.260 31493 32007 I net.osmand: GpxDbHelper Time to loadGpxItems 4173 ms, 3445 items {Initializing app}
05-16 01:34:39.288 31493 32007 I net.osmand: GpxDbHelper Time to loadGpxDirItems 28 ms items count 3445 {Initializing app}
05-16 01:34:39.288 31493 32007 I System.out: Initialized GPX_DB_INITIALIZED in 4201 ms
05-16 01:34:39.367 31493 32007 I System.out: Initialized POI_FILTERS_INITIALIZED in 79 ms

Still, 3sec feels a bit 'laggyy', of course, that's what we still have #19683 for, I guess.