Fileman: paginate instead of keeping a list of all elements in directory

[gullradriel]

Description of the feature you're suggesting.

I want a fileman caching system to allow low memory print navigation/file choosing and custom saved directory sorting.
It would be based on storing/updating/using hidden .fileman.db files at the root of each directory on the sdcard to quickly open and read the current displayed 8 lines

Overview of the how fileman cache update should work (will be edited as the conversations are giving feedback)

• at PP start: update_all_fileman_db
• at any time we need to read a directory content and navigate/choose a file: open target/.fileman.db and use it. When clicking on a directory, open directory/.fileman.db and so on until a file is selected
• instead of (the number of files in the directory + the included directories) display is using 8 menu entries (or 8 whatever will be used to represent one line) in Fileman utility and embedded views

Each directory modification / deletion / creation should call update_fileman_path_db on:

• the target directory and sub dirs
• the directory including the target itself

Each file modification / deletion / creation should call update_fileman_path_db on:

• the target directory

Menu list process:

• set current_index to 0 , highlighted to 0
• call refresh_menu( current .fileman.db , current_index, highlighted ) at start and on scroll

refresh_menu:

• open target .fileman.db
• get sort type
• skip lines until reaching 'current_index', test if not out of bounds
• read 8 lines, for each read line store entry type, entry name, entry size if any
• test current_index/highlighted position/nb_files boundaries to see if a current_index scroll is needed

Additionally:

• the number of files and the number of directories should be stored and updated when needed
• we may keep track of number of subdirs/subfiles
• we can design .fileman.db so it keep tracks of the chosen sort type on each directory and we can directly display the sort type and sorted list
• the list would be sorted at creation and when changing sort type, saving both sort type and sorted list
• each file entry line must have: type file, full path to file, filename, size, (attributes ?)
• each dir entry line must have: type dir, full path to dir, dirname, (attributes ?)

[NotherNgineer]

Another thing I'd like to see in Fileman is the ability to sort files by name or time stamp (this is probably even more unlikely if we only load a partial list into memory, but I thought perhaps the design could take that into consideration)

[NotherNgineer]

Just a thought: I think we set aside about 70KB of RAM space for baseband, and I wonder if we could either allocate space for the file list from that, and/or move part of the File Manager code to baseband. I can imagine messages to baseband to "add this file to list", "sort file list by name (or date)", or "return info on file index 15" so I can display it.

Of course this assumes baseband doesn't need to be running when selecting files, and there would still be a maximum file count (perhaps 1000 vs 100).

[NotherNgineer]

Using DFU, it also appears to me that File Manager isn't completely deallocating space for previously-opened folders until File Manager is exited.

Example: When I start File Manager from the utilities menu, there's about 25KB free (M0 core + M0 heap) when looking at the root directory. But if I open and close my SCREENSHOTS folder with 95 files in it, available memory has dropped to <18KB while viewing the root directory. The missing >7KB seems to be deallocated when File Manager is exited, but perhaps we could look into deallocating this space when the view of the large folder is exited. Otherwise it seems that I can get intermittent Guru "out of memory" faults when navigating between folders.

[gullradriel]

Yes, it's also causing 'out of memory' on 'random' or after a few apps launch.
That thing need to be rewritten, even if we loose the sorting.
The only way I see for us to keep the sorts and keep speed is to have a procedure that would list sdcard content and put a hidden .fileman.dat at the root of each dir, containing the list of files with size/dirs sorted.
That procedure should rely on a single procedure 'update_fileman_path_db(path)' that we can call in some other parts.
Then the fileman would just open the .fileman.dat at given path, read it, and just seek on the file.
Any procedure that is modifying/adding/deleting folder/files may call update_fileman_path_db on the path itself or the path of the file.

[gullradriel]

@NotherNgineer I'm editing the OP to reflect what I think we need now, and how it should be done

[gullradriel]

Solved by #2077