DB Backend PR

[wolf-sigma]

Thanks @jgeumlek!

[jgeumlek]

I've been busy with other things, but this pull request shouldn't cause much trouble if you make sure the database path respects the new config folders. The XML import Aloshi wrote appears to work fine, but I haven't tested this feature extensively.

If you want to revert the menu restructuring I did while keeping the new menu features, most of it should be fairly straight forward on just cut-pasting the code that builds the menus, being sure to also move over any needed helper functions/closures.

The easy-to-overlook changes are the edits to the HelpPrompts and input handlers for the menus, since I changed the behavior start vs. select. Don't forget the input handler in the View Controller either. (The restructuring was done for an arcade machine setting, and might not suit your purposes.)

I've been thinking of changing the database schema away from what upstream had, since it puts strange constraints on the metadata for filters. If I do so, I'll be sure to write code to pull the information out of the old schema, since it might actually have users now.

[wolf-sigma]

hey @jgeumlek - thanks a ton. I'm trying to work through this to figure out how we can merge into the branch. Fixing the library scale issues is a major priority for me.

I've started a design doc that covers what I'm trying to accomplish, but want feedback from others.
https://docs.google.com/document/d/1J7PXBnFqU_8KVYOtlwxjADvUC4TcUTEmHOa7KkR0sSU/edit?usp=sharing

So that I don't have to go through all the commits and reverse engineer your changes, can you help me with documenting the following things from this PR?

• ERD of the DB
• Any UI Changes
• Anything else you feel is relevant

I'm trying to accomplish several requests by generalizing any metadata changes, so I want to take our time merging this in.

[zigurana]

The biggest improvement this would bring is to allow dynamic listing (filtering) based on all game properties, not just system-type.
My main question though is: how does this affect performance?
The assumption seems to be that the SQL (or some other db) would increase performance on lower-end HW such as the RPi, but I would like to see a side-by-side comparison before I trust it.

[jgeumlek]

Sorry, my life has gotten busy again. Here's an info dump about the questions above.

DB layout:
Taken from Aloshi, there is one table files with a column for every bit of game metadata. there isn't really much of an ERD, as there is pretty much just one entity.

(fileid, systemid, filetype, fileexists, name, desc, image , thumbnail, rating, releasedate, developer, publisher, genre, players, playcount, lastplayed, PRIMARY KEY (fileid, systemid))

One column is systemid, just the internal system identifier name. System info is not in the DB, but still just in es_systems.cfg.

filetype is either GAME (1), FOLDER (2), or FILTER (3).

fileexists is essentially unused on my fork, and is intended to store whether this file was found on the filesystem.

Files have a fileid resembling ./path/to/file.ext where the . is later expanded to the system root. Children of folders are returned from the DB by just checking the fileid of the folder is a prefix. Multilevel folders are handled correctly.

Due to this simple one table schema, filters had to abuse some existing columns. genre stores the query string, players stores the limit number, and developer stores the ORDER BY clause. A filter just builds a query to execute when asked to return its children.

There is an additional table, tagtable with three string columns (fileid,systemid,tag). It is fairly self explanatory, and a stored view named tags selects the tags for a particular file.

Aloshi had built some automagic to handle updating the table should a new metadata column be added or removed.

New DB Layout
I have a branch newschema where there is a table filelist with the information required to display a file (fileid,systemid,fileexists,filetype,name,desc,image,thumbnail). Based on the filetype, one can then search in md_game, md_folder, or md_filter for the rest of the metadata. There is automagic to handle the schema changing in the future here as well.

Filtering here is still limited to just the md_game metadata.

UI Changes
I made several UI tweaks to improve the behavior, such as making ratings go up in half stars, making menus cycle around when you reach the top and bottom, and escaping single line text edits when pressing up or down, and some fixes to the slide animation. Most selections where one picks a subset of systems (e.g. scraping) default to selecting only the current system. The detailed info display updates more reasonably when over a folder, or when entering a folder.

I also added new menu items, like manually scanning the filesystem for files to put in the database or remove missing files, as well as adding or removing tags.

You'll likely want to revert these changes: The two menus (start/select) are now one menu (select) with (start) used to start games. My previous comment goes into this more. The other change is putting system volume at the root of this menu.

Performance
I do not own a Pi or similar device, but sqlite has been pretty speedy for me. It think it might perform better than the previous xml/store the whole hierarchy in memory approach, but I do not know.

Obviously a user could write a complicated filter than takes time to compute. Putting this power in the user's hand seems fair. I have an install with about 60k file entries due to some cruft from importing/duplicate files (this happened back on the XML version, and I haven't bothered to clean it up), and it handles well on a desktop. Having a dense amount of tags may slow down every tag search though. But for modest tagging, I have not seen anything. Loading images for display seems to be the only performance hiccup I've noticed.

The biggest killer on ES start up time is scanning the filesystem. Auto refreshing the game lists is nice, but this does not seem to be the appropriate price to pay. Making the database refresh a manual action is a large savings, as this likely happens rarely compared to starting ES. It's possible SSDs negate this, I wouldn't know.

Design document comments
These are noble goals. I can't comment on the text searching, but I can say that the tag functionality already can cover most of these, whether it be a "favorite" tag, a "kid" tag, or a "series zelda" tag. These tags could later be ascended to a dedicated DB column if needed. The ability to specify a filter for a whole system both allows hiding games (like BIOS files) as well as creating new meta-systems, such as one holding favorites from all systems.

It does not really make managing metadata any easier. Since it is a sqlite tables, I have been using sqlite3 and some scripting to do some database edits.

There is a function in the DB class to return the tags of a file, but no code actually retrieves them to display. This would require somehow changing the theme structure to make a place to show tags... (Or perhaps just tack them onto the description?) You'll probably want to leave the old theme structure behind and move to a setup where a text item has an attribute declaring which bit of metadata it wishes to show.

Known Bugs and comments
Cache invalidation is the main problem, now that a file can appear in multiple systems and generating lists can be expensive if the user makes a complex filter.

These aren't showstoppers, just that the ordering/filtering of the gamelist and the metadata like last-played may be stale.

Any listing can be refreshed by changing the sorting, or entering/leaving a folder.

Writing this requires handling the case where the gamelist is empty, which was asserted to never happen in the upstream. I've caught quite a few cases where this assert might happen, but there might still be a few lurking around. It leads to a crash with a clear message about asserting the list size is greater than 0, so these are generally easy to fix.