PC-side music library database builder

[noisetta]

Idea

Instead of rescanning the music library on the device after adding music,
build the SQLite database directly on the PC and have the device use it on boot.

How It Would Work

1. User adds music to SD card on their PC
2. Script runs on PC, reads metadata from audio files and builds the database
3. SD card inserted into device
4. Modified hiby_player.sh detects pre-built database on SD card and copies
   it to internal storage
5. hiby_player launches with database ready — no scanning needed

Benefits

• Faster library updates — PC is much more powerful than the device CPU
• Better metadata handling for non-Latin scripts (Arabic, Japanese etc.)
• Instant playback after inserting SD card

Implementation Plan

• Python script (cross-platform) for database generation
• macOS GUI app (@Jepl4r) using tkinter
• Linux AppImage (@noisetta)
• Modified hiby_player.sh to detect and copy pre-built database

Database schema is fully documented from ADB exploration — see roadmap discussion for details.

[Jepl4r]

This are all the tables inside the database.

This are the ones I explored for now

ALBUM2_TABLE is for albums (this may be used by the device to display the properties of a song)
ALBUM_TABLE is for albums and ius used by the device to show the albums list.
ARTIST_TABLE is for artists and is used by the device to show the artists list.
ARTIST2_TABLE is for artists (this may be used by the device to display the properties of a song)
MEDIA_TABLE is for songs and is used by the database to show the songs list.
MEDIA2_TABLE contains the paths of where the audio files are stored on the SD-Card. In my case there are two entries: one where all the songs are stored a:\Music\*. and the second is the location of where it saved the playlists exported/imported by the device a:\playlist_data\*.

[Jepl4r]

Did a test with mutagen to read all the tags inside the music files. It took 132 seconds to read 6198 songs on my M1 Pro.

[noisetta]

Wow, nice research! The a:\ path prefix is important , our script will need to convert local file paths to HiBy's internal SD card notation.

The 132 second scan time for 6198 songs on an M1 is already a massive improvement over what the device can do.

For the initial version should we focus on just the core tables needed to get the library displaying correctly — MEDIA_TABLE, ALBUM_TABLE, ARTIST_TABLE, GENRE_TABLE, FORMAT_TABLE and MEDIA2_TABLE — and leave the secondary tables for later?

[Jepl4r]

As of now I'm writing a Python script to only touch the needed Tables to display the Artists, Albums and Songs.
The rest, like COLLECT_TABLE are updated during use. This one for example is updated whenever you click the heart and the song is added to the Favorites.

[Jepl4r]

When I finish writing the script I'll test it on the device to see if it is correct or not.

I'll for sure need to fix some bugs with the creation.

[noisetta]

Nice. Let me know when you have something to test and I'll run it on my end too. If you share the script I can test it on Linux and we can compare results. Also worth noting, from our ADB session we know the database version is 20181015 and the VERSION_TABLE just stores a single integer, so that's one easy table to populate correctly.

[Jepl4r]

Will do.
Yeah I'm not touching that table. The M3U_TABLE is another one we do not have to touch, it is managed by the device to export the playlists.

[Jepl4r]

@noisetta I modified hiby_player.sh do the paths look correct to you?

#!/bin/sh

killall    hiby_player    &>/dev/null
killall -9 hiby_player    &>/dev/null

# If the file usrlocal_media.db exists on the SD, copy it to the internal memory by forcing the overwriting (-f)
if [ -f "/usr/data/mnt/sd_0/usrlocal_media.db" ]; then
    cp -f /usr/data/mnt/sd_0/usrlocal_media.db /usr/data/usrlocal_media.db
    # Ensures that writing to flash memory is complete
    sync
fi

if [ -f "/usr/bin/batd" ]; then
killall    batd    &>/dev/null
killall -9 batd    &>/dev/null
/usr/bin/batd -v -s -t5 -o /mnt/sd_0/batlog.txt &
fi

#/usr/bin/hiby_player &>/dev/null
/usr/bin/hiby_player
sleep 1
reboot

[noisetta]

The init.d approach is better — add an S91_db_sync script that runs after the SD card is mounted (S21) but before hiby_player starts (S92). That way timing is guaranteed and no sleep needed. I can build a test firmware with this script if you want to try it.

[Jepl4r]

oh ok, send me the firmware to test it.

[noisetta]

Hey, i just uploaded it as a pre-release:

https://github.com/hiby-modding/hiby-mods/releases/tag/v1.4-test

Let me know how it goes!

[Jepl4r]

@noisetta tried your firmware with the script but still doesn't work. The device doesn't copy the database file to the internal storage.

[Jepl4r]

I have an idea. We could patch the binary to have a new entry inside the settings to replace the database file.
Not easy to do though.

[Jepl4r]

@noisetta
Managed to patch the hiby_player binary to use an hidden entry inside the settings page to copy the database file and it is working.

Changed {"operating_instruction":0} to {"operating_instruction":1} inside set_function.json

Patched the entry inside settings to execute cp /usr/data/mnt/sd_0/usrlocal_media.db /usr/data/usrlocal_media.db instead of showing the QR Code (the QR Code when scanned sends you to a 404 website, nothing important is lost)

When clicking the entry inside the settings the database file is copied to the internal memory and the device freezes for 1-2 seconds before it restarts (this is needed otherwise the database isn't reloaded)

Note: the restart does not take much time, from my understanding only hiby_player is restarted (takes not more than 2 seconds)

Found some bugs with the script generated database. DSD files are not scanned correctly and the alphabetical sorting does not work, the songs are all scrambled. Will fix this and send you the script with the patched binary.

[Jepl4r]

Ok, the script finds all the music files on Windows and Linux. Only macOS has problems finding some of them.
Probably because those folders use the NFD format, which macOS ignores. nothing I can do about this.

[Jepl4r]

Renaming the folders in windows worked. The folders now use the NFC (one codepoint per character) format instead of NFD (base + combining marks)

[noisetta]

Really clever approach using the existing settings entry , much cleaner than a boot script. The NFD/NFC issue on macOS is a known quirk of Apple's filesystem, good find on the Windows rename workaround. Take your time with the DSD and sorting fixes, looking forward to seeing the script when it's ready.

[Jepl4r]

Yeah, macOS is really weird with filesystems that are not his own.
As for the script, it creates the database with all the songs but the alphabetical ordering is wrong still. I need to fix that.

I'm now patching the binary to open a page when clicking on the added entry. The page will have two options: Copy database to SD and Copy database from SD just to have more control on the database.

[noisetta]

That's much more user friendly and I feel the backup direction is a useful bonus feature.

[Jepl4r]

Not simple to patch in though LMAO.
I'm trying and retrying and everytime when clicking on the patched option to test I get a segviolation / segfault error:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7fd200c0} ---
write(1, "\n\n", 2)                     = 2
write(1, "HiBy Player crashed by signal SIGSEGV.\n", 39) = 39
write(1, "Call Trace:\n", 12)           = 12
write(1, "\t[0x73de6c]\n", 12)          = 12
write(1, "Process maps:\n", 14)         = 14
write(1, "\n\n", 2)                     = 2
+++ exited with 255 +++

[Jepl4r]

FINALLY some progress!!!

[Jepl4r]

Some more progress has been done. The copy to SD option works.
Copy from SD works too but I need to locate the function that refreshes the list after a database update is done, just like the Update database option does. (without it a reboot is needed to load the copied database)

[Jepl4r]

Another thing I noticed is that after a reboot, even if the device shows the list of songs from the new database, the search function does not work. It could be what I said in the comment above or something that is missing from the database that the device reads.

[Jepl4r]

For today this is enough. I need some good sleep.

[noisetta]

Really solid progress. Two way copy working is the core feature done. The search issue is likely the SEARCH_TABLE not being populated by the script, from our ADB exploration that's a separate table that mirrors MEDIA_TABLE data. Worth checking if it's empty in the generated database. As for the refresh function, that's probably the same internal call that the 'Update Database' menu option triggers. @Tartarus6 might be able to locate it through decompilation since they've been digging into hiby_player.

Have a good sleep!

[Jepl4r]

You're probably right. The script doesn't populate the SEARCH_TABLE.

[Jepl4r]

hmm, took a look at the database created by the device and the SEARCH_TABLE isn't populated. It is probably populated on-the-fly with the search results.

[Jepl4r]

IT WORKS!!! No functions needed from the database updater.
MEDIA2_TABLE wasn't populated at all, that was the problem.

[Jepl4r]

@noisetta Here's the firmware with the patched binary https://drive.google.com/file/d/16ucuIYDbj55gbkQz1YcABKBdwoF__taf/view?usp=sharing

And here's the script for the creation of the database Update_Database.py For now the script only works if run from the SD Card with a valid usrlocal_media.db

The script uses mainly mutagen with a fallback to tinytag if mutagen fails to read the tags.

For now only the english sys_set.ini has the strings needed to show the option names. If you use another language they'll be blank.

If all is good for you I'll finish translating all the other languages (Deepl is a good candidate to have consistent translations across all languages) and create a PR for the theme to have a new folder named Unified Theme (Sorting + Database), update the readme accordingly and create two other PRs with the binary and the script. (I'll probably modify the script to work from any directory on the PC and make the user select the correct drive before proceeding with the database creation)

Note: the options inside the Database Manager page will show a dot. This is inherited from the Font size page I used as a template to make the patch. Nothing I can do about it.

[noisetta]

Amazing work! This is a huge milestone. I'll download the firmware and script and test on my end. The plan for separate PRs (binary, script, theme variant) makes sense, keeping them modular is the right approach. The language translation via DeepL is a good idea too, consistent translations across all languages would make this much more polished.

One question: does the script currently need to be run from the SD card root, or just needs a valid usrlocal_media.db present somewhere?

[Jepl4r]

The script needs to be run from the SD Card root

I've done it like this just to test the database creation. I'll modify it to work from whatever folder after you also confrim that it's working.

[Jepl4r]

You may need to do a restart after you copied the database from the SD Card to the device.
I'll see if I can get the database to refresh on-the-fly, otherwise I'll make the option do a reboot after copy.
That's the best thing I can think of without overcomplicating things.

[Jepl4r]

@noisetta Found the function to reload the database without rebooting and patched the binary. Unpack the firmware I sent you before and build it using this hiby_player.zip

[Jepl4r]

@noisetta The GUI app for macOS is complete.

How do you want me to send it to you?
Google drive link works?

[Jepl4r]

HiBy_Database_Updater_GUI.py

This script is to create the linux executable. Make sure you have customtkinter and pyinstaller
This command should create a single file inside the dist folder

PyInstaller --name "HibyOS Database Updater" \
            --windowed --onefile \
            --collect-data customtkinter \
            --hidden-import mutagen.flac \
            --hidden-import mutagen.mp3 \
            --hidden-import mutagen.mp4 \
            --hidden-import mutagen.oggvorbis \
            --hidden-import mutagen.dsf \
            --hidden-import mutagen.aiff \
            --hidden-import mutagen.wavpack \
            --hidden-import mutagen.asf \
            --hidden-import mutagen.apev2 \
            --hidden-import mutagen.oggopus \
            HiBy_Database_Updater_GUI.py

Note: This will not create the appimage, just the executable.

[noisetta]

GUI works perfectly on Linux after installing python3-tk and rebuilding with --hidden-import tkinter. During my test it auto-detected the SD card, generated the database in 4.5s, all 112 tracks across 10 albums correct. Ready to package as an AppImage when you're ready to submit the PR.

[Jepl4r]

I already did the PRs, you can merge them if there are no problems.

[Jepl4r]

Btw the macOS app is only for Apple Silicon. Intel Macs will not be able to run it, I don't have a Mac with an Intel CPU to compile the app for x86.

[noisetta]

Checking the PRs now. On the Intel Mac issue — no worries, we can note in the documentation that the macOS app requires Apple Silicon and Intel Mac users can run the Python script directly instead. Most modern Macs are Apple Silicon anyway.