Files/folders are not recognized in library scan

[microrache]

I have some mp3-files and folders my music library constantly refuses to scan/import. These files/folders all share a rather "exotic" name, with Turkish or Greece or Russian Characters instead of standard-codepage-1252-characters. See the following screenshots comparing media library and file-/folder-structure:

Am I correct supposing that all files/folders containing non-codepage-1252-characters are ignored while scanning? And if the answer is not codepage-1252, what are the "allowed" characters? Are these restrictions necessary/useful, thinking of middle-east/asian users of the software?

[michaelherger]

Is LMS running on Windows? Is the data stored on local media or attached/mounted drives or shares? What filesystem are you using on that disk?

Please run a scan with the scanning logging group enabled (Settings/Advanced/Logging), then provide a copy of your scanner.log.zip file.

[microrache]

Thank you for the hint about the filesystem, indeed that seems to be the issue: I am running a Windows Server 2012 R2 with a Software-JBOD-solution called DriveBender (which combines several physical drives into one drive letter and does some file-mirroring-stuff).

I tried to move one of the not-recognized albums to the internal drive C of the Windows Server (this drive is not touched by DriveBender) and voilà, the album is recognized by the scan. So it definitely seems to be a DriveBender-issue.

What I wonder about is, why I can access the files normally with any file manager. I can copy, rename, delete, edit, etc. with these files. I can open them in e.g. MP3Tag, play them etc. So I am curious about if and how (i.e. with which name/size/etc.) the LMS-filescanner recognizes these files at all (this information would also give me more details to address the DriveBender-developers). Do you have an idea how I could reproduce the scanner's file-identification-logic? I figured out the method Slim::Utils::Scanner::Local->find() in slimserver/Local.pm, but ... yah ... Perl 🙄. I am thinking of some perl-commands which list the directory's files and their attributes, helping me to understand how the filescanner sees these files...

[michaelherger]

I'd start with a log file. As I said enable scanner logging in Settings/Advanced/Logging, run a scan, and see what it reports for those files. LMS and charsets on file systems have a difficult relationship.

[microrache]

Okay, here we go: I copied those albums to a new folder, added this one as a new source and did an incremental scan (scanner.log.zip). Seems that non-1252-characters are read by the scanner-module as question marks only:

[20-03-24 08:51:18.3922] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\Paradox Obscur [20-03-24 08:51:18.3981] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\Paradox Obscur\2020 - SYN?ESIS [20-03-24 08:51:18.4193] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away [20-03-24 08:51:18.4672] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik [20-03-24 08:51:18.4794] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\10 - Kuruyordu Nehir.mp3 [20-03-24 08:51:18.4915] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\01 - Soluk.mp3 [20-03-24 08:51:18.5036] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\02 - Asimilasyon.mp3 [20-03-24 08:51:18.5157] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\03 - Uzakta.mp3 [20-03-24 08:51:18.5218] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\04 - Narin Yaln?zl?k.mp3 [20-03-24 08:51:18.5335] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\05 - Hayaller.mp3 [20-03-24 08:51:18.5454] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\06 - Katarsis.mp3 [20-03-24 08:51:18.5514] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\07 - Uçtu Belirsizli?e.mp3 [20-03-24 08:51:18.5631] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\08 - Gerçekten Özleyince.mp3 [20-03-24 08:51:18.5692] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\09 - Yan?mda.mp3 [20-03-24 08:51:18.5751] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found D:\ServerFolders\Musik\test\She Past Away\2015 - Narin Yalnizlik\11 - ?çe Kapan?? II.mp3

Strangely the filename-encoding seems to be correct later in the readTags-module:
[20-03-24 08:51:20.8327] Slim::Formats::readTags (267) Report for file:///D:/ServerFolders/Musik/test/She%20Past%20Away/2015%20-%20Narin%20Yalnizlik/08%20-%20Ger%E7ekten%20%D6zleyince.mp3:

But I still wonder why it works when adding the files to the native file system.

[mherger]

Are both filesystems NTFS? Or what are they?

[microrache]

They are supposed to be NTFS, yes. The native file system is definitely NTFS and the ServerFolders-drive is the mentioned JBOD-system provided by DriveBender, which Windows also recognizes as NTFS. To be honest, it's not a super-big issue at all, I can name the files with codepage-1252-characters and that's it. I just found it strange, I never experienced any encoding-issues with that DriveBender-solution before, no matter if accessing the ServerFolders with Linux, Android or Windows-clients. Long story short: I don't want to steal your time in this minor issue, but if you like we could inspect this further. Do you have an idea, why the filenames appear with question marks in the Scanner-log, but with correct HTML-entities later in the ReadTags-log?

[microrache]

Maybe that's interesting for you: I moved the albums onto the native (i.e. physical) drive and scanned again. Here the filenames are cut off in old 8.3-style on the "unknown" characters:

[20-03-24 15:16:42.5755] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur [20-03-24 15:16:42.6089] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1 [20-03-24 15:16:42.6104] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\01 - Mantras.mp3 [20-03-24 15:16:42.6118] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\02 - Fester.mp3 [20-03-24 15:16:42.6131] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\03 - X-Spell.mp3 [20-03-24 15:16:42.6145] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\04 - Abeyance.mp3 [20-03-24 15:16:42.6158] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\05 - Réflexe.mp3 [20-03-24 15:16:42.6171] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\06 - Inferno.mp3 [20-03-24 15:16:42.6185] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\07 - Fester (Alternative Mix).mp3 [20-03-24 15:16:42.6199] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\08 - Fester (Protector 101 Remix).mp3 [20-03-24 15:16:42.6212] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\Paradox Obscur\2020-S~1\09 - Réflexe (Tanya Baltunova Remix).mp3 [20-03-24 15:16:42.6244] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away [20-03-24 15:16:42.6299] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik [20-03-24 15:16:42.6313] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\01 - Soluk.mp3 [20-03-24 15:16:42.6327] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\02 - Asimilasyon.mp3 [20-03-24 15:16:42.6341] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\03 - Uzakta.mp3 [20-03-24 15:16:42.6356] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\04-NAR~1.MP3 [20-03-24 15:16:42.6377] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\05 - Hayaller.mp3 [20-03-24 15:16:42.6396] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\06 - Katarsis.mp3 [20-03-24 15:16:42.6410] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\07-UTU~1.MP3 [20-03-24 15:16:42.6423] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\08 - Gerçekten Özleyince.mp3 [20-03-24 15:16:42.6437] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\09-YAN~1.MP3 [20-03-24 15:16:42.6451] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\10 - Kuruyordu Nehir.mp3 [20-03-24 15:16:42.6465] Slim::Utils::Scanner::Local::Async::__ANON__ (148) Found C:\test\She Past Away\2015 - Narin Yalnizlik\11-EKA~1.MP3

[michaelherger]

Could it be your main filesystem does not support legacy file formats any more? That's indeed a mechanism we're using to work around some limitations. Maybe https://support.microsoft.com/en-us/help/121007/how-to-disable-8-3-file-name-creation-on-ntfs-partitions is of any help?

[microrache]

Okay, I tried several settings for 8dot3-filenames now, but I can't get it to work as it seems there is a general limitation for 8dot3-fallback in the DriveBender-software.

For clarification: 8dot3-fallback does work on the system drive - the scanner finds files with non-codepage-1252-characters there by reading them as 8dot3-filenames. But the "virtual" pool-drive provided by DriveBender seems to not allow this 8dot3-fallback (no matter what is set in fsutil), so the scanner always recognizes these files with question marks for any non-codepage-1252-character, which then leads to the file being ignored on import.

Anyway, I think I just have to take care that filenames are in codepage-1252 only. Thank you for your support!