Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Player can't find files with extended Latin characters? #839

Closed
dwmtz opened this Issue Mar 20, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@dwmtz
Copy link

dwmtz commented Mar 20, 2016

It seems like the Player has problems when loading some files with extended Latin characters.
This is what happens in Blue Star Story:

[2016-03-20 18:32:48] Debug: Loading Map Map0121.lmu
[2016-03-20 18:32:48] Debug: Tree: PigVillage < Mapa Swiata
[2016-03-20 18:32:48] Debug: Cannot find: ChipSet/WioskaŚwiniakówBezDrzew (!)
[2016-03-20 18:32:48] Warning: Image not found: ChipSet/WioskaŚwiniakówBezDrzew

screenshot_8
RPG_RT can load it just fine.
I've been struggling to figure out whether it's something on my end, the Player, or the game's database.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Mar 20, 2016

Rename ChipSet/WioskaświniakówBezDrzew to ChipSet/WioskanŚwiniakówBezDrzew (capital Ś)
Marking this as a file finder bug

@Ghabry Ghabry added the FileFinder label Mar 20, 2016

@fdelapena

This comment has been minimized.

Copy link
Contributor

fdelapena commented Mar 20, 2016

This would require Unicode case-insensitive string comparison with u_strcasecmp() from ICU (ustring.h) and something equivalent for the iconv fallback.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Mar 20, 2016

Or u_strToLower

@dwmtz

This comment has been minimized.

Copy link
Author

dwmtz commented Mar 20, 2016

Windows 7 64-bit, Player64.

Debug: Detected encoding: ibm-5346_P100-1998

I had it set to 1250 in RPG_RT.ini before, changing to 1251 and 1252 didn't do anything. The file actually uses the lower case "ś" character by default.
Does that mean the Player ignores files unless it gets a perfect match with what's in the database?

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Mar 20, 2016

The Player does a case insensitive match but this currently only works for ASCII, not for anything extended.
This is for supporting platforms that are case sensitive... (all except Windows)

@BlisterB

This comment has been minimized.

Copy link
Member

BlisterB commented Mar 21, 2016

I have a problem similar with some projects (not everyone) with the Android app.
Some files with filenames containing "é", "è", or "à" aren't find.

If I change the filename with ES Explorer with the exact name, it works.
Hint : When I delete a "é" character, I have to press 2 times the back button, after 1 time it became a "e". So I suppose it's a similar problem : encoding.

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Mar 21, 2016

BlisterBoy, it seems Android does not normalize filenames.
Probably stores them as e + [combiner] ` = è instead of è.

In theory this should be solvable by asking ICU to normalize the unicode. But I never had this behaviour under Windows, so hard to debug because debugging native code under Android sucks :/

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented Mar 27, 2016

carstene1ns proposes a unicode-lowercase solution in gentools: EasyRPG/Tools/pull/22
The same can be used here.
BlisterBoys issue can be resolved using "unorm2_normalize" https://ssl.icu-project.org/apiref/icu4c/unorm2_8h.html#a0a596802db767da410b4b04cb75cbc53
Both should be added to the FileFinder, too.
Also open for discussion is, whether iconv supports this. (of course ICU is superior but some ports use iconv only)

@Ghabry Ghabry added this to the 0.5.0 milestone Apr 11, 2016

@Ghabry

This comment has been minimized.

Copy link
Member

Ghabry commented May 11, 2017

Possible solution. We would just need to replace all "Utils::LowerCase" calls with ReaderUtils::Normalize in FileFinder (putted this in ReaderUtil because of ICU which is linked into liblcf and maybe useful for other tools that need this...)

// Converts UTF-8 to Lowercase and then Decomposes it
std::string ReaderUtil::Normalize(const std::string &str) {
#ifdef LCF_SUPPORT_ICU
	UnicodeString& uni = icu::UnicodeString(str.c_str()).toLower();
	UErrorCode err = U_ZERO_ERROR;
	const Normalizer2* norm = icu::Normalizer2::getNFKCInstance(err);
	UnicodeString f = norm->normalize(uni, err);
	std::string res;
	f.toUTF8String(res);
	return res;
#else
	// LAME
	std::string result = str;
	std::transform(result.begin(), result.end(), result.begin(), tolower);
	return result;
#endif
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.