MalformedInputException related to auto update mods and maps bricks FAF

[magge-faf]

The error has traces back to 24 Nov 2022, but recently the "java.nio.charset.MalformedInputException: Input length = 1" is an occurring phenomena for several users or at least got reported more often than usual. The MalformedInputException basically bricks their FAF, and can only be solved temporary (for some users, not all apparently) with a complete wipe of all files and a reinstallation - until the Game.prefs (not 100% sure on that) breaks again.

Follow user report from Discord, and some are here in this thread: https://forum.faforever.com/topic/5186/mod-error-without-any-mod/24

To Reproduce
I can not reproduce it on my machine, I would guess the users with the issue have a specific map(s) or mod(s) XY which malformed their file when they auto update - Is there any way to enclose the issue further by asking the users which mods/maps they have installed, or about their Game.prefs ? Their installation and path profile looked clean so far, no OneDrive or Cyrillic letters - Usually the generic basic stuff to check first for.

Log or error message
Log file of a user:

client (12).log

[Sheikah45]

This is caused by the game changing the encoding of the game.prefs file from UTF-8 which the client can read to ANSI1 which the client cannot read.

It is unknown exactly what causes the encoding to change but right now my thought is some mod has a different file encoding and the game writes the game.prefs in the ANSI encoding.

We made a change to the client so that the game could still launch but maps and mods will not be updated.

At the moment there isn't really a plan on how to address or fix this. since utf-8 is the expected and correct encoding.

[PathKnower]

Maybe it would make sense to handle those scenarios at least on the client side instead of leave it as is since it basically unplayable when this issue occurs. Or create a ticket for the Game team with reasonable priority?

[Sheikah45]

I have already talked with the game team about it and they are just as puzzled. And it currently is handled on the client side just you get a notification that your selected mods could not be updated or changed

[Garanas]

Initially we thought this would fix it:

• Fix encoding issues fa#4449

But apparently it does not for everyone. I'm not sure where else to look.

[Garanas]

@PathKnower were you playing on FAF Beta Balance branch?

[PathKnower]

@Garanas never, main balance only

[Garanas]

That would be the only place where #4449 would not have an impact. One last thing we could do - could you send me your current mods folder along with your preference file? Perhaps if I have the same mods, we can reproduce it on my system. You can use www.wetransfer.com if the content is too much for Github.

[PathKnower]

At this moment I'm not nearby my pc now, as soon as I reach it I will share things

[PathKnower]

@Garanas here, game.prefs and mods.
mods.zip

[magge-faf]

Out of interest, I did a lookup over all files to print out all "malformed" ones aka ISO-8859-1 like the game.prefs:

Not sure if it helps, but maybe it leads to something. Maybe a mod introduces the ISO-8859-1 format at one point and FAF checks their files and is overtaking its attributes for its game.prefs - Dunno.

---

from pathlib import Path

import chardet

rootdir = Path(r'C:\Users\username\Desktop\files')

for path in rootdir.rglob('*'):
    if path.is_file():
        with open(path, 'rb') as f:
            result = chardet.detect(f.read())
            if result['encoding'] == 'ISO-8859-1':
                print(f'{path} is encoded in {result["encoding"]} with confidence {result["confidence"]}')

---

Output:

C:\ProgramData\Anaconda3\envs\checkEncodingOfFiles\python.exe C:\Users\username\PycharmProjects\checkEncodingOfFiles\main.py 
C:\Users\username\Desktop\files\Game.prefs is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\Advanced target priorities\textures\Expand2.dds is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\hook\lua\ui\game\economy.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\hook\lua\ui\game\gameresult.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\announcement.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\build-power.blua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\diplomacy.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\score_board.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\score_board_scalingUI.blua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\score_manager.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\SupremeScoreBoard\modules\score_popup.lua is encoded in ISO-8859-1 with confidence 0.73
C:\Users\username\Desktop\files\mods\ZeP_MiniMapZoom\hook\lua\ui\game\minimap.lua is encoded in ISO-8859-1 with confidence 0.73

[magge-faf]

I drafted a solution, it solves the issue on my dev machine every time I re-recreate the problem environment. It needs more testing, of course, if it is being considered at all.

magge-faf/faf-client@24818c1

---

Basically, I just catch the ISO_8859_1 exception and decode its content to UTF_8

The 2nd catch in writeActiveMods is probably not needed, because the initial exception comes from readActiveMods, but removing code is the easy part.

[Garanas]

@Sheikah45 what are your thoughts on the solution of Magge?

[Sheikah45]

If we go this route then rather than have the try catch decide which encoding is used it would be better to have a prioritized list of encodings that are used and the first one that works is used.

So a for loop with a try catch continue would probably be preferable and less fragile

[magge-faf]

@Sheikah45 'prioritized list of encodings' - You mean a list of the malformed charsets to check for, when the exception is caused? It is necessary that Game.prefs is encoded in UTF-8 to work with the client, right?

[Sheikah45]

It isn't technically required. The client just assumes it is utf8.

Actually before this came up the client was reading and writing it in ASCII.

So there is no required encoding. As long as java supports it the client can read it. Just previously it was thought that the game only supported encoding files as ASCII/utf8 so that was the same assumption the client made.

In light of this it might be more reasonable to try multiple encodings and just use the one that works.

[magge-faf]

@Sheikah45 Thanks for your input. Your thoughts about this solution? magge-faf@0326da5

---

I iterate through all known charsets with 'for' and break out when it works - it fixed our problem scenario, even without converting the file.

Theoretically, a user can still have an exception by actually having a really malformed file, but that needs extra investigation, when that case happens.

[Sheikah45]

Yeah that looks good from a conceptual standpoint. If you make a pr I can add some other comments.