Cannot load library in OMEdit from a directory with accented letters

[casella]

Unpack the attached zip file, that contains two libararies in an accented letter directory.

When trying to open TestPackage/package.mo, I get:

[1] 00:13:40 Translation Error
[C:/dev/OM64bit/OMCompiler/Compiler/FrontEnd/ClassLoader.mo: 313:9-313:128]: Internal error loadCompletePackageFromMp failed for unknown reason: mp=D:/Temp/Lib èèè pack=TestPackage

[2] 00:13:40 Scripting Error
Failed to load package TestPackage () using MODELICAPATH D:/Temp/Lib èèè/.

When I try to open ExternalMedia 3.3.0/package.mo instead, I get

[1] 00:15:02 Grammar Warning
[D:/Temp/Lib èèè/ExternalMedia 3.3.0/package.mo: 2:1-23:18]: Media was referenced in the package.order file, but was not found in package.mo, Media/package.mo or Media.mo.

and an empty ExternalMedia package is shown in the Libraries Browser.

[casella]

@adrpo has already implemented this feature using short directory names in #8573, for use by the package manager and linker/compiler. This also needs to be done when loading libraries in OMEdit.

[casella]

Short directory names seem to be used now, since the error I get with 1.19.0-beta.1 uses them. However, the issue is not yet fixed. Please unzip the TestAccentedLetterDir.zip.

If you attempt to load TestLibrary/package.mo from OMEdit under Windows, you get

[1] 01:14:23 Translation Errore
[C:/OM119/OM64bit/OMCompiler/Compiler/FrontEnd/ClassLoader.mo: 334:9-334:128]: 
Internal error loadCompletePackageFromMp failed for unknown reason: mp=D:/Temp/TestAccentedLetterDir/Lib èèè pack=TestPackage

[2] 01:14:23 Scripting Errore
Failed to load package TestPackage () using MODELICAPATH D:/Temp/TestAccentedLetterDir/Lib èèè/.

while if you run omc test.mos, you get

false
"[C:/OM119/OM64bit/OMCompiler/Compiler/FrontEnd/ClassLoader.mo:334:9-334:128:writable]
Error: Internal error loadCompletePackageFromMp failed for unknown reason: mp=D:/Temp/TestAccentedLetterDir/Lib èèè pack=TestPackage
Error: Failed to load package TestPackage () using MODELICAPATH D:/Temp/TestAccentedLetterDir/Lib èèè/.
"

@adrpo, as I understand the class loader for Windows must also be fixed, otherwise it would be impossible to load libraries from accented directories at all.

[AnHeuermann]

I changed omc_fopen. Do we need to use it in OMEdit as well? I did not change it there, because it uses different stuff around fopen.

@adeas31 Should we check wat OMEdit is using and if omc_fopen is usable there?

[adeas31]

The library loading error is from compiler so if you have changed the compiler to use omc_fopen then it should be good enough. OMEdit only uses fopen for its own log files.

[casella]

@AnHeuermann, @adeas31 please agree on who can take care of this and please make sure it is also cherry-picked to maintenance/v1.19, so it will go into 1.19.0 as well. Thanks!

[AnHeuermann]

I'll give it a go today.

[AnHeuermann]

First issue is, that function getProgramFromStrategy is able to fail silently when getting a file name from a hash table. I re-actiavted the error message in that function:

"[D:/workspace/OpenModelica/OMCompiler/Compiler/FrontEnd/ClassLoader.mo:827:9-827:184:writable] Error: Internal error HashTable missing file D:/workspace/Testitestetst/issue-8777/TestAccentedLetterDir/Lib èèè/TestPackage/Media.mo - all entries include:
D:/workspace/Testitestetst/issue-8777/TestAccentedLetterDir/Lib èèè/TestPackage/package.mo

But the root problem is, that the string for the directory path changes from Lib èèè to Lib èèè. Could be an encoding issue.

---

On Ubuntu unzip did extract the directory to .../Lib KKK/.... I had to change it manually but then there is still no issue on Ubuntu.

By the way, is the typo in Media.mo relevant?

with TestPackage; <---- within?
package Media
end Media;

[AnHeuermann]

Okay, the issue goes way deeper than what I could change quickly.

When creating a mos script test.mos

print("Test: Lib èèè\n\n");

in Windows I get

# omc test.mos 
Test: Lib èèè

I can add the error message #8985 to the master.
@casella Do you want the error message in maintenance/v.19 as well?

But I don't think we can solve this issue fast.
I opened a new ticket for this: #8986

[casella]

First issue is, that function getProgramFromStrategy is able to fail silently when getting a file name from a hash table. I re-actiavted the error message in that function:

"[D:/workspace/OpenModelica/OMCompiler/Compiler/FrontEnd/ClassLoader.mo:827:9-827:184:writable] Error: Internal error HashTable missing file D:/workspace/Testitestetst/issue-8777/TestAccentedLetterDir/Lib èèè/TestPackage/Media.mo - all entries include:
D:/workspace/Testitestetst/issue-8777/TestAccentedLetterDir/Lib èèè/TestPackage/package.mo

But the root problem is, that the string for the directory path changes from Lib èèè to Lib èèè. Could be an encoding issue.

On Ubuntu unzip did extract the directory to .../Lib KKK/.... I had to change it manually but then there is still no issue on Ubuntu.

You can recreate the problem with Lib äää on your German keyboard.

By the way, is the typo in Media.mo relevant?

with TestPackage; <---- within?
package Media
end Media;

Of course it should be within, sorry. Anyway, we are not even getting to the point where that file is read from the file system.

[casella]

Okay, the issue goes way deeper than what I could change quickly.

When creating a mos script test.mos

print("Test: Lib èèè\n\n");

in Windows I get

# omc test.mos 
Test: Lib èèè

Is this just a problem of graphical rendering of the output on the Windows terminal? In that case, I wouldn't bother.

I can add the error message #8985 to the master.

OK. Can be helpful for debugging.

@casella Do you want the error message in maintenance/v.19 as well?

I don't think getting an Internal Error on getProgramFromStrategy (BTW, what the hell does that function name mean?) helps end-user much.

But I don't think we can solve this issue fast. I opened a new ticket for this: #8986

I'm still not convinced that this is the issue. Getting a bad rendering of pathnames with accented letters in error messages on a Windows cmd console is slightly annoying, but no big deal. Not being able to load libraries from directories with accented letters can completely break the package manager, if the username contains accented letters (or Russian characters, as it happened once with one of my students). That is a much bigger deal, particularly in corporate environments, where you cannot install stuff where you want in the file system, but you must stick to your home directory.

[casella]

I mean, the package manager and the linker work fine in that case. We just need to use the same mechanism for the package loader.

[perost]

I don't think getting an Internal Error on getProgramFromStrategy (BTW, what the hell does that function name mean?) helps end-user much.

It loads a file (as an Absyn.Program) using a given strategy: on-demand (loads directly from filesystem) or hashtable (loads from a hash table of pre-loaded files). It actually makes sense in context.

[sjoelund]

I wonder if ANTLR internally loads things fine. Does loadFile work with accents in paths?

OpenModelica/OMCompiler/Parser/parse.c

Line 457 in 934e149

input = antlr3AsciiFileStreamNew(fName);

It might also exist other functions that do not support this as Windows is a mess.

[AnHeuermann]

loadFile is working correctly in the case below:

loadFile("Lib èäöüß/model.mo"); getErrorString();

What is the difference in running the original problem? We do stuff differently for files named package.mo.

But I can't debug this without #8986 being fixed because my prints won't show the right stuff and I don't have Eclipse with MDT set up on my Windows system.

[AnHeuermann]

@mahge and myself tracked it down to void* System_moFiles(const char *directory). Using FindFirstFileW on unicode string pattern will solve the issue. I'll create a PR for it.

[AnHeuermann]

I'll create the cherry pick for maintenance/v1. 19 on Monday.

[casella]

I'd keep this open until we've actually tested it on 1.19.0-dev.beta5 😅

[mahge]

This should be now fixed with #9044. We probably need to do more of these adjustments in more places. For now we can take them as they occure.

Please reopen if the issue is still there.

[casella]

I'll test it tomorrow with the nightly build and re-close if it is fine. If it works for me, then it would be nice to backport the fix to maintenance/v1.19, so we get it in 1.19.1. Thanks!

[casella]

@mahge, I tested #9044 with the latest nightly and now everything works for me 😃

Could you please port it to maintenance/v1.19? Thanks!

[mahge]

Fixed in #9044 and ported to maintenance/v1.19 in #9063.

[casella]

I prefer to keep this open until I've actually checked in on the built version, just in case. I'll take care of that.

[casella]

See https://github.com/OpenModelica/OpenModelica/projects/7

[casella]

Works fine in 1.19.2-dev.beta2.