Incompatible dataset JSONs files between Mac and Windows

[domthibault]

Hi!

This week I've experience a litlle encoding problem in the Dataset JSON files created through the "write" function.

When creating a Mac JSON file, the file can't be opened on a Windows machine. After a few tests, I can say that :

• the other way around works (Windows --> Mac)
• The file can be read in a text editor
• I opened the file created in BBEdit, changed the encoding to Windows and now the file can be opened on the Windows machine.

Specs of my system :

• Macbook M1 Max
• MacOS 12.3.1
• Max 8.3.1

I don't have the specs of the Windows machine, sorry.

[tremblap]

@weefuzzy is it because we use the system file saving? I reckon it is the same for all OSes but I'll need to try and move this to core if this is the case

[weefuzzy]

There will be some cursed interaction between Windows locale settings and std::ifstream. The mac will (I think) always emit UTF-8 but IIRC it's possible to get windows into a state where it thinks it doesn't understand such things. Moving to core.

[weefuzzy]

Hi @domthibault

I'm now back at my desk. To help me narrow down the problem could you tell me a bit more about the problem you're having? Is it that the files made on the Mac just fail to read properly on Windows (some kind of 'invalid format' error), or that they get read but then you get garbage?

[domthibault]

Hi @weefuzzy,

The files created on a Mac using the 'write' function of fluid.dataset~ cannot be read by a fluid.dataset~ on a Windows machine. A message saying 'File not found' is displayed in the Max console. The file, however, can be opened in a [dict] object using the 'read' function on the same Windows machine.

I've attached a file that causes problem.
Mosaique_AnalyzedCorpus_Dataset.json.zip

[weefuzzy]

Thanks, that's interesting – we're using the Max SDK to resolve paths, but maybe we're not doing it quite right.

One further question: does the full path to the file in question have any special characters in it?

[domthibault]

I will have to check with my student who encountered the problem. Give me a few minutes

[domthibault]

@weefuzzy After checking with the student : there were special characters in his path. When he removes them, the file can be loaded in fluid.dataset~.

Thanks for your help!

[weefuzzy]

Thanks @domthibault – glad there's a workaround for now. I'll see if I can get it fixed in the meantime.

[domthibault]

Hi @weefuzzy, after further discussing the problem with my student, the problem remains (even without special characters in the path).

I think this info can help you in the search of the bug : when I reencode the Mac file (which is by default Unix LF) to a Windows (CRLF) file, it then works. I'm guessing it has to do with carriage returns and line feeds... :-)

[weefuzzy]

Oh no! So, just so that I have this clear in my mind, if you don't re-encode the file, then your student is getting 'file not found'? But all is well if you do? I'll write some tests and see if I can figure out what's happening...

[domthibault]

Exactly.