Unsupported TNT file

[saramortara]

Dear Martin,

I'm trying to read the TNT matrix from Mirande 2008 (Appendix S5 file characidae.tnt) using ReadTntCharacters() and I'm getting the following error:

Error in toupper (lines): invalid multibyte string 3842
In addition: Warning messages:
1: In grep ("'", lines, fixed = TRUE):
   input string 3857 is invalid at that locale
2: In grep (";", lines, fixed = TRUE):
   input string 3842 is invalid at that locale

Do you have any idea of what is happening? Can it be a problem of encoding? Is there a way to control it?

Best,

Sara

[ms609]

Hi Sara, thanks for this report, and sorry that you're having trouble opening the file. I've made some small changes to the ReadTntCharacters() function and can now open the Mirande file successfully on my local installation, though I didn't encounter the encoding issue.

Could you try installing the development version of 'TreeTools' with devtools::install_github('ms609/TreeTools') and see whether that brings you success? If not, could you let me know the value reported when you type getOption('encoding') into the R console, and if native.enc, the value of Sys.getlocale("LC_CTYPE")?

Thanks,

Martin

[saramortara]

Hi Martin, thank you so much for your reply. I've installed the last version from github however, I still can't read the matrix correctly. My OS is Linux and my Sys.getlocale("LC_CTYPE") is pt_BR.UTF-8. I know that Mirande tnt file is set to ISO-8859-1. Is there a way to control the encoding without changing my settings? Best, Sara

…

On Mon, Mar 29, 2021 at 7:28 AM Martin R. Smith ***@***.***> wrote: Hi Sara, thanks for this report, and sorry that you're having trouble opening the file. I've made some small changes to the ReadTntCharacters() function and can now open the Mirande file successfully on my local installation, though I didn't encounter the encoding issue. Could you try installing the development version of 'TreeTools' with devtools::install_github('ms609/TreeTools') and see whether that brings you success? If not, could you let me know the value reported when you type getOption('encoding') into the R console, and if native.enc, the value of Sys.getlocale("LC_CTYPE")? Thanks, Martin — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4OYDAMOSPCW5WJ62KSSPTTGBP4BANCNFSM4Z6G4A6A> .

-- Sara Mortara + 55 21 99750-6413

[ms609]

Hi Sara,

It's strange that R is failing to autodetect the encoding of the file (which I suspect is the issue; I hate grappling with encoding errors!)

One thing to try would be to modify the encoding of the Mirande file, for example by opening with Notepad++ and using the "Encoding→UTF-8" menu option. (RStudio can do the same thing.)

Failing that there are a few other changes to the code that I can experiment with; of course it's tricky to debug issues that can't be reproduced locally!

Let me know how you get on,

Martin

[saramortara]

Yes, of course changing the encoding works. Thanks! Just as a suggestion for your function, maybe converting to UTF-8 so `toupper()` can work properly may be a good call. I couldn't think of a better suggestion than what is below (I added after line 4 and then your function works for me without modifying the file encoding on the source): ``` lines <- readLines(filepath, warn = FALSE) # Missing EOL might occur in user-generated # encoding test before continuing file.encoding <- readr::guess_encoding(lines) if(!file.encoding$encoding %in% "UTF-8") { lines <- iconv(lines, encoding) } ``` I know my code would add a new dependency to your package, so I'm not even making a PR just an invitation to make something in respect of encoding (which is always a bummer). In the end, maybe just a note on the documentation would help others ;) Again, thanks for your help!

…

On Mon, Mar 29, 2021 at 10:51 AM Martin R. Smith ***@***.***> wrote: Hi Sara, It's strange that R is failing to autodetect the encoding of the file (which I suspect is the issue; I hate grappling with encoding errors!) One thing to try would be to modify the encoding of the Mirande file, for example by opening with Notepad++ and using the "Encoding→UTF-8" menu option. (RStudio can do the same thing.) Failing that there are a few other changes to the code that I can experiment with; of course it's tricky to debug issues that can't be reproduced locally! Let me know how you get on, Martin — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4OYDBHLNCUZV2HSPGRBTTTGCHWPANCNFSM4Z6G4A6A> .

-- Sara Mortara + 55 21 99750-6413

[ms609]

Thanks for the suggestion. readLines() already does its best at guessing the encoding (successfully in my case; I wonder why it's failing on your system) – perhaps 'readr' has a more sophisticated protocol?

In the first instance I've updated the function documentation; if you get the chance, perhaps you could check that the "Details" text makes sense and the new example works on your machine as well as mine?
Thanks,
Martin