Detect and convert ANSI texts

[pbstudent]

bullet interpreted as ¥

Sinclair, S. & G. Rockwell. (2022). Contexts. Voyant Tools. Retrieved September 4, 2022, from https://voyant-tools.org/?query=resources&corpus=e44f6aba0a6c711b581e316dbff2ad0d&view=Contexts

services • identify human and other resources
services ¥ identify human and other

[ajmacdonald]

That's odd. Can you please describe the file format and encoding of the source document (or email a copy to voyanttools @ gmail)?

[pbstudent]

Since the problem you identified was the ANSI plain text encoding instead of UTF-8, is there a way Voyant Tools could automatically recognize a plain text file and convert to UTF-8 ?

[ajmacdonald]

See: https://tika.apache.org/1.28/api/org/apache/tika/parser/txt/CharsetDetector.html

[ajmacdonald]

Note to self: add option under Processing to force a particular encoding.

[pbstudent]

Thanks Andrew. As promised a long time ago I am providing my URL to the bibliographic list discussing Voyant Tools: https://dt.athabascau.ca/jspui/handle/10791/419 <https://dt.athabascau.ca/jspui/handle/10791/419> It is not a great work but sufficed under my health conditions. Fortunately Voyant Tools helped fill the gap. Typical responses were that Voyant Tools was complicated. I disagreed. The software is not complicated to use, rather it disguises the algorithms that make the corpus visualized. Perhaps more discussion in an advanced section of the help site for each tool could provide the supporting algorithms. I found no other application on the Internet that could break down a corpus so nicely as Voyant Tools. Thank you. Cheers, Steve

…

On Dec 20, 2023, at 18:45, Andrew MacDonald ***@***.***> wrote: Note to self: add option under Processing to force a particular encoding. — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFZHF65JPBCIYY7QIBARBDDYKMP4ZAVCNFSM6AAAAAAQG56Z22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRUHA4DMNJQGY>. You are receiving this because you authored the thread.

[ajmacdonald]

Thank you very much Steve.
I will add it to the Gallery when that gets updated.