A magic character \128 in CC: Tweaked character encoding

[Den666Bah666]

I need to make lexer and I have a little issue with understanding:

1. What encoding CC: Tweaked uses, because it is a bit hard to relate CC: Tweaked codes to real UTF8/16 chars
2. What the hell \128 means? It has the same space letter as \32, but instead of it, \128 not working with %S pattern, for example:

for part in ("test\128words\128and other"):gmatch("%S+") do
    print(part)
end

I hope someone will know more than me

[Wojbie]

So, CC font is kinda custom combination of few different charsets, and some specific stuff made just for CC.

As you can see, \128 character (or 0x80 in this chart) is part of 2 rows of so called drawing characters, that are used to draw pixel-like images using text. It is not considered a white space, but "all 6 pixels of background color", that is why %s pattern is not hitting it.

Here's my favorite image showing which charset what part of it comes from:

[SquidDev]

What encoding CC: Tweaked uses, because it is a bit hard to relate CC: Tweaked codes to real UTF8/16 chars

ComputerCraft uses latin1 + some extra things (see this image for a breakdown). However, nobody actually writes files using that encoding — I'd just stick to parsing ASCII.

What the hell \128 means? It has the same space letter as \32, but instead of it, \128 not working with %S pattern, for example:

That's a non-breaking space.

Lua patterns are not locale/charset aware — they only work with ASCII characters. %s uses the same rules as the C isspace function, so is equivalent to \f\n\r\t\v. It doesn't actually make complete sense with CC's charset, as \f\v are ♀♂ respectively :/. Again, I'd ignore this codepage for everything aside from writing to the terminal!

[Wojbie]

I was under impression that non-breaking space was 0xA0 so \160 ? Surprising to learn about %s tho, makes one wonder if that's something users run into often.

[SquidDev]

Oh you're right 🤦. Got confused between 0x80 and 0xa0, as they're in the same column, just different rows. I think it is equivalent to normal space then — looking at the CC <-> unicode conversion tables I put together a while back, I do just convert it to one.

[Den666Bah666]

If I understood correctly, 0x80 is a empty char for drawing in CC:T and 0xa0 is a non-breaking space just for text? Also, thanks for normal tables of CC:T charset