newcharmap documentation

[aaaaaa123456789]

The documentation for newcharmap states that it creates a new, empty charmap. However, this charmap is most certainly not empty:

	newcharmap test
	charmap "a", 1

SECTION "Test", ROM0[0]
Test:
	db "b"

Ideally, there should be a way to create an actually empty charmap, so that the above gives an error. But if this is not possible, at least the documentation should be updated.

[ISSOtm]

The behavior is that all characters must translate to something. When would you need another behavior?

[aaaaaa123456789]

I'd expect (and until I tested this, I thought this is how it worked) undefined characters to give an error, as a way of preventing non-existent characters (such as random Unicode lookalikes) from making it into games.

[ISSOtm]

Well then, I think documentation should state that whenever a character is not found in a charmap, it's simply copied as-is in the ROM. That way, the behavior of an empty charmap would make sense.

[Rangi42]

Looking at lexer.c, it appears that:

• Source code must be encoded in UTF-8, or in ASCII (a compatible 7-bit subset of UTF-8)
• Unless the current character map says otherwise, bytes are copied into the ROM as-is (as they're encoded in the source code file). So db "ña" means db $c3, $b1, $61, since ñ is U+00F1 which is UTF-8 encoded c3 b1 (and a is U+0061, 61 in UTF-8 and ASCII).
• So if the source code has a different encoding (one that's still ASCII-compatible enough for the keywords and opcodes to work), non-ASCII characters might get inserted differently.

[ISSOtm]

Frankly, strings can use a different encoding than the rest of the document; I don't think we should try to handle those documents, as its out of spec anyway.

[aaaaaa123456789]

Is there anything that cares about string encoding? Should there be?
As long as the encoding is an ASCII superset (so the lexer can find important characters like quotes and backslashes), is there any harm to treating strings like simple binary blobs? Charmaps are binary substitutions, after all.

[Rangi42]

Whatever it is, the documentation to be accurate. Right now it says "By default, a character map contains ASCII encoding", which is incorrect. A default character map is empty, and string characters not in the charmap are inserted directly as their encoded bytes. That encoding is expected/allowed to be UTF-8, not ASCII (although of course ASCII is a subset of it).

As long as the encoding is an ASCII superset (so the lexer can find important characters like quotes and backslashes), is there any harm to treating strings like simple binary blobs?

That's what I was trying to hedge against in the wording, but the current behavior actually doesn't allow that. db "ña" in an ISO-8859-1 file gives an "Input string is not valid UTF-8!" error.