json_query can not unescape ANSI

[HarryCordewener]

For multiple purposes, I'm making use of the json() functions when storing information.
However, when reverting the stored information, json_query's unescape does not care about ansi.

th json(string,render(ansi(Yr,foo[ansi(gh,bar)]baz),ansi)) => "\u001B[31;43mfoo\u001B[1;32mbar\u001B[0m\u001B[31;43mbaz\u001B[0m\u001B[0m"

th json_query(json(string,render(ansi(Yr,foo[ansi(gh,bar)]baz),ansi)),unescape) =>?[31;43mfoo?[1;32mbar?[0m?[31;43mbaz?[0m?[0m

This inability to safely go back and forth is an issue that's bitten me multiple times now. And I'd like to not have to implement some weird work-arounds to make json come back with proper ansi colors.

[HarryCordewener]

Calling forth the @shawnw !

[HarryCordewener]

A negative effect of the submitted bugfix is that is can cause ansi bleeding as if one used an ansi-sub, if you have a player who is willing to be a jerk like that:

th json_query("\\u001B\[31;43mfoo\\u001B\[38;5;236m\\u001B\[48;5;53mbar\\u001B\[31;43mbaz",unescape)Word%rword

I'm looking into ways to solve that, but since this is ANSI encoding, and not MARKUP, there's no easy way to 'nicely' end the markup.

[shawnw]

Colorized text is an abomination unto Nuggan.

[HarryCordewener]

@shawnw - Could you explain why json(string,...) transfers it to xterm256 ansi, instead of remaining as markup? I assume there was a reason for it.

[shawnw]

I don't know why it does that.

[HarryCordewener]

Ahh, that was @grapenut that did that. I should have used Git Blame.
Hey @grapenut - thoughts on a JSON definition for 'markupstring' for back-and-forth on this?

Alternatively, for me to shift json(string) from XTERM256 to MSG_MARKUP. But I think that breaks backwards compatibility, which I don't really want to risk.

[grapenut]

I chose to render as XTERM256 (rather than the internal Penn markup format) because that's the most acceptable format for interoperability outside of Penn. Primarily, I think that JSON being sent to HTTP servers/clients or out-of-band MUSH clients should not be using the internal Penn markup that nobody else understands.

It was not a perfect solution for cases of local-only JSON use, but I think it was the right decision given the alternatives. The output that the json(string) function produces must be valid JSON, which means the control codes will be mapped to unicode escape sequences regardless of whether that's ANSI or Penn markup. No matter which markup syntax you use, if you want to store control codes in JSON and then redisplay them on the MUSH, you will have to use an unsafe unescape method to re-render them.

If you change to markup instead of ANSI, your unescape method will have the same problem with color bleeding, but now additionally you'll have to deal with being able to insert any arbitrary Penn markup (including out-of-band JSON and HTML/Pueblo!), and probably causing general undefined behavior with unterminated markup sequences.

I think the best solution is to not use json(string) to encode colorized strings that you intend unescape and redisplay in-game. I'm still not sure what the specific application is, but as long as you aren't using oob() or @http, you should be able to fake the JSON syntax (if that's what's important) without using json(string) to keep the strings in normal Penn markup. Trying to build a json(object) with invalid JSON strings probably fails an internal check though, but there's nothing stopping you from making your JSON object string completely without using any json() functions.

[HarryCordewener]

To lead off - I believe you made the right decision.

But coming back to the topic of making it so I can show stuff that I store in the database: The issue doesn't lie within the encoding of a string using json(string), not strictly. The issue is that once put into JSON, I can't pull it back out in colorized/tagged form with json_query(...,unescape). It's one-way. I'm currently working to introduce json(markupstring) for the option of storing the markup. As you noted, it comes with its own problems. I'm looking at making sure tags that are opened, are closed within the json read, but I'm starting to believe that may not be easy - or even impossible.

[grapenut]

Yeah it is definitely one-way. Even if you fake the JSON format and insert the raw markup, your document store is probably going to either complain or strip the control codes if they don't get encoded as unicode escapes.

How about adding a wizard-only json_query(...,unescape_markup)?

[HarryCordewener]

Yeah, I can't insert the raw markup. The unescaping you are doing to UTF8 is the only way to do this. JSON will throw a fit otherwise. unescape_markup is probably the 'cheapest' or 'easiest' way to do this. I still may keep the json(markupstring,...) as well to allow markup storage or just ansi storage.