[bug] 500 ISE on posting

[mirabilos]

Describe the bug with a clear and concise description of what the bug is.

I get a 500 Internal Server Error on posting something.

Log:

@40000000636af1dd1084ad04 timestamp="09/11/2022 00:18:27.276" func=router.loggingMiddleware.func1 level=ERROR latency=32.315051ms clientIP=2001:4dd7:3371:0:21f:3bff:fe0d:cbb1 userAgent="Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" method=POST statusCode=500 path=/api/v1/statuses error="Error #01: ERROR: unterminated quoted string at or near \"'<p><span class=\"h-card\"><a href=\"https://mastodon.social/@Alisca\" class=\"u-url mention\" rel=\"nofollow noreferrer noopener\" target=\"_blank\">@<span>Alisca</span></a></span> Mr “Schei�\" (SQLSTATE 42601)\n" msg="Internal Server Error: wrote 54B"

The request (according to Firebug, or whatever this is called now) was:

{"status":"@Alisca@mastodon.social Mr “Schei� Encoding” has, unfortunately, died.","in_reply_to_id":"01CQE88MJ0M72BZ60JSMTFYWXQ","visibility":"unlisted"}

I think you’re misprocessing the U+FFFD here.

What's your GoToSocial Version?

0.5.2 git-c31f219

GoToSocial Arch

amd64 binary

Browser version

firefox-esr 102.4.0esr-1~deb11u1

What happened?

500 instead of posting

What you expected to happen?

Posting instead of 500

How to reproduce it?

Post a single U+FFFD character, not even as reply, is sufficient.

I expect to be able to post all valid UCS codepoints, that is:

• from C0 control characters, at least U+0009 and U+000A
• U+0020‥U+007E
• U+00A0‥U+D7FF
• U+E000‥U+FFFD
• U-00010000‥U-0010FFFD with the exception of U-????FFFE and U-????FFFF

Anything else we need to know?

No response

[NyaaaWhatsUpDoc]

Thanks for reporting! That is an interesting one... Possibly a result of passing through a combination of libraries during string sanitizing. Will get on that :)

[mirabilos]

Wild semi-educated guess: someone set an encoding validation library to replace with U+FFFD then checks for the presence of that to cause an error. Hm, though that’s not so likely to throw a 500. Maybe some component just dislikes them in the input.

[tsmethurst]

This looks like an error is being returned by the database, and then by https://github.com/uptrace/bun -- the ORM that we use.

[mirabilos]

Nope, the database is fine:

gotosocial=# UPDATE statuses SET content='<p>�</p>', text='�' WHERE id='01GHGKE8MFA1D894DDQTV6ZHQN';
UPDATE 1

Apparently, it even can serve this: https://toot.mirbsd.org/@mirabilos/statuses/01GHGKE8MFA1D894DDQTV6ZHQN

[tsmethurst]

Aha okay thanks!

[mirabilos]

There were no instructions on with what parameters to create the database, so I just used sensible defaults:

    Name    |   Owner    | Encoding  | Collate |  Ctype  |   Access privileges   
------------+------------+-----------+---------+---------+-----------------------
 another    | another    | SQL_ASCII | C       | C       | 
 gotosocial | gotosocial | UTF8      | C.UTF-8 | C.UTF-8 | 

The another database is what I generally use, for speed, if I know the application is not caring about the encoding of the data in the database (as relates to use in queries) and does its own thing, but I defaulted to “safer” settings here.

[tsmethurst]

hey @mirabilos, can you still reproduce this one? just wanted to check on it

[mirabilos]

Yep.

[mirabilos]

I can post U+FFFD � now, in 0.10.0-rc3 git-f431974 (but I haven’t tested the over a million other codepoints).

[tsmethurst]

I'll close this now then, since we can't realistically test the million other codepoints. If it comes up again, we can reopen the issue.

[mirabilos]

tobi dixit:

I'll close this now then, since we can't realistically test the million other codepoints. If it comes up again, we can reopen the issue.

That’s what I was thinking, but I wanted to leave the actual closing to you. Thanks!