[Bug] False "could not be fully tokenized" error when a cfg value contains emoji or special characters

[TheGamerzs]

Summary

If a value in server.cfg contains an emoji or any special character (é, €, Japanese, etc.), the server prints an error at startup saying the line "could not be fully tokenized and the config variable will miss some content".

The error is wrong — nothing is actually missing. The value is stored in full: /dynamic.json returns the complete hostname and it shows as expected in the launcher, emoji included.

The two numbers in the message ("Tokenized until 32 of 35") are just the same line measured two different ways — one counts characters, the other counts bytes. Plain-ASCII values never show the error because both counts are equal; every special character makes the counts differ, which falsely triggers it.

Issue type

Server

Repro rate

Always

Server build version

106 (FXServer-early-access b106 win32), 139 (FXServer-early-access b139 win32)

Platform

Windows

OS version / distribution

Windows 11 Pro (10.0.26200)

Using txAdmin?

No

Hosting provider

Self-hosted

Machine type

Dedicated

DDoS protection

No

Steps to Reproduce

1. Add to server.cfg (saved as UTF-8): sv_hostname "Sunset Valley RP 💸"
2. Start the server and watch the console.
3. Check /dynamic.json or the launcher — the hostname is complete, despite the error.

Expected Behavior

No error — the value parses fine and nothing is lost.

Actual Behavior

[error] [citizen-server-i] Line sv_hostname "Sunset Valley RP 💸" could not be fully tokenized and the config variable will miss some content. Tokenized until 32 of 35

…but /dynamic.json returns {"hostname":"Sunset Valley RP 💸", ...} — the full value, and the launcher shows it correctly too.

Evidence

Console output from one test boot — the error appears for every value with special characters, never for plain ASCII, and in every case the value was stored intact:

sv_hostname "ASCII Control Server"      -> no error
sv_hostname "Café Señorita RP"          -> Tokenized until 30 of 32
sv_hostname "Euro € Server ✔"           -> Tokenized until 29 of 33
sv_hostname "東京 Drift RP"              -> Tokenized until 25 of 29
sv_hostname "Happy 😀 Server"           -> Tokenized until 28 of 31
sv_hostname "Sunset Valley RP 💸"       -> Tokenized until 32 of 35

The gap between the two numbers is always exactly the extra bytes the special characters take (each emoji adds 3, each accented letter adds 1) — the line was parsed to the end every time.

Additional Context

The check behind this message seems to compare the parsed length in characters against the line length in bytes, so it can never pass for non-ASCII text. Happy to test again on a newer build if that helps.

[wolfram-cfx]

Thanks, we are looking into this.

[stelianclaudiu92-commits]

Great comment. UP!

[Gogsi]

A fix for this issue has been implemented and is available in August 31 patch. Please try again and let us know if the issue persists. Thanks for your help!

[TheGamerzs]

Retested on FXServer-early-access b139 win32 (the artifact published today at 15:11 UTC, which the August 31 patch notes point to) — the issue persists, unchanged:

[error] [citizen-server-i] Line sv_hostname "Sunset Valley RP 💸" could not be fully tokenized and the config variable will miss some content. Tokenized until 32 of 35

Same for é/€/CJK/😀 values, with the same numbers as before. As in the original report the values are still stored fine (/dynamic.json returns the full hostname) and plain ASCII doesn't warn — only the false error is still there.