Unwrap on from_utf8 causes panic with non-UTF-8 database server/client encodings

[chamoretto]

Environment

• Rust toolchain version: 1.53.0-nightly (74874a690 2021-03-30) x86_64-pc-windows-msvc
• IDE name and version: CLion 2020.3.3 (CL-203.7717.62)
• Operating system: Windows 10 10.0
• sqlx version:
• • name = "sqlx-core"
• • version = "0.5.1"
• • source = "registry+https://github.com/rust-lang/crates.io-index"
• • checksum = "b1cad9cae4ca8947eba1a90e8ec7d3c59e7a768e2f120dc9013b669c34a90711"

Describing problem

If the postgres database has an encoding other than UTF-8, sqlx cannot connect to it due to an error in converting the message bytes to utf-8 encoding. Sample code:

What is in the dsn does not matter.
When I run this code, I get this:

Full stacktrace is available here - https://gist.github.com/ShagonRU/6bff4316a33895021b3aaed8554be16a

===

Next are my assumptions based on stacktrace.
After getting error (or another response) from database, this ( sqlx-core-0.5.1\src\postgres\message\response.rs:139:26 ) code panics:

Decode uses this trait impl ( sqlx-core-0.5.1\src\io\decode.rs:13:9 ):

Here is called this decode ( sqlx-core-0.5.1\src\postgres\connection\stream.rs:95:48 ):

So. As a result, If the database is not in UTF-8 encoding, then the user will simply get panic and will not be able to do anything about it. There is also no explanation anywhere about this moment. Moreover, as will be shown in the example above, the server can be in UTF-8 encoding, but if the client is not in it, there will be the same panic.
For example - my docker container with postgres:

And i getting the same error as with my local postgres.
I can only connect to a remote server from my system. (fortunately I have remote server, as well as the second non-Windows system, but this is still confusing)

Possible solutions

• Remove .unwrap() and return correct error about encoding (???, i don'n know what user can do with this error)
• Find out what encoding was used and decode messages to utf-8 from it.

[mehcode]

This should be impossible. Something else must be going on. The data we get back for an error message must be in the client encoding which SQLx sets to UTF-8.

What are some more details about your environment? Windows? How is Postgres running and what version of postgres?

[chamoretto]

This should be impossible. Something else must be going on. The data we get back for an error message must be in the client encoding which SQLx sets to UTF-8.

What are some more details about your environment? Windows? How is Postgres running and what version of postgres?

My Rust environment is presented in issue. I've been using sqlx for half of a year, the same thing was reproduced on nightly 1.49, 1.50, 1.51, 1.52 versions of Rust.
Other components of the environment:
Windows 10 20H2 Build 19042.867. Main encoding - Windows 1251, at cmd.exe - cp866. My source files all in UTF-8.
Docker version 20.10.5, build 55c4c88
I am using Postgresql inside of a docker container:

> root @ localhost: / # psql --version
psql (PostgreSQL) 12.6 (Debian 12.6-1.pgdg100 + 1)

The encoding inside the container can be seen in the screenshots in the issue.
This database is doing fine, the Python application works fine with the same database. I use same image with same docker-compose.yml file at our dev server, and i can connect and use database in this case. Only in this case, in case of remote DB. I can't use local databases now using sqlx.

Local postgres - psql (PostgreSQL) 11.4, pgAdmin 4.9.

(some glitches because of cp866)

Nevermind which database i use, because i won't connect to them. As you can see at the very first screenshot at the start of the topic, i can just write let dsn = "postgres://", and error still the same.

It is impossible, but it's happening, as you can see at stacktrace (gist at topic):

thread 'test' panicked at 'called `Result::unwrap()` on an `Err` value: Utf8Error { valid_up_to: 0, error_len: Some(1) }', C:\Users\gosha\.cargo\registry\src\github.com-1ecc6299db9ec823\sqlx-core-0.5.1\src\postgres\message\response.rs:139:26

It is this unwrap that falls.

I think it's not about the database encdoings, but about system encoding or some kind of UB related to work with raw bytes data.

[Pzixel]

I've used a MRE repo (branch sql-windows-problems-empty) presented by @ShagonRU and I can confirm I could reproduce it locally:

I'm using non-russian en-US locale everywhere so it must be a problem with sqlx itself

[mehcode]

Oh I believe it's happening. I'm just a little floored that it is. Documentation has been known to lie and postgres docs are quite bad. I'll see what I can figure out. Thank you for the expanded details.

[chamoretto]

Oh I believe it's happening. I'm just a little floored that it is. Documentation has been known to lie and postgres docs are quite bad. I'll see what I can figure out. Thank you for the expanded details.

I've tried to inspect your code with debugger and find out the error returned from libpq using memory view. By some reason, code at response.rs:139:26 unable to read this message properly from buf.

First debug point sqlx-core-0.5.1\src\postgres\connection\stream.rs:95:48 - and go deeper
Second - sqlx-core-0.5.1\src\postgres\message\response.rs:139:26, screenshot was made here.

[Pzixel]

Looks like buf is not inited or something like this (weird len and invalid pointer):

(lldb) p buf
(bytes::bytes::Bytes) $13 = {
  ptr = 0x5600cecdc6c0c253 ""
  len = 3621738852849566022
  data = (p = core::cell::UnsafeCell<mut tuple<>*> @ 0x000001e090da26a0)
  vtable = 0xe5f2e0e2eee7fceb
}
(lldb) x 0x5600cecdc6c0c253
error: Invalid access to memory location. 

But it could be a debugger mistake as well

Debug print gives:

[sqlx-core\src\postgres\message\response.rs:137] buf.len() = 175
[sqlx-core\src\postgres\message\response.rs:139] &buf[v.0 as usize..v.1 as usize] = [
    194,
    192,
    198,
    205,
    206,
]

[Pzixel]

I think I decoded the output. Response is

[83,194,192,198,205,206,0,86,70,65,84,65,76,0,67,50,56,80,48,49,0,77,239,238,235,252,231,238,226,224,242,229,235,252,32,34,112,122,105,120,101,34,32,237,229,32,239,240,238,248,184,235,32,239,240,238,226,229,240,234,243,32,239,238,228,235,232,237,237,238,241,242,232,32,40,239,238,32,239,224,240,238,235,254,41,0,70,100,58,92,112,103,105,110,115,116,97,108,108,101,114,95,49,50,46,97,117,116,111,92,112,111,115,116,103,114,101,115,46,119,105,110,100,111,119,115,45,120,54,52,92,115,114,99,92,98,97,99,107,101,110,100,92,108,105,98,112,113,92,97,117,116,104,46,99,0,76,51,51,51,0,82,97,117,116,104,95,102,97,105,108,101,100,0,0]

Decoding as WIN1251

SВАЖНО00VFATAL00C28P0100Mпользователь "pzixe" не прошёл проверку подлинности (по паролю)00Fd:\pginstaller_12.auto\postgres.windows-x64\src\backend\libpq\auth.c00L33300Rauth_failed0000

Meaning it's an auth error which is in WIN1251 for unknown reason

[Pzixel]

I think I've reproduced it.

Try to login with invalid credentials, on windows (it's important, I couldn't reproduce it on linux machine). You will get:

thread 'main' panicked at 'called Result::unwrap() on an Err value: Database(PgDatabaseError { severity: Fatal, code: "28P01", message: "password authentication failed for user "pzixe"", detail: None, hint: None, position: None, where: None, schema: None, table: None, column: None, data_type: None, constraint: None, file: Some("d:\pginstaller_12.auto\postgres.windows-x64\src\backend\libpq\auth.c"), line: Some(333), routine: Some("auth_failed") })', src\main.rs:10:10
stack backtrace:

Now go to the postgresql.conf and set lc_messages = 'Russian_Russia.1252'. Now try again. You will get this panic

In a nutshell: You can't expect database returning responses in UTF8. In fact, it returns it in any custom codepage specified in configuration. I believe sqlx should not assume all responses are performed in UTF8.

Alternatively (if you don't want to depend on crates for parsing different locales. Could be hidden under feature gate though) you can just invent another enum variant InvalidDbResponseRaw(Box<[u8]>) and use it instead of panic.

See DataGrip message btw

It actually knows that response is not guaranteed to be UTF8

[Pzixel]

You can execute SHOW lc_messages command to determine what is encoding for errors for this DB instance:

[Pzixel]

Sorry for a longread, I just was investigating this for several hours and was posting my insights here.

The problem arises from incompatibilty between expectations that DB always returns responses in UTF8 and reality where it's configurable and can be anything postgres supports.

This gap should be filled somehow. Probably via detect encoding default feature which performs some initial checks. But it doesn't help with errors like this since you need to connect in order to execute SHOW and you probably can't. I have no good solution for this.

[chamoretto]

Sorry for a longread, I just was investigating this for several hours and was posting my insights here.

The problem arises from incompatibilty between expectations that DB always returns responses in UTF8 and reality where it's configurable and can be anything postgres supports.

This gap should be filled somehow. Probably via detect encoding default feature which performs some initial checks. But it doesn't help with errors like this since you need to connect in order to execute SHOW and you probably can't. I have no good solution for this.

I think proper way may be someting like:

1. Trying connect to db
2. Matching by results of decoding
3. If decoding function didn't executed successfully, we can decode input by some another way (i.e. with some encodings feature enabled) or just throw InvalidDbResponseRaw(Box<[u8]>) as you pointed above. And we should do this not only for database errors i think, but for other messages too. At least users would be able to inspect their problems with encoding, not just look at panicking code.
4. After connection: If we able to decode message or we was lucky to guess encoding, we can save it to some Encoder, which can do some requests and ask postgres about Server encoding and server messages encoding. Using this info we can now fetch data with any encoding and any messages, because now we knew encoding and can do things. But now we need to use this Encoder thing.

Can if affect real time performance? I don't know. But in can be hidden by the feature flag.
I think feature like this can be useful in some situations:

• If production enviroment is configured for utf-8 only, but developers from many countries using different local encodings. In this case feature may be developers-related
• If some buisiness have database with some strange encoding because of legacy (like latin-1 or cp1251/cp1252/cp866, or some country-related encoding), but developers are not affected by this problem. In this case feature may be are production-related.

[mehcode]

Here is a relevant thread in the postgres mailing list: https://postgrespro.com/list/thread-id/1896813

It seems that errors during startup, notably before authentication, are returned in whatever encoding the server is configured for, regardless of the client encoding.

[mehcode]

This seems to be a fairly common issue actually. Drivers that have the problem:

• https://github.com/jackc/pgx
• https://github.com/lib/pq
• https://github.com/eclipse-vertx/vertx-sql-client

[mehcode]

Found how JDBC fixed this:

zemian/pgjdbc@bd6de1b#diff-92be561701c813957ebca8ca97c2a1e3e0eb9a822c4681dcd2cc0ddfcb34451e

Basically this can only happen for auth errors during connection start-up and will only be FATAL errors. This limits the problem domain a lot. And JDBC then "predicts" the encoding using what it found for FATAL.

[abonander]

And JDBC then "predicts" the encoding using what it found for FATAL.

I just hope it's not lost on anyone how hilarious of a hack this is.

[cdecompilador]

I have the same error with the query macro, for example this let rows = query!("select 1 as one").fetch_one(&mut db_pool).await?; genertes the same error for me.
OS: Windows (64-bits)

[Pzixel]

@cdecompilador the easiest way to solve the issue right now is set encoding on server to be latin. The easiest way is set lc_messages = 'English_United States.1252' (or whatever UTF8-compatible encoding you have, such as en_US.utf8 for instance) in postgresql.conf and reboot postgres instance.

[paulzhang5511]

thread 'main' panicked at 'called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 0, error_len: Some(1) }', C:\Users\zhang.cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\sqlx-core-0.5.5\src\postgres\message\response.rs:139:26

   severity_s = from_utf8(&buf[v.0 as usize..v.1 as usize])
                        .unwrap()
                        .try_into()
                        .ok();

[dodoqang]

If your database is installed on windows, you can comment out the code from b's'to b'c' and try again

[abonander]

Instead of trying to sniff the encoding of the string, I think we should just detect this condition and emit a meaningful error message.