Consistent null pointer exceptions from RLEScanPartialInternal when multiple databases are loaded

[jghoman]

What happens?

We’re seeing consistent null pointer exceptions (NPEs) when executing queries in a Rust-based multiple-DuckDb-database-loaded scenario. A full stack trace from a coredump is here, but the relevant call is:

(gdb) info threads
  Id   Target Id                      Frame 
* 1    Thread 0x7f264abfd700 (LWP 48) 0x0000562085dbda04 in void duckdb::RLEScanPartialInternal<signed char, true>(duckdb::ColumnSegment&, duckdb::ColumnScanState&, unsigned long, duckdb::Vector&, unsigned long) ()
  2    Thread 0x7f26497fb700 (LWP 50) 0x00007f2651cac388 in futex_abstimed_wait_cancelable (private=0, 
    abstime=0x7f26497f8c10, clockid=0, expected=0, futex_word=0x56208d515c00) at ../sysdeps/nptl/futex-internal.h:323
  3    Thread 0x7f2650d9c700 (LWP 45) 0x00007f2651cac388 in futex_abstimed_wait_cancelable (private=0, 

This stack trace is consistent across all crashes, and occurs either here as originating from our code, or from the TaskScheduler

These stack traces are identical across the NPEs.


We’ve got a Rust application, which is using the duckdb-rs library, via which we access multiple DuckDB databases. We’re using R2D2 to pool the connections for each database.

The potential twist that may impact this is: The individual databases are newer versions of the same underlying data, but the database files themselves have different names (if that matters; I don’t think it does?) The content of the tables within each are different, there are the same named tables, etc. within each file. For example, we have dataset-A-123.duckdb loaded, and then may also host dataset-A-345.duckdb. 


We consistently see the NPEs if we don’t do any queries for a while (order of 10s of minutes) and then do a routine query. Unfortunately, it’s not consistent enough to be able to reliably or deterministically reproduce in a unit test; the queries against the databases are very complex and dynamic.

I’ve checked that there’s been no recent changes to the RLEScanPartialInternal code wherein the crashes are happening. We’re running v1.3.2 though also saw this with v1.3.1.


Questions:

• Each DuckDB file that’s opened should have its own memory space, correct? There’s no way that the loading of a new DuckDB file should impact the existing one, specifically taking into account that the two or more files could have schemas and tables with the same name.
• Is there any way to specifically isolate the memory between multiple DuckDB files when they’re loaded via the same process (out-of-process is not an option here).
• Is there something specific to the RLE encoding code that would trigger this? It’s the same call every time. The exact instruction that crashes is:



=> 0x0000562085dbda04 <+100>:   movzwl (%r12,%rax,2),%eax


Movzwl is used to move a 16-bit value to a 32-bit value, filling the upper 16 bits with zero. However, the address from which it’s trying to read (intended to be stored in the rax register) is zero when being called, hence the NPE. I’ve not yet got back to see what specifically is invoking this instruction.

To Reproduce

I'm still working to be able to reliably reproduce this.

OS:

x86_64-linux

DuckDB Version:

1.3.2

DuckDB Client:

Rust

Hardware:

Relatively arge memory instances (64GB) with Duck configured to use ~60GB.

Full Name:

Jakob Homan

Affiliation:

ASF

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[Tishj]

NPE: Null Pointer Exception ?

[jghoman]

Yes. I've updated the text to clarify. Thanks.

[Mytherin]

Thanks for the report!

This is likely an issue with concurrently opening the same database file multiple times within the same process, that should be mitigated in duckdb-rs, and not a bug in DuckDB.

It is not safe to operate on the same DuckDB database file concurrently unless locks are used. DuckDB resolves this in normal operation, either by:

• Using file locks, for cross-process communication
• Using mutexes, for inter-process communication

When the same DuckDB database is opened multiple times within the same process on Linux and MacOS, both of these locks are side-stepped, leading to incorrect behavior and potential race conditions/crashes as invariants no longer hold. This is because file locks are handled at the process-level, and thus ignored in this case, and mutexes are handled at the database-level, and thus also side-stepped.

The connection pool library you are using is likely opening the same DuckDB database multiple times, and hence causing this issue.

I would file this in the duckdb-rs repo - duckdb-rs should use the global database instance cache to mitigate this, which forces shared connections to go to the same underlying DuckDB database. Until then, I would recommend not using the Rust connection pool library as it seems to not be safe to use together with duckdb-rs at this point.

If you can reproduce the issue without the connection pool library we would be happy to investigate further of course.

[jghoman]

Hey Mark-

Thanks for the quick and detailed response. The situation's not exactly as what you're describing. We're not opening the same DuckDB file multiple times, rather it's along the lines of:

\data\thedataset-v1.duckdb contains
    * table1
    * table2
\data\thedataset-v2.duckdb contains
    * table1
    * table2
    * table3

We will have thedataset-v1.duckdb open for some amount of time, and then load thedataset-v2.duckdb. thedataset-v1.duckdb will remain open for a brief time while as we finish serving any in-flight queries against it. The connections to both DuckDB databases are pooled via separate instances the DuckDB component of r2d2 as described below; I've verified that it appears is correctly releasing/dropping connections as they close.

So, two different duckdb database files with different names with similar but slightly different content. There is only ever a single connection to a specific DuckDB database at a time, and hence a open file per DuckDB, and a single r2d2 per connection. I had thought that perhaps r2d2 was doing something similar to what you're describing, but have checked the codepath for r2d2[1], duckdb-rs and the duckdb-rs-r2d2 component but it does not appear to be doing so.

Specifically, for each DuckDB database file we open[2], we create a DuckDbConnectionManager, which is parameterized on the file_path. This DuckDbConnectionManager is effectively an Arc<Mutex<Connection>>, which has a reference to the file's path (for non-in-memory databases) and a RefCell to an InnerConnection, which is essentially a struct of the exported ffi duckdb_database and duckdb_connection handles from the C library. The duckdb_connection is opened based on that original path, so should be tied directly to that file. It is this duckdb_connection that r2d2 then maintains access to.

With that new DuckDbConnectionManager, we build a new and separate Pool, which is parameterized on that same connection from before. I've gone through the r2d2 code here and don't see a way that two pools could interact, except in the scenario you described above, where the underlying connections are querying the same backend with no other guards. The duckdb-rs code does not make use of the global cache functionality exported via ffi, so would be prey to the situation you describe[3].

The question then becomes, within DuckDB, is a database defined by the underlying file that supports it (or its in-memory equivalent), or by some other by-name reference where table names could cause collisions and confuse the underlying buffer management. IIUC, it's the former, so despite the similar table names, there shouldn't be a concern of having two different files open at the same time.

Given this info, would you suggest another path?
Thanks again!

---

[1] That being said, the r2d2 library isn't getting much love and its documentation in regards to its Rust implementation is woefully out of date (the provided link 404s as that repo is gone with its code absorbed into the core duckdb-rs repo.). I'll open some tickets to remediate the latter issue, and try to find time on the former...

[2] We're also opening everything in read-only mode.

[3] duckdb-rs does have a prepared statement cache, but that's tied to the Connection so again, shouldn't be able to affect queries across statements as all the 'caching' is done in the Rust code itself via a simple LRU cache.

[Mytherin]

The question then becomes, within DuckDB, is a database defined by the underlying file that supports it (or its in-memory equivalent), or by some other by-name reference where table names could cause collisions and confuse the underlying buffer management. IIUC, it's the former, so despite the similar table names, there shouldn't be a concern of having two different files open at the same time.

Yes, each DuckDB instance should be isolated - there should be no link between different open databases, regardless of if they have the same table names.

[2] We're also opening everything in read-only mode.

How are the database files being written if everything is opened in read-only mode? Is this happening in the same process, or does this happen in a different process?

Given this info, would you suggest another path?

Perhaps you could try not to use the connection pool, and verify if the issue still persists without using it?

You should be able to use the clone method to create multiple connections to the same database.

[jghoman]

How are the database files being written if everything is opened in read-only mode? Is this happening in the same process, or does this happen in a different process?

A different, batch process generates the new datasets on a cadence, places them on shared storage, and these processes copy those files when necessary to local and opens them there. This real-time serving layer is then elastic to query load.

Everything worked great until a few weeks ago; I checked the changelog to see if something in the recent point releases may have impacted the RLE encoding code or the DuckDB file format that might explain this, but didn't see anything.

Since it's the same stacktrace, the same assembly instruction each time consistently - despite being unable to reproduce reliably in testing - I hoped this report might pique someone's memory who had been in that path recently.

[Mytherin]

You could try to disable RLE explicitly using set disabled_compression_methods='rle'; on the writer to see if that resolves the issue.

[mlafeldt]

Hey @jghoman - please let me know if you've had the chance to test Mark's suggestions:

1. cloning the connection without using R2D2
2. disabling RLE

[jghoman]

Hey @mlafeldt - we're experimenting with this now. Will update. Thanks.

[jghoman]

We've not seen this since removing r2d2, but we had some other changes go in as well, so I can't definitively say this was the cause. I'll keep an eye out to see if I can more reliably reproduce. Thanks for the investigation.