INTERNAL Error: Invalid unicode (byte sequence mismatch) detected in segment statistics update

[manticore-projects]

What happens?

Appender10MRows	1	2.634649
Appender10MRows	2	2.635860
Appender10MRows	3	2.641729
Appender10MRows	4	2.636160
Appender10MRows	5	2.630352
Appender10MRowsPrimaryKey	1	8.375196
Appender10MRowsPrimaryKey	2	terminate called after throwing an instance of 'duckdb::InternalException'
  what():  INTERNAL Error: Invalid unicode (byte sequence mismatch) detected in segment statistics update

To Reproduce

Normal Performance Test, called by

build/release/benchmark/benchmark_runner

OS:

Linux archlinux 6.2.12-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 20 Apr 2023 16:11:55 +0000 x86_64 GNU/Linux

DuckDB Version:

Latest Git

DuckDB Client:

Shell

Full Name:

Andreas Reichel

Affiliation:

manticore-projects Co. Ltd.

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[manticore-projects]

This problem is not reproducible. I ran the test suite 10 times and it occurred just 1 time.
So unless someone has an idea what triggers is, we can close the issue since I won't be able to trace it to the source.

[Mytherin]

It looks to me like it is a problem in the benchmark itself where the data generator is randomly generating invalid UTF8 and inserting that into the database using the C++ API (avoiding the UTF8 verification until the data reaches the storage).

[manticore-projects]

Yes, that was my assumption too and so I reported this issue.
However, I can look into this only much later, after fixing some JDBC stuff. We can keep it as a reminder. Can we assign priorities?

[lnkuiper]

I have a hunch about what it could be.

We could try running the benchmark with the build flag DESTROY_UNPINNED_BLOCKS=1 and DISABLE_STRING_INLINE=1 and see if we can reproduce.

Edit: cannot reproduce even with these flags

[rsund]

Reproducible parquet in #5882 (although not benchmark related).

[Tishj]

I'm gonna take a guess at the issue, from looking into it a bit.
TL;DR the pointer is saved directly and it outlives the scope

We create a stack allocated char area_code[6];, then we create a string_t from this memory
if DUCKDB_DEBUG_NO_INLINE is set, when constructing the string_t, we store this pointer directly.
Then later at:

string_t StringVector::AddStringOrBlob(Vector &vector, string_t data) {
	D_ASSERT(vector.GetType().InternalType() == PhysicalType::VARCHAR);
	if (data.IsInlined()) {
		// string will be inlined: no need to store in string heap
		return data;
	}
	if (!vector.auxiliary) {
		vector.auxiliary = make_buffer<VectorStringBuffer>();
	}
	D_ASSERT(vector.auxiliary->GetBufferType() == VectorBufferType::STRING_BUFFER);
	auto &string_buffer = (VectorStringBuffer &)*vector.auxiliary;
	return string_buffer.AddBlob(data);
}

This pointer is also not allocated, because it's supposed to be inlined, so we store the pointer in the Vector.

But if DUCKDB_DEBUG_NO_INLINE is not set, then I don't see how this can cause a problem

As a side note, shouldn't DUCKDB_DEBUG_NO_INLINE also make IsInlined always return false?