Segmentation fault (or various memory errors) on WAL application

[paulewog]

What happens?

When applying a WAL file from an INSERT INTO ... SELECT FROM ... GROUP BY, a variety of errors occur, including:

• corrupted double-linked list (Linux / AMD64)
• Segmentation fault (OSX)
• Also the below set of logs (OSX):

duckdb(31324,0x119e43600) malloc: *** error for object 0x6: pointer being freed was not allocated
duckdb(31324,0x119e43600) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

If the group-by is removed, it does not occur.

If the inserts are broken up using LIMIT/OFFSET with breaks inbetween, this also works.

Will upload the CSV.

To Reproduce

CREATE TABLE temp AS SELECT column0, last(column1) FROM 'anon-data.csv.gz' GROUP BY column0;

OS:

OSX, Linux AMD64

DuckDB Version:

• 0.6.1
• Also tried latest completed build off master - a3ea165

DuckDB Client:

CLI

Full Name:

Paul Ellsworth

Affiliation:

Tenable, Inc.

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[paulewog]

Data used to reproduce:
anon-data.csv.gz

[paulewog]

The group-by clause seems to make the problem show up very repeatedly; however, it sometimes shows up even without it; for example:

$ rm -f test.db ; rm -f test.db.wal ; duckdb test.db "create table test as select * FROM 'anon-data.csv.gz';"
duckdb(40732,0x111832600) malloc: *** error for object 0x7fc4cb809404: pointer being freed was not allocated
duckdb(40732,0x111832600) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

[Mytherin]

Thanks for the report! I can verify that this seems to be causing a crash in the FSST compression algorithm. @samansmink could you perhaps pick this up after the holidays?

[paulewog]

Thanks, @Mytherin :) Is there a way to bypass FSST?

[samansmink]

@paulewog you can use this pragma intended for testing:

pragma force_compression="dictionary";

this will disable all other compression schemes except dictionary encoding. Note that this will also disable the other compression schemes though, so your storage size will increase quite a bit.

Currently you can only pick one compression scheme to force or disable them all using:

pragma force_compression="uncompressed";

[paulewog]

Ok, thank you! I'll test it out, if only to validate :)

[paulewog]

(I can confirm that pragma force_compression="dictionary"; does make things pass. :) )