DuckDB memory limit breached when db files are attached

[pradp]

I'm testing duckdb in my java application for data analysis. I'm running my application in kubernetes with constrained memory environment and found this below behaviour

Scenario 1:

1. Create connection pool with JDBC URL with .db file with 50 GB data
2. Set Pod memory in kube deployment to 12 GB, JVM Max heap memory -Xmx to 4GB, DuckDB set memory_limit='4GB' ;
3. Run any complex queries, the memory doesn't breach 12GB pod limit and pod memory is <=8GB any time

Scenario 2:

1. Create connection pool with JDBC in memory ATTACH .db file with 50 GB data
2. Set Pod memory in kube deployment to 12 GB, JVM Max heap memory -Xmx to 4GB, DuckDB set memory_limit='4GB' ;
3. Run any complex queries, pod crashes due to OOM. Duckdb 4GB memory limit is not respected.

Is this the expected behviour for ATTACH use cases?

[mustafahasankhan]

I faced a similar issue, I explicitly set -Xms1G -Xmx7G and pod memory-request to 4Gi and memory-limit to 10Gi, while DuckDB memory_limit was set to 4GB
Found, memory breaching, it went upto 8.54 GB.

[pradp]

@mustafahasankhan Just to confirm, did you also use ATTACH?

[mustafahasankhan]

@pradp No I didn't.

[mustafahasankhan]

I am close to Scenario 1, attaching empty .db file in JDBC url when creating connection, yet.