gc() corruption-related crashes introduced in 3.2.4 affecting DBI occurs on memory overload

[ajdamico]

hi, r core has disclaimed responsibility for this bug. this bug is hugely disruptive for DBI users and it's likely a gc() bug not a DBI or RSQLite bug but i do not have the technical know-how to distill this down any further than i already have. the examples provided to the r-core team are minimal and easily reproducible, but (as they correctly mention) rely on this contributed package.

https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16734

could someone on the RSQLite team please help me reconfigure my minimal reproducible example so that we can re-open this r-project ticket? here's how simple it is

# initiate R with "C:\Program Files\R\R-3.2.4\bin\x64\Rterm.exe" --max-mem-size=35M
library(RSQLite)
db <- dbConnect( SQLite() )
for( i in 1:1000 ) { dbWriteTable( db , 'x' , mtcars , append = TRUE ) }
# CRASH

thank you!

cc @hannesmuehleisen @hadley @krlmlr

[hadley]

Have you tried using the dev version? I can't reproduce locally with:

library(DBI)
db <- dbConnect(RSQLite::SQLite())
for(i in 1:1e6) { 
  dbWriteTable(db , 'x', mtcars, append = TRUE) 
}

I also don't see any problems when running under valgrind.

[ajdamico]

hi, i think the problem happens when RAM starts paging, so doubt you can reproduce on any OS until you max out your hardware -- 1e6 may not be enough? installing the dev version of RSQLite is failing for me. it's possible this bug is windows-only, but both hannes and i have reproduced on different windows computers.. note that it occurs on MonetDBLite also which is why i originally filed under DBI thanks

[hadley]

Well 1000 x 35 mb = 35 gb, which is more memory than I have. So I suspect it's a windows only bug.

[ajdamico]

sorry if i'm being dense, object.size(mtcars) * 10^6 is only 6.7gb? i agree it might be windows-only.. thanks for your time, hadley

[ajdamico]

hi @krlmlr any chance you could explore this in the next month or two? thanks for all your work

[krlmlr]

Can reproduce it on my Windows machine with R3.3.0-patched.

I only get crashes for specific values of --max-mem-size:

• 30, 31: works
• 32-35: crashes at 330 < i <= 340
• 36-37: crashes at 340 < i <= 350
• 38, 50: works

[krlmlr]

Can also reproduce on Linux with the following script, crashing with 670 < i <= 680:

ulimit::memory_limit(318)
library(RSQLite)
db <- dbConnect( SQLite() )
for( i in 1:1000 ) { if (i %% 10 == 0) print(i); dbWriteTable( db , 'x' , mtcars , append = TRUE ) }
# CRASH

Unfortunately, ulimit doesn't work on OS X.

[krlmlr]

I'm fairly certain that this is an R problem. Working on isolating a trigger to reopen the R ticket.

[ajdamico]

crazy. thanks! anything to resubmit the bug to rcore would be helpful..
what's going on internally is far beyond me. but this error happens in
MonetDBLite as well so probably a base R problem. thank you for your time
:)

On Sep 7, 2016 11:38 PM, "Kirill Müller" notifications@github.com wrote:

Can also reproduce on Linux with the following script, crashing with 670 <
i <= 680:

ulimit::memory_limit(318)
library(RSQLite)db <- dbConnect( SQLite() )for( i in 1:1000 ) { if (i %% 10 == 0) print(i); dbWriteTable( db , 'x' , mtcars , append = TRUE ) }# CRASH

Unfortunately, ulimit doesn't work on OS X.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#135 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AANO5_k5gOKH970RtXrNLNVAARdDgwwwks5qnsx9gaJpZM4HyXPl
.

[krlmlr]

The problem seems to occur only if the memory limit is set to something barely above the minimum requirements to run the code. I've seen at least four different error messages and tracebacks.

The worst, however, seems to be an infinite recursion, which I've reported at https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17144.

[ajdamico]

hey, thanks a million

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue and link to this old issue if necessary.