Would you mind try to call setDT(data) before the unique call and report the results back here?

Just check if it’s due to the null self pointer.

I tried calling setDT(data) first, but the result was the same.

Oops, having your data would be necessary then.

Please find the data set here

Thanks for the data, I can reproduce this.

UPDATE: the error is thrown from

OK, I create a smaller data set (23MB) that can reproduce this. The cause, I believe, is the recursive function radix_r() is called for too many times (too deep), leading to an overflow on the calling stack.

I don't know why this happens for this specific dataset as I can't reproduce it with randomly generated data.

Code

library(data.table)
dt <- readRDS('~/Downloads/data_debug_tan.rds')
dt2 <- data.table:::duplicated.data.table(dt)

Data

data_debug_tan.rds.zip

Debugged message

Error: C stack usage  8094212 is too close to the limit

Its reproducible with the following setting:

• A wide data.table here with at least 500 columns (at least for my stack size)
• Duplicated rows

As @shrektan pointed out it's forderv and there void radix_r() what actually errors.

Example

x = matrix(rnorm(5000), nrow=10)
idx = sample(10, 20, TRUE)
DT = as.data.table(x[idx,])
forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE)
# Error: segfault from C stack overflow

Calling it a second time also changes the error to

x = matrix(rnorm(5000), nrow=10)
idx = sample(10, 20, TRUE)
DT = as.data.table(x[idx,])
forderv(DT, by=names(DT), sort=FALSE, retGrp=TRUE)
# Error in colnamesInt(x, by, check_dups = FALSE) : 
#   Internal error: savetl_init checks failed (0 100 0x556d9d49d540 0x556d9e05f430). please report to data.table issue tracker.

Error seems to have been introduced by 05c0d45, no segfault at 092fec3

edit:
What seems to be the problem here is that nradix gets so big that the recursion radix += 1 until radix+1 == nradix leads to the stack overflow.

I think I'm running into this with a data.table that is 18k columns, but I'm not getting any errors in R, it just crashes. If a fix is on the roadmap, is there a recommended way to de-duplicate wide data.tables in the meantime?

I think I'm running into this with a data.table that is 18k columns, but I'm not getting any errors in R, it just crashes. If a fix is on the roadmap, is there a recommended way to de-duplicate wide data.tables in the meantime?

You should at least get the # Error: segfault from C stack overflow when running your code from terminal.

AFAIA nobody is working on this fix yet. If the method which is failing is unique then you can fall back to call unique.data.frame(DT) on your data.table using the method from base.

edit: Another way to avoid this is to enlarge your stack size. Under linux you can do this with ulimit -s unlimited to set the stack size to unlimited. You can check that this worked via

Cstack_info()["size"]
#> size 
#>   NA