Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setDT error when list is too big #3121

Closed
chuk-yong opened this issue Oct 19, 2018 · 2 comments
Milestone

Comments

@chuk-yong
Copy link

@chuk-yong chuk-yong commented Oct 19, 2018

First I created a list of 100 elements with each element containing 3 words:
`> w100 <- topfeat_first10M[1:100]

str(w100)
List of 100
$ : chr [1:3] "thanks" "for" "the"
$ : chr [1:3] "one" "of" "the"
$ : chr [1:3] "a" "lot" "of"
$ : chr [1:3] "i" "want" "to"
$ : chr [1:3] "to" "be" "a"
$ : chr [1:3] "going" "to" "be"`

then I converted this list using setDT. It returned a dt with 3 columns which was exactly what I wanted:
`> w100DT <- transpose(setDT(w100))

str(w100DT)
Classes ‘data.table’ and 'data.frame': 100 obs. of 3 variables:
$ V1: chr "thanks" "one" "a" "i" ...
$ V2: chr "for" "of" "lot" "want" ...
$ V3: chr "the" "the" "of" "to" ...

  • attr(*, ".internal.selfref")= `

Next, I created a list with 1million elements and ran the same instruction, but I got this:
`> w1M <- topfeat_first10M[1:1000000]

w1MDT <- transpose(setDT(w1M))
Error in setDT(w1M) :
All elements in argument 'x' to 'setDT' must be of same length`

What's happening? Memory issue? Anyway to get around it?

@MichaelChirico

This comment has been minimized.

Copy link
Member

@MichaelChirico MichaelChirico commented Oct 19, 2018

look at table(lengths(w1M))

@chuk-yong

This comment has been minimized.

Copy link
Author

@chuk-yong chuk-yong commented Oct 19, 2018

`> table(lengths(w1M))

 3      4      5      6      9 

998543 1365 87 4 1 `

thanks! and I wonder why I ended up with elements with length other than 3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.