VECTOR_ELT issues with empty data.tables with R 4.6.

[rickhelmus]

Hello,

Since R 4.6 I am experiencing some errors in a project that heavily uses data.table. The following MWE throws an error:

> data.table::data.table()[, 1, by = numeric()]
Error in `[.data.table`(data.table::data.table(), , 1, by = numeric()) : 
  attempt access index 0/0 in VECTOR_ELT

I see this error on my MacBook with R 4.6 and the latest data.table CRAN release and from GitHub. This error is also visible on CI with Windows. For either my MacBook and CI R 4.5 doesn't throw this error.

Thanks,
Rick

[MichaelChirico]

I can't speak to why this doesn't error on R 4.5.0 -- best guess is somewhere higher in the code we get an early return that doesn't happen on 4.6.0. The fix in #7750 should be applied regardless.