R 3.6.0 (but not R 3.5.3) incorrectly reports memory.limit()

[WilDoane]

On Windows 10 64bit

Base R's memory.limit() function reports wildly different results in (a) R Console and (b) RStudio 1.3.2xx's console for R 3.6.0.

R3.5.3 reports 16279, as expected on my system, in both RConsole and RStudio.

R3.6.0 reports 16279 (R Console) and erroneously reports 17592186044416 (RStudio console)

memory.limit() results:

.	R 3.5.3	R 3.6.0
R Console	16279	16279
RStudio 1.3.2xx	16279	17592186044416

######################################################################################
# R 3.5.3 (R console)
######################################################################################

> Sys.info()
       sysname        release        version        machine 
     "Windows"       "10 x64"  "build 16299"       "x86-64" 

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          5.3                         
year           2019                        
month          03                          
day            11                          
svn rev        76217                       
language       R                           
version.string R version 3.5.3 (2019-03-11)
nickname       Great Truth                 

> rstudioapi::getVersion()
Error: RStudio not running

> memory.limit()
[1] 16279

> memory.size()
[1] 41.87


######################################################################################
# R 3.6.0 (R console)
######################################################################################

> Sys.info()
       sysname        release        version         machine 
     "Windows"       "10 x64"  "build 16299"         "x86-64" 

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          6.0                         
year           2019                        
month          04                          
day            26                          
svn rev        76424                       
language       R                           
version.string R version 3.6.0 (2019-04-26)
nickname       Planting of a Tree          

> rstudioapi::getVersion()
Error: RStudio not running

> memory.limit()
[1] 16279

> memory.size()
[1] 42.77



######################################################################################
# R 3.5.3 (RStudio)
######################################################################################

> Sys.info()
       sysname        release        version      machine    
     "Windows"       "10 x64"  "build 16299"     "x86-64"    

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          5.3                         
year           2019                        
month          03                          
day            11                          
svn rev        76217                       
language       R                           
version.string R version 3.5.3 (2019-03-11)
nickname       Great Truth                 

> rstudioapi::getVersion()
[1] ‘1.3.236’

> memory.limit()
[1] 16279

> memory.size()
[1] 74.1



######################################################################################
# R 3.6.0 (RStudio)
######################################################################################
> Sys.info()
       sysname        release        version      machine    
     "Windows"       "10 x64"  "build 16299"     "x86-64"    

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          6.0                         
year           2019                        
month          04                          
day            26                          
svn rev        76424                       
language       R                           
version.string R version 3.6.0 (2019-04-26)
nickname       Planting of a Tree          

> rstudioapi::getVersion()
[1] ‘1.3.236’

> memory.limit()
[1] 1.759219e+13

> memory.size()
[1] 43.41


######################################################################################
# R 3.5.3 (RStudio)
######################################################################################

> Sys.info()
       sysname        release        version      machine    
     "Windows"       "10 x64"  "build 16299"     "x86-64"    
> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          5.3                         
year           2019                        
month          03                          
day            11                          
svn rev        76217                       
language       R                           
version.string R version 3.5.3 (2019-03-11)
nickname       Great Truth                 

> rstudioapi::getVersion()
[1] ‘1.3.260’

> memory.limit()
[1] 16279

> memory.size()
[1] 74.43


######################################################################################
# R 3.6.0 (RStudio)
######################################################################################

> Sys.info()
       sysname        release        version      machine    
     "Windows"       "10 x64"  "build 16299"     "x86-64"    

> R.version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          6.0                         
year           2019                        
month          04                          
day            26                          
svn rev        76424                       
language       R                           
version.string R version 3.6.0 (2019-04-26)
nickname       Planting of a Tree          

> rstudioapi::getVersion()
[1] ‘1.3.260’

> memory.limit()
[1] 1.759219e+13

> memory.size()
[1] 55.22

[ronblum]

@WilDoane Thank you for filing this issue. In addition to 1.3, I can reproduce this in 1.2.1335.

[RubyFore]

I am also experiencing this issue (Rstudio version 1.2.1335, R Version 3.6.1) and it is currently preventing me from increasing the memory limit in R studio. Would be great to get this fixed as I am limited to working on Windows and prefer to work in Rstudio! Any workarounds or fixes I can take?

I have set --max-mem-size= 2G, which is reflected in R console but not in RStudio.

#----------Rstudio-----------------------------
> R.Version()
$platform
[1] "x86_64-w64-mingw32"

$arch
[1] "x86_64"

$os
[1] "mingw32"

$system
[1] "x86_64, mingw32"

$status
[1] ""

$major
[1] "3"

$minor
[1] "6.1"

$year
[1] "2019"

$month
[1] "07"

$day
[1] "05"

$`svn rev`
[1] "76782"

$language
[1] "R"

$version.string
[1] "R version 3.6.1 (2019-07-05)"

$nickname
[1] "Action of the Toes"

> RStudio.Version()
$citation

To cite RStudio in publications use:

  RStudio Team (2018). RStudio: Integrated Development for R. RStudio, Inc., Boston, MA
  URL http://www.rstudio.com/.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {RStudio: Integrated Development Environment for R},
    author = {{RStudio Team}},
    organization = {RStudio, Inc.},
    address = {Boston, MA},
    year = {2018},
    url = {http://www.rstudio.com/},
  }


$mode
[1] "desktop"

$version
[1] ‘1.2.1335’

> memory.size()
[1] 96.38
> memory.limit()
[1] 1.759219e+13
> memory.limit(size=2000)
[1] 1.759219e+13
Warning message:
In memory.limit(size = 2000) : cannot decrease memory limit: ignored

#--------------------------But in R console-----------------------------------

> R.Version()
$platform
[1] "x86_64-w64-mingw32"

$arch
[1] "x86_64"

$os
[1] "mingw32"

$system
[1] "x86_64, mingw32"

$status
[1] ""

$major
[1] "3"

$minor
[1] "5.3"

$year
[1] "2019"

$month
[1] "03"

$day
[1] "11"

$`svn rev`
[1] "76217"

$language
[1] "R"

$version.string
[1] "R version 3.5.3 (2019-03-11)"

$nickname
[1] "Great Truth"

> memory.limit()
[1] 2048

```

[LukasTangLowell]

For the time being, is there a workaround? I have to switch between RGui and Rstudio in my project because of this bug.
I.e. althoug I have plenty of RAM left, I cannot finish regressions because it tells me it cannot allocate a vector of size...
However, I get the message "cannot decrease memory limit: ignored" when I set the memory limit, just like in the snapshot above.

[ronblum]

@kevinushey Is there a possible workaround for this?

[kevinushey]

Not that I'm aware of, unfortunately.

It seems unlikely that the wrongly-reported memory limit would be related to allocation failures (which typically implies the R session is running out of memory). I could be wrong, though, and a reproducible example would be helpful (e.g. something that reproduces in RStudio but not RGui)

[WilDoane]

In my case, I'm not experiencing an allocation error, but the way the ff package initializes it's batch chunk size causes a cascading failure later when chunks of a huge size are attempted.

https://r-forge.r-project.org/scm/viewvc.php/pkg/ff/R/zzz.R?view=markup&revision=127&root=ff

if (getRversion()>="2.6.0")  # memory.limit was silently changed from 2.6.0 to return in MB instead of bytes
        options(ffbatchbytes=memory.limit()*(1024^2/100))
      else
        options(ffbatchbytes=memory.limit()/100)
    }

ReprEx:

library(tabplot) # depends on ff
tableplot(mtcars) # works in RGui, not in RStudio

[kevinushey]

Thanks for the bug report. I see a similar error:

> tableplot(mtcars) # works in RGui, not in RStudio
Error in if (by < 1) stop("'by' must be > 0") : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In chunk.default(from = 1L, to = 32L, by = c(double = 23058430092136940),  :
  NAs introduced by coercion to integer range

If I try to set the ffbatchbytes option, it still seems to fail, though:

> tableplot(mtcars) # works in RGui, not in RStudio
Error in if (any(B < 1)) stop("B too small") : 
  missing value where TRUE/FALSE needed
In addition: Warning message:
In bbatch(n, as.integer(BATCHBYTES/theobytes)) :
  NAs introduced by coercion to integer range

Either way, it seems clear that some R packages rely on the R session to return accurate output from memory.limit(), so we should try to fix this if possible.

[SAB83]

Hi everyone,

Does any find any solution for fixing this problem? I really need to increase memory limitation in Rstudio to run my analysis.

Thanks

[philbrierley]

Any progress on this? It means I can't use RStudio,

[kevinushey]

Some notes:

Normally, one can set the environment variable R_MAX_MEM_SIZE to change the memory limit. It appears this isn't respected by R when running in RStudio.

The default value for the memory limit is R_SIZE_T_MAX ref, which I suppose is what we're seeing here.

R later tries to set the memory limit when parsing command line options here:

https://github.com/wch/r-source/blob/e732fe187ee0d680da79330347c2c5fb0ce1da11/src/gnuwin32/system.c#L913-L924

But we don't actually ever use R's own command line parser. Instead, we manually set the memory limit here:

rstudio/src/cpp/r/session/REmbeddedWin32.cpp

Lines 103 to 136 in c6a59ab

	void setMemoryLimit()
	{
	// set defaults for R_max_memory. this code is based on similar code
	// in cmdlineoptions in system.c (but calls memory.limit directly rather
	// than setting R_max_memory directly, which we can't do because it
	// isn't exported from the R.dll
	// some constants
	const DWORDLONG MB_TO_BYTES = 1024 * 1024;
	const DWORDLONG VIRTUAL_OFFSET = 512 * MB_TO_BYTES;
	// interograte physical and virtual memory
	MEMORYSTATUSEX memoryStatus;
	memoryStatus.dwLength = sizeof(memoryStatus);
	::GlobalMemoryStatusEx(&memoryStatus);
	DWORDLONG virtualMemory = memoryStatus.ullTotalVirtual - VIRTUAL_OFFSET;
	DWORDLONG physicalMem = memoryStatus.ullTotalPhys;
	// use physical memory on win64. on win32 further constrain by
	// virtual memory minus an offset (for the os and other programs)
	#ifdef _WIN64
	DWORDLONG maxMemory = physicalMem;
	#else
	DWORDLONG maxMemory = std::min(virtualMemory, physicalMem);
	#endif
	// call the memory.limit function
	maxMemory = maxMemory / MB_TO_BYTES;
	r::exec::RFunction memoryLimit(".rs.setMemoryLimit");
	memoryLimit.addParam((double)maxMemory);
	Error error = memoryLimit.call();
	if (error)
	LOG_ERROR(error);
	}

So that above code seems to no longer be working as expected.

[kevinushey]

I think this is the commit that broke things for us:

wch/r-source@24fc047

Note that it's possible to raise the memory limit, but not lower it, using memory.limit(). With this change, RStudio effectively uses a "very large" memory limit that cannot be altered.

I'm looking into solutions...

[kevinushey]

This will be fixed in the next daily builds of RStudio. If all goes well, we may consider backporting to the v1.2 release.

[LukasTangLowell]

I am about to re-calibrate a model. Is the backporting planned anytime soon?

[kevinushey]

We're considering back-porting this fix, but want to be absolutely sure it doesn't cause any other regressions (since the change we had to make to fix this is a bit dangerous).

(Moving to v1.2-patch milestone just so it doesn't get lost)

[jmcphers]

Backported to 1.2 in 74c0a1e.

[dfalty]

Verified fixed 1.2.5013.

[kmhum]

I am quite new to R/Rstudio and learning the language to handle big data sets. I've come across this same issue now on RStudio version 1.2.5001 where memory.limit() is reporting 1.759219e+13.

I'm trying to learn all the terminology here, but is there a way I can backport the fix for RStudio Desktop? It seems like the fix detailed here has only been confirmed for RStudio Server. Or do I need to manually make changes as referenced above by jmcphers?

For whatever it is worth, I am running a 64-bit Windows 10 system. Thanks for any insight you all may have.

[kevinushey]

The issue should be fixed in the latest preview release, which we hope to promote to a patch release very soon.