Increase max memory default

[mdsumner]

I routinely set this value much higher to avoid system users filling up temp folders.

Could it be adaptive to the current system? I think the default is much too low at any rate.

[lbusett]

Hi, a bit late on this, but I just noticed this thread and I am currently struggling with memory management on some large datasets analysis, so I am interested in the topic.

@mdsumner Increasing the default maxmemory seems risky to me. Already with the current defaults (maxmemory = 1e09 cells, corresponding to 4 / 8GB), when working with large satellite images I have to be careful to avoid getting into trouble on "standard" rigs with 8 / 16 GB of RAM.

@rhijmans The new estimatemem functionality introduced in 3c36860 could be a lifesaver. Just a couple of comments / suggestions if I may:

• On windows, the call memavail <- 0.5 * utils::memory.size(NA) - utils::memory.size(FALSE) used in canProcessInMemory() does not seem to return the current amount of free memory, but the total memory, minus that currently "taken" by "R". A possible way to get the amount of free memory could be using something like:

    mem <- system2("wmic", args = "OS get FreePhysicalMemory /Value", 
                   stdout = TRUE)

and parsing the results.

• The dynamic calculation of free memory is triggered by the estimatemem option, which is a TRUE/FALSE. Would it make sense to allow instead the user to set a "percentage of free memory" to be used for the calculation of memavail (currently, it is "hardcoded" to 0.5) ?

• I was thinking that a similar approach could be used to dynamically compute the chunksize used in out-of-memory operations, when using WriteStart / WriteValues / WriteStop constructs exploiting BlockSize. This could allow optimizing the number/size of chunks with respect to free memory.

HTH

[mdsumner]

Admittedly my request here was relevant to a pre 2.6-7 version, it was increased with that version and had been causing me regular disk issues for a long time until I set the limit manually.

It's just been increased again (2.7-15 just hit CRAN).

[lbusett]

@rhijmans that system2("wmic"..... returns info in kB ( at least on my machine). The change implemented in f76fb22 should account for this.

Note that I just did a PR implementing the correction, and also suggesting a couple more changes related to "memory management": see #11

HTH