FAQ: Allocate More Memory

Andrea edited this page Sep 21, 2016 · 5 revisions
Clone this wiki locally

FAQ: How to allocate more memory?

I increased more memory for OpenRefine but get an error "could not reserve enough space"

If you plan to use more than 3 Gigs of RAM for OpenRefine you need to

and set it as your default JAVA_HOME.

If you are working with large data sets, you'll need to allocate more memory than the provided default. If you get OutOfMemoryError reported or if your operations are taking a very long time to run, you should allocate more memory if possible. You will, however, need to balance that allocation with the needs of your operating system.

Even if you don't get OutOfMemoryError, when you are getting close to running out of heap space (memory), the system will spend more and more time managing the heap (ie garbage collecting) and less time doing useful work on your operations.

A typical best practice is to start with no more than 50% of your available physical memory, since certain OS's will utilize up to 1 Gig or more of physical RAM.

Example: If you have 2 Gigs of memory and run Windows Vista or 7, you might only have 1 gig of physical memory left over. Starting with a setting of 512m (50% of 1 Gig) can avoid disk caching problems processing large files with many rows and columns. (A 200 meg text file of 331,000 rows x 50 columns)

Side note about dealing with larger datasets

If your project is big enough to need more than the default amount of memory, you probably also want to consider turning off "detect types automatically" on import as well. It's convenient, but less efficient than explicitly converting any columns that you need as a data type other than the default "string" type.

Windows

Open a "command prompt", 'cd' to the folder where OpenRefine is installed, then type

refine.bat /m 2048m

where "2048m" is the maximum amount of memory (in Mb) that you want OpenRefine to use. To change your default, you can edit refine.ini to specify the memory. Note that this file is only read, if you use refine.bat, not google-refine.exe directly.

If you decide to use the google-refine.exe directly, then set your parameters in the google-refine.l4j.ini file found in the same folder.

# Launch4j runtime config

# initial memory heap size

-Xms256M

# max memory memory heap size

-Xmx2048M

# -XX:+UseLargePages

# -Dsomevar="%SOMEVAR%"

Mac

If you have downloaded the ".dmg" package and you start OpenRefine by double clicking on it, follow these instructions

  • close OpenRefine
  • "control-click" on the OpenRefine icon (opens the contextual menu)
  • click on "show package content" (a finder window opens)
  • open the "Contents" folder
  • double click on the "Info.plist" file (a Property List Editor app comes up)
  • expand the "java" section at the bottom
  • double click on the "VMOptions" content (the right end side)
  • change "${java.options}" with "-Xms256M -Xmx1024M"
  • save file
  • start OpenRefine again

Linux (or Mac):

If you have downloaded the ".tar.gz" package and you start OpenRefine from the command line, use the "-m xxxM" parameter like this

./refine -m 1024m

The number doesn't need to be a multiple of 1024, but always check that your computer is working properly, leaving a fair amount of RAM for the browser and the operating system to function. To make the change permanent/default, edit the refine.ini file and specify the new value.