FAQ: Allocate More Memory

Daniel Sedlacek edited this page May 2, 2018 · 3 revisions

FAQ: How to allocate more memory?

I increased more memory for OpenRefine but get an error "could not reserve enough space"

If you plan to use more than 3 Gigs of RAM for OpenRefine you need to

and set it as your default JAVA_HOME.

If you are working with large data sets, you'll need to allocate more memory than the provided default. If you get OutOfMemoryError reported or if your operations are taking a very long time to run, you should allocate more memory if possible. You will, however, need to balance that allocation with the needs of your operating system.

Even if you don't get OutOfMemoryError, when you are getting close to running out of heap space (memory), the system will spend more and more time managing the heap (ie garbage collecting) and less time doing useful work on your operations.

A typical best practice is to start with no more than 50% of your available physical memory, since certain OS's will utilize up to 1 Gig or more of physical RAM.

Example: If you have 2 Gigs of memory and run Windows Vista or 7, you might only have 1 gig of physical memory left over. Starting with a setting of 512m (50% of 1 Gig) can avoid disk caching problems processing large files with many rows and columns. (A 200 meg text file of 331,000 rows x 50 columns)

Side note about dealing with larger datasets

If your project is big enough to need more than the default amount of memory, you probably also want to consider turning off "Parse cell text into numbers, dates, ..." on import as well. It's convenient, but less efficient than explicitly converting any columns that you need as a data type other than the default "string" type.

Windows

There are two options for increasing the memory used by OpenRefine on Windows.

Using openrefine.exe

If you run OpenRefine by clicking on openrefine.exe, you will need to edit the openrefine.l4j.ini file found in the same folder and edit the line

# max memory memory heap size

-Xmx1024M

The line -Xmx1024M defines the amount of memory available to OpenRefine in Megabytes (actually Mebibytes). To increase the memory change the number '1024' to a larger number (e.g. update to -Xmx2048M to make 2048Mb (i.e. 2Gb) of memory available to OpenRefine). In order to use 2Gb of memory, or above, you need to be using a 64-bit version of Java. You will need to ensure that your system is set to use the 64-bit version of Java, otherwise OpenRefine will not start after editing openrefine.l4j.ini.

When starting OpenRefine with openrefine.exe, the version of Java used is that set via the Java control panel (see "How do I enable the latest Java version on my Windows system?" on the Java website.) The version being used must be 64-bit otherwise OpenRefine will not start.

Using refine.bat

On Windows, OpenRefine can also be run by using the file refine.bat which is in the main OpenRefine folder. If you start OpenRefine using refine.bat, the memory available to OpenRefine can be specified either through command line options, or through the refine.ini file.

To set the maximum amount of memory on the command line when starting OpenRefine with refine.bat, open a Windows "command prompt", 'cd' to the folder where OpenRefine is installed, then type

refine.bat /m 2048m

where "2048m" is the maximum amount of memory (in Mb) that you want OpenRefine to use.

To change your default, you can edit refine.ini to specify the memory. You will need to edit the line that reads REFINE_MEMORY=1024M. Note that this file is only read, if you use refine.bat, not openrefine.exe directly.

Once again you need to ensure that a 64-bit version of Java is used if you want to increase the memory to 2048Mb (2Gb) or above. When running OpenRefine from refine.bat, the version of Java used is specified using the JAVA_HOME environment variable. See these instructions on setting the JAVA_HOME environment variable.

Mac

If you have downloaded the ".dmg" package and you start OpenRefine by double clicking on it, follow these instructions

  • close OpenRefine
  • "control-click" on the OpenRefine icon (opens the contextual menu)
  • click on "show package content" (a finder window opens)
  • open the "Contents" folder
  • open the "Info.plist" file with any text editor (like Mac's default TextEdit)
  • simply change “-Xmx1024M” string into something else, eg “-Xmx2048M” or “-Xmx8G”.
  • save the file
  • start OpenRefine again

Linux (or Mac)

If you have downloaded the ".tar.gz" package and you start OpenRefine from the command line, use the "-m xxxM" parameter like this

./refine -m 1024m

The number doesn't need to be a multiple of 1024, but always check that your computer is working properly, leaving a fair amount of RAM for the browser and the operating system to function. To make the change permanent/default, edit the refine.ini file and specify the new value.

If you don't want to set this option on the command line each time, you can also set it in the 'refine.ini' file. Edit the line

REFINE_MEMORY=1024M

Make sure it is not commented out (doesn't start with a '#' character) and change '1024' to a higher value. Save the file, and when you next start OpenRefine it will use this value.

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.