Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
FAQ: Allocate More Memory
FAQ: How to allocate more memory?
I increased more memory for OpenRefine but get an error "could not reserve enough space"
If you plan to use more than 3 Gigs of RAM for OpenRefine you need to
and set it as your default
If you are working with large data sets, you'll need to allocate more memory than the provided default. If you get OutOfMemoryError reported or if your operations are taking a very long time to run, you should allocate more memory if possible. You will, however, need to balance that allocation with the needs of your operating system.
Even if you don't get OutOfMemoryError, when you are getting close to running out of heap space (memory), the system will spend more and more time managing the heap (ie garbage collecting) and less time doing useful work on your operations.
A typical best practice is to start with no more than 50% of your available physical memory, since certain OS's will utilize up to 1 Gig or more of physical RAM.
Example: If you have 2 Gigs of memory and run Windows Vista or 7, you might only have 1 gig of physical memory left over. Starting with a setting of 512m (50% of 1 Gig) can avoid disk caching problems processing large files with many rows and columns. (A 200 meg text file of 331,000 rows x 50 columns)
Side note about dealing with larger datasets
If your project is big enough to need more than the default amount of memory, you probably also want to consider turning off "Parse cell text into numbers, dates, ..." on import as well. It's convenient, but less efficient than explicitly converting any columns that you need as a data type other than the default "string" type.
There are two options for increasing the memory used by OpenRefine on Windows.
If you run OpenRefine by clicking on
openrefine.exe, you will need to edit the openrefine.l4j.ini file found in the same folder and edit the line
# max memory memory heap size -Xmx1024M
-Xmx1024M defines the amount of memory available to OpenRefine in Megabytes (actually Mebibytes). To increase the memory change the number '1024' to a larger number (e.g. update to
-Xmx2048M to make 2048Mb (i.e. 2Gb) of memory available to OpenRefine). In order to use 2Gb of memory, or above, you need to be using a 64-bit version of Java. You will need to ensure that your system is set to use the 64-bit version of Java, otherwise OpenRefine will not start after editing openrefine.l4j.ini.
When starting OpenRefine with openrefine.exe, the version of Java used is that set via the Java control panel (see "How do I enable the latest Java version on my Windows system?" on the Java website.) The version being used must be 64-bit otherwise OpenRefine will not start.
On Windows, OpenRefine can also be run by using the file
refine.bat which is in the main OpenRefine folder. If you start OpenRefine using refine.bat, the memory available to OpenRefine can be specified either through command line options, or through the refine.ini file.
To set the maximum amount of memory on the command line when starting OpenRefine with refine.bat, open a Windows "command prompt", 'cd' to the folder where OpenRefine is installed, then type
refine.bat /m 2048m
where "2048m" is the maximum amount of memory (in Mb) that you want OpenRefine to use.
To change your default, you can edit refine.ini to specify the memory. You will need to edit the line that reads
REFINE_MEMORY=1024M. Note that this file is only read, if you use refine.bat, not openrefine.exe directly.
Once again you need to ensure that a 64-bit version of Java is used if you want to increase the memory to 2048Mb (2Gb) or above. When running OpenRefine from refine.bat, the version of Java used is specified using the JAVA_HOME environment variable. See these instructions on setting the JAVA_HOME environment variable.
If you have downloaded the ".dmg" package and you start OpenRefine by double clicking on it, follow these instructions
- close OpenRefine
- "control-click" on the OpenRefine icon (opens the contextual menu)
- click on "show package content" (a finder window opens)
- open the "Contents" folder
- open the "Info.plist" file with any text editor (like Mac's default TextEdit)
- simply change “-Xmx1024M” string into something else, eg “-Xmx2048M” or “-Xmx8G”.
- save the file
- start OpenRefine again
Linux (or Mac)
If you have downloaded the ".tar.gz" package and you start OpenRefine from the command line, use the "-m xxxM" parameter like this
./refine -m 1024m
The number doesn't need to be a multiple of 1024, but always check that your computer is working properly, leaving a fair amount of RAM for the browser and the operating system to function. To make the change permanent/default, edit the refine.ini file and specify the new value.
If you don't want to set this option on the command line each time, you can also set it in the 'refine.ini' file. Edit the line
Make sure it is not commented out (doesn't start with a '#' character) and change '1024' to a higher value. Save the file, and when you next start OpenRefine it will use this value.