Add free memory detection and Notification to user

[thadguidry]

... on 1st startup of OpenRefine.
Also add better/extra batch script output to warn user of low memory prior to starting Java VM.

This needs to work from Windows, Linux, and OSX.

Windows: C:\Users\Thad>wmic os get FreePhysicalMemory /format:list
Linux: free -m | grep -oP '\d+' | head -n 1
OSX: vm_stat ??? but what additional options to parse out the free mem ? or does top -l 1 | grep ^PhysMem get us closer to that ?

Question: Do we assume 50% of that for OpenRefine use to be a good citizen or can we push it to 75% ?

[jackyq2015]

@thadguidry Auto setting is fine. But leaving the decision to end user might also an option. Your question is hard to answer because only end user knows their situation such as size of data set, complexity etc. Instead of auto set it, a responsive warning message when Out of Mem occurs could helpful

[thadguidry]

"Instead of auto set it, a responsive warning message when Out of Mem occurs could helpful"
@jackyq2015 Indeed, that also is needed.

[ettorerizza]

You are obviously right, @jackyq2015 . But we can safely bet that, nowadays, most users will increase the memory and give it a default level based on their hardware capabilities. They will then increase it on a case-by-case basis. Half of the RAM seems to me a good starting point. I give more personally (about 75%, 12GB out of 16), but I am kind of heavy user.

[magdmartin]

Can we allow the user to set up the memory from the preference page? We can show to the user how much memory is available on the machine and let him/her adjust.

like @jackyq2015 mentioned, I will cautious to auto set the memory as 1) we don't know the size of the data set and 2) we don't know if the user is running and other RAM greedy application

[thadguidry]

@magdmartin on point 2 we do know what the FREE memory is...and taking 50% of that the first time they run OpenRefine shouldn't cause too many problems. Yeah, we can do preference setup.

[thadguidry]

@jackyq2015 Can you confirm that the OSX options above work well enough for us to parse out the FREE memory ?

[jackyq2015]

OXS don't have the free command. But I confirmed this works:
vm_stat | perl -ne '/page size of (\d+)/ and $size=$1; /Pages\s+([^:]+)[^\d]+(\d+)/ and printf("%-16s % 16.2f Mi\n", "$1:", $2 * $size / 1048576);'

Link:
https://apple.stackexchange.com/questions/4286/is-there-a-mac-os-x-terminal-version-of-the-free-command-in-linux-systems

[thadguidry]

@jackyq2015 hmm... there's a Perl dependency... can you confirm if this works also ?
top -l 1 | grep ^PhysMem gets us closer to that ?

[jackyq2015]

Yes. it works:
$ top -l 1 | grep PhysMem PhysMem: 1473M used (439M wired), 4543M unused.
$ top -l 1 | grep PhysMem | awk '{print $6}' 4535M

[jackyq2015]

I just realised there is some issue if we want to implement this:

1. it will automatically set the RAM for user which will disable the possibility to tune the JVM themselves. Unless you provide extra switch to skip the automatically setting or enforce the user setting. That defeats the purpose of the intention to ease the process.
2. The calculated RAM size may too small to too big to have the application launched.
   For example 32 bit system allow max around 4G, but 64 bit system is different. Also the different JVM vendors have its limitation differently. There are lots of combination of OS, Arch and JVM, not even mention about the user preference.

I would suggest that we leave this to end user or only provide the suggestion instead of set it directly.

Thoughts?

[thadguidry]

@jackyq2015 I thought of that also. Let's change the need of this issue then to a notification only.

We inspect (using the 3 OS memory lookups that found to work well above) and then advise the user on startup... "Hey ! We noticed that you have 6 Gigs of Free Memory available....OpenRefine can run better when given more memory. Read our FAQ on how to allocate more memory here: blahURL"

[jackyq2015]

sure. we can do that.

[thadguidry]

This can even be a Javascript popup

[thadguidry]

@jackyq2015 Beautiful. Good Job.

D:\git\OpenRefine>refine
You have 12219M of free memory.
You current configuration will allow to use 1400M of memory.
OpenRefine can run better when given more memory. Read our FAQ on how to allocate more memory here:
https://github.com/OpenRefine/OpenRefine/wiki/FAQ:-Allocate-More-Memory
.
14:47:49.668 [ refine_server] Starting Server bound to '127.0.0.1:3333' (0ms)
14:47:49.670 [ refine_server] refine.memory size: 1400M JVM Max heap: 1407188992 (2ms)
14:47:49.682 [ refine_server] Initializing context: '/' from 'D:\git\OpenRefine\main\webapp' (12ms)

[tfmorris]

No one is ever going to see a couple of random lines of text at the beginning of the server log (which usually isn't even visible). It also doesn't appear to be accurate on OS X where it says:

You have 477M of free memory.
Your current configuration is set to use 1400M of memory.

when I have a couple of GB free even after Refine has started up and if I gave it more memory the operating system would just reclaim more memory from other processes (in other words "free memory" isn't really a good criterium to use).

GC overhead or heap usage might be better feedback for the user and, of course, any feedback needs to be presented through the web interface where they can see it.

[thadguidry]

@tfmorris Yeap. Agree. The text only helped for a small number of users. Getting the web UI to present memory info would be really nice. (hmm, also of note, we get part of that for free now on the spark-prototype branch as part of the Spark's UI memory management at http://<driver>:4040)