Memory sizes differences

[nogalac]

I run OSHI libraries on both Windows and Linux.
It seemes to me that it works well on Windows, but not on Linux.
Please, See below what i've noticed.
Do i do something wrong ?

OSHI commands:
hal.getMemory().getTotal())/1024.0/1024.0
hal.getMemory().getAvailable())/1024.0/1024.0

On windows7:
-TaskMgr shows :
- Total: 8075 GB
- Available: 2032 GB

- OSHI shows:
	- Total: 8075 GB
	- Available: 2207 GB

On Linux (red Hat 6/7):
- cat /proc/meminfo shows:
- Total: 11912.35 GB
- Free: 215732 kB

- OSHI shows:
	- Total: 11912.35 GB
	- Available: 2304 GB

On Linux, it looks as if hal.getMemory().getAvailable() returns the used memory instead of free memory.

[YoshiEnVerde]

I'm pretty sure #424 is relevant to this.
Even so, you're mixing things here... Free Memory is not the same as Available memory.

Running the same code in Ubuntu, I get:

• cat meminfo:

  1. MemTotal: 3950112 kB (That's 3857.53MB)
  2. MemAvailable: 568100 kB (That's 554.79MB)

• OSHI:

  1. Total Memory: 3857.53MB
  2. Available Memory: 535.55MB

That looks pretty accurate to me (Only about 20MB difference in the available value, which could be attributed to many things)

In Linux, Available Memory is an amalgam of a few other memory sets, one of which is Free Memory. This includes cached blocks of memory, and such, that aren't strictly free (not allocated) but are available for use.

That's why you get 200MB free vs 2.3GB available. You must have the extra 2.1GB allocated to some Available set of memory.

I'm not up to date on which exact version of everything is in each RHEL build, but just going by dates you might not have the right metrics available in RHEL6, but they should be there in RHEL7

[dbwiddis]

@YoshiEnVerde has answered this but I'll amplify. On Linux the "free" memory slowly approaches zero the longer your system is operating, because memory released by programs returns to a "cached" or "buffer" pool rather than being "free". (It's possible but unnecessary to randomly free it.)

For a trip down memory lane, see #23 which was my first contribution to this project, and I got hooked. :)

[nogalac]

Thanks a lot for your answers. I will continue to investigate.

I would like to get the amount of memory as "top" displays it:

Mem: 12198248k total, 11327824k used, 870424k free, 516600k buffers

I think it is more significant to understand if my linux servers need memory add-on or not.

Thanks and have a nice day.

[dbwiddis]

We will not display the memory as top displays it, because the "free" memory from top is meaningless. Much of the buffered memory is immediately available to applications. Knowing how much is available to you is a much better measure, and that is the field that OSHI reports. You are more than welcome to use the util classes to parse command lines as needed to gather more information.

[Abu-Abdullah]

@dbwiddis
please take a note on the numbers from Raspi Pi:

memory.getAvailable : 118.7 MiB
memory.getTotal : 242.6 MiB

at the same time:

pi@jr-5453cc:~ $ free -k
              total        used        free      shared  buff/cache   available
Mem:         248448       54552       76264       12708      117632      132228
Swap:        102396        8192       94204

there is a difference between the available from the two free command and oshi

[dbwiddis]

What does the Pi say for cat /proc/meminfo? This is what OSHI parses.

Also, were these literally taken at the exact same time? Any program running (including Java, using oshi) may consume some memory, causing small differences.

[spyhunter99]

@dbwiddis any interest in a jenkins cluster to get better os test coverage with controlled hardware parameters? i keep seeing more and more issues like this and it's really a test coverage issue. We can only do so much with travis's linux environment

[dbwiddis]

I am more than happy to test in more places and on more systems. I have very little time to figure out how to set it all up myself! Can you help set it up?

[Abu-Abdullah]

cat /proc/meminfo

another capture of information.

pi@jr-5453cc:~ $ cat /proc/meminfo
MemTotal:         248448 kB
MemFree:           77460 kB
MemAvailable:     136964 kB
Buffers:           20404 kB
Cached:            92928 kB
SwapCached:          776 kB
Active:            86944 kB
Inactive:          58632 kB
Active(anon):       7908 kB
Inactive(anon):    37028 kB
Active(file):      79036 kB
Inactive(file):    21604 kB
Unevictable:          12 kB
Mlocked:              12 kB
SwapTotal:        102396 kB
SwapFree:          94204 kB
Dirty:                72 kB
Writeback:             0 kB
AnonPages:         31560 kB
Mapped:            16620 kB
Shmem:             12692 kB
Slab:              14584 kB
SReclaimable:       7836 kB
SUnreclaim:         6748 kB
KernelStack:         832 kB
PageTables:         1048 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      226620 kB
Committed_AS:     175960 kB
VmallocTotal:     770048 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:               64 kB
CmaTotal:           8192 kB
CmaFree:            3036 kB
pi@jr-5453cc:~ $ free -k
              total        used        free      shared  buff/cache   available
Mem:         248448       49812       77460       12692      121176      136972
Swap:        102396        8192       94204
pi@jr--5453cc:~ $ java -cp .:* SystemInfoTest
Initializing System...
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
GNU/Linux Raspbian GNU/Linux 9 (stretch) build 4.19.42+
Checking computer system...
manufacturer: unknown
model: unknown
serialnumber: unknown
firmware:
  manufacturer: unknown
  name: unknown
  description: unknown
  version: unknown
  release date: unknown
baseboard:
  manufacturer: unknown
  model: unknown
  version: unknown
  serialnumber: unknown
Checking Processor...
ARMv6-compatible processor rev 7 (v6l)
 1 physical CPU package(s)
 1 physical CPU core(s)
 1 logical CPU(s)
Identifier:  Family ? Model ? Stepping ?
ProcessorID: 0000000000000000
Checking Memory...
Memory: 123.4 MiB/242.6 MiB
Swap used: 8 MiB/100.0 MiB
Checking CPU...
Uptime: 2 days, 05:33:42
Context Switches/Interrupts: 19602191 / 10885716
CPU, IOWait, and IRQ ticks @ 0 sec:[1895650, 0, 1055540, 188616580, 328500, 0, 63500, 0]
CPU, IOWait, and IRQ ticks @ 1 sec:[1895680, 0, 1055540, 188617570, 328500, 0, 63500, 0]
User: 2.9% Nice: 0.0% System: 0.0% Idle: 97.1% IOwait: 0.0% IRQ: 0.0% SoftIRQ: 0.0% Steal: 0.0%
CPU load: 2.9% (counting ticks)
CPU load: 1.6% (OS MXBean)
CPU load averages: 0.44 0.15 0.10
CPU load per processor: 57.7%
Max Frequency: 1 GHz
Current Frequencies: 1 GHz
Checking Processes...
Processes: 83, Threads: 114
   PID  %CPU %MEM       VSZ       RSS Name
 28002  85.5 10.0 206.2 MiB  24.3 MiB java
 27972   1.8  1.8   6.3 MiB   4.3 MiB bash
 27959   1.6  2.7  16.2 MiB   6.7 MiB sshd
   585   0.5 11.1 247.1 MiB  27.0 MiB java
 27969   0.2  1.8  16.2 MiB   4.4 MiB sshd
Checking Sensors...
Sensors:
 CPU Temperature: 50.8°C
 Fan Speeds: []
 CPU Voltage: 0.0V
Checking Power sources...
Power: Unknown
Checking Disks...
Disks:
 /dev/mmcblk0: (model: Unknown - S/N: Unknown) size: 7.9 GB, reads: 98226 (1.1 GiB), writes: 0 (391.8 MiB), xfer: 0 ms
 |-- /dev/mmcblk0p1: mmcblk0p1 (vfat) Maj:Min=179:1, size: 45.0 MB @ /boot
 |-- /dev/mmcblk0p2: mmcblk0p2 (ext4) Maj:Min=179:2, size: 7.9 GB
Checking File System...
File System:
 File Descriptors: 994/23856
 / (Local Disk) [ext4] 3.7 GiB of 7.2 GiB free (50.8%), 381114 of 451232 files free (84.5%) is /dev/root  and is mounted at /
 tmpfs (Ram Disk) [tmpfs] 12 MiB of 12 MiB free (100.0%), 31054 of 31056 files free (100.0%) is tmpfs  and is mounted at /home/pi/snapshots_tmp
 /dev/loop0 (Local Disk) [ext4] 8.3 GiB of 9.7 GiB free (85.9%), 2456791 of 2457600 files free (100.0%) is /dev/loop0  and is mounted at /home/pi/snapshots
 /dev/mmcblk0p1 (Local Disk) [vfat] 20.1 MiB of 42.2 MiB free (47.5%), 0 of 0 files free (NaN%) is /dev/mmcblk0p1  and is mounted at /boot
 /home/pi/.jr_cam (Mount Point) [ecryptfs] 3.7 GiB of 7.2 GiB free (50.8%), 381114 of 451232 files free (84.5%) is /home/pi/.jr_cam  and is mounted at /home/pi/.jr_cam
Checking Network interfaces...
Network interfaces:
 Name: wlan0 (wlan0)
   MAC Address: b8:27:eb:54:53:cc
   MTU: 1500, Speed: 0 bps
   IPv4: [192.168.1.29]
   IPv6: []
   Traffic: received 216931 packets/213.0 MiB (0 err); transmitted 74606 packets/29.3 MiB (0 err)
Checking Network parameterss...
Network parameters:
 Host name: jr-5453cc
 Domain name: jr-5453cc
 DNS servers: [192.168.1.1]
 IPv4 Gateway: 192.168.1.1
 IPv6 Gateway:
Checking Displays...
Displays:
Checking USB Devices...
USB Devices:
 DWC OTG Controller (Linux 4.19.42+ dwc_otg_hcd) [s/n: 20980000.usb]
Checking Sound Cards...
Sound Cards:
SoundCard@4aed64 [kernelVersion=Advanced Linux Sound Architecture Driver Version k4.19.42+., name=bcm2835_alsa - bcm2835 ALSA, codec=]

it seems it is different

[cilki]

@spyhunter99 I've thought about testing with Jenkins before, but the problem seems to be finding somewhere to host it (and cost). See oshi/oshi6#5.

[spyhunter99]

I'll open a separate ticket for the jenkins setup

[dbwiddis]

OSHI pulls the MemAvailable value from cat /proc/meminfo. It would have pulled 136964 kB from your output if it were running. This would calculate to 133.8 MiB. However, when you ran OSHI you began running a new java process, which took some memory. I don't know exactly how much it was using when it grabbed the available memory, but we can see later on in the process list that the java process (28002) is by then consuming 24.3 MiB of memory.

OSHI's used/total shows 123.4 MiB/242.6 MiB meaning 119.2 MiB is free, so we've used up 14.6 MB in memory between the command and having oshi fetch the value. Given it goes from 0 to 24.3 and you're running this command somewhere in the middle, this all seems normal to me.

To do a true side by side test you need to have the java process running, outputting the memory at precisely the same time you're checking it independently. And given that on linux, OSHI directly pulls from the output of cat /proc/meminfo it should match identically.