Skip to content

Commit

Permalink
Improve memory workaround docs (#2591)
Browse files Browse the repository at this point in the history
  • Loading branch information
PeterJohnson committed Mar 2, 2024
1 parent 3708a45 commit e6a6f74
Showing 1 changed file with 54 additions and 12 deletions.
66 changes: 54 additions & 12 deletions source/docs/software/basic-programming/java-gc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,13 @@ If you anticipate your application creating a large number of short-lived object
Fixing Out of Memory Errors
---------------------------

If the JVM cannot allocate memory, the program will be terminated. As an embedded system with only a small amount of memory available (256 MB on the roboRIO 1, 512 MB on the roboRIO 2), the roboRIO is particularly susceptible to running out of memory. If you continue to run out of memory even after investigating with VisualVM and taking steps to minimize the number of allocated objects, a few different options are available to make additional memory available to the robot program.
If the JVM cannot allocate memory, the program will be terminated. As an embedded system with only a small amount of memory available (256 MB on the roboRIO 1, 512 MB on the roboRIO 2), the roboRIO is particularly susceptible to running out of memory.

.. admonition :: No amount of system tuning can fix out of memory errors caused by out-of-control allocations.
If you are running out of memory, always investigate allocations with :doc:`VisualVM </docs/software/advanced-gradlerio/profiling-with-visualvm>` first.
If you continue to run out of memory even after investigating with VisualVM and taking steps to minimize the number of allocated objects, a few different options are available to make additional memory available to the robot program.

- Disabling the system web server
- Setting sysctls (Linux kernel options)
Expand All @@ -47,26 +53,48 @@ Implementing most of these options require :doc:`connecting with SSH </docs/soft
Disabling the System Web Server
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The built-in NI system web server provides the webpage seen when using a web browser to connect to the roboRIO. It also is used by the Driver Station's data log download functionality. However, it consumes several MB of RAM, so disabling it will free up that memory for the robot program to use. There are several ways to disable the web server:
The built-in NI system web server provides the webpage (the :doc:`roboRIO Web Dashboard </docs/software/roborio-info/roborio-web-dashboard>`) seen when using a web browser to connect to the roboRIO, e.g. to change IP address settings. It also is used by the Driver Station's data log download functionality. However, it consumes several MB of RAM, so disabling it will free up that memory for the robot program to use. There are several ways to disable the web server:

The first and easiest is to use the RoboRIO Team Number Setter tool. Versions 2024.2.1 and later of the tool have a button to disable or enable the web server. However, a few teams have reported that this does not work or does not persist between reboots. There are two alternate ways to disable the web server; both require connecting to the roboRIO with SSH and logging in as the ``admin`` user.
The first and easiest is to use the :doc:`RoboRIO Team Number Setter </docs/software/wpilib-tools/roborio-team-number-setter/index>` tool. Versions 2024.2.1 and later of the tool have a button to disable or enable the web server. However, a few teams have reported that this does not work or does not persist between reboots. There are two alternate ways to disable the web server; both require connecting to the roboRIO with SSH and logging in as the ``admin`` user.

1. Run ``/etc/init.d/systemWebServer stop; update-rc.d -f systemWebServer remove; sync``

2. Run ``chmod a-x /usr/local/natinst/etc/init.d/systemWebServer``
2. Run ``chmod a-x /usr/local/natinst/etc/init.d/systemWebServer; sync``

To revert the alternate ways and re-enable the web server, take the corresponding step:

1. Run ``update-rc.d -f systemWebServer defaults; /etc/init.d/systemWebServer start; sync``

2. Run ``chmod a+x /usr/local/natinst/etc/init.d/systemWebServer; sync``

Setting sysctls
^^^^^^^^^^^^^^^

Several Linux kernel options (called sysctls) can be set to tweak how the kernel allocates memory. Several options have been found to reduce out-of-memory errors. Setting these options requires connecting to the roboRIO with SSH and logging in as the ``admin`` user, then running the following commands:
Several Linux kernel options (called sysctls) can be set to tweak how the kernel allocates memory. Several options have been found to reduce out-of-memory errors:

- Setting ``vm.overcommit_memory`` to 1 (the default value is 2). This causes the kernel to always pretend there is enough memory for a requested memory allocation at the time of allocation; the default setting always checks to see if there's actually enough memory to back an allocation at the time of allocation, not when the memory is actually used.
- Setting ``vm.vfs_cache_pressure`` to 1000 (the default value is 100). Increasing this causes the kernel to much more aggressively reclaim file system object caches; it may slightly degrade performance.
- Setting ``vm.swappiness`` to 100 (the default value is 60). This causes the kernel to more aggressively swap process memory to the swap file. Changing this option has no effect unless you add a swap file.

You can set some or all of these options; the most important one is ``vm.overcommit_memory``. Setting these options requires connecting to the roboRIO with SSH and logging in as the ``admin`` user, then running the following commands:

.. code-block:: text
echo "swappiness=100" > /etc/sysctl.conf
echo "vfs_cache_pressure=1000" >> /etc/sysctl.conf
echo "overcommit_memory=1" >> /etc/sysctl.conf
echo "vm.overcommit_memory=1" >> /etc/sysctl.conf
echo "vm.vfs_cache_pressure=1000" >> /etc/sysctl.conf
echo "vm.swappiness=100" >> /etc/sysctl.conf
sync
The ``/etc/sysctl.conf`` file should contain the following lines at the end when done (to check, you can run the command ``cat /etc/sysctl.conf``):

.. code-block:: text
vm.overcommit_memory=1
vm.vfs_cache_pressure=1000
vm.swappiness=100
To revert the change, edit ``/etc/sysctl.conf`` (this will require the use of the vi editor) and remove these 3 lines.

Periodically Calling the Garbage Collector
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand All @@ -90,16 +118,30 @@ Sometimes the garbage collector won't run frequently enough to keep up with the
Setting Up Swap on a USB Flash Drive
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

A swap file on a Linux system provides disk-backed space that can be used by the system as additional virtual memory to put infrequently used data and programs when they aren't being used, freeing up physical RAM for active use such as the robot program. It is strongly recommended to not use the built-in non-replaceable flash storage on the roboRIO 1 for a swap file, as it has very limited write cycles and may wear out quickly. Instead, however, a FAT32-formatted USB flash drive may be used for this purpose. This does require the USB flash drive to always be plugged into the roboRIO before boot. A swap file can be set up by plugging the USB flash drive into the roboRIO USB port, connecting to the roboRIO with SSH and logging in as the ``admin`` user, and running the following commands:
A swap file on a Linux system provides disk-backed space that can be used by the system as additional virtual memory to put infrequently used data and programs when they aren't being used, freeing up physical RAM for active use such as the robot program. It is strongly recommended to not use the built-in non-replaceable flash storage on the roboRIO 1 for a swap file, as it has very limited write cycles and may wear out quickly. Instead, however, a FAT32-formatted USB flash drive may be used for this purpose. This does require the USB flash drive to always be plugged into the roboRIO before boot.

.. caution:: Having a swap file on a USB stick means it's critical the USB stick stay connected to the roboRIO at all times it is powered.

This should be used as a last resort if none of the other steps above help. Generally needing swap is indicative of some other allocation issue, so use VisualVM first to optimize allocations.

A swap file can be set up by plugging the USB flash drive into the roboRIO USB port, connecting to the roboRIO with SSH and logging in as the ``admin`` user, and running the following commands. Note the vi step requires knowledge of how to edit and save a file in vi.

.. code-block:: text
fallocate -l 100M /u/swapfile
mkswap /u/swapfile
swapon /u/swapfile
echo "#!/bin/sh" > /etc/init.d/addswap.sh
echo "[ -x /sbin/swapon ] && swapon --ifexists /u/swapfile" >> /etc/init.d/addswap.sh
echo ": exit 0" >> /etc/init.d/addswap.sh
vi /etc/init.d/addswap.h
chmod a+x /etc/init.d/addswap.sh
update-rc.d -v addswap.sh defaults
sync
The ``/etc/init.d/addswap.sh`` file contents should look like this:

.. code-block:: text
#!/bin/sh
[ -x /sbin/swapon ] && swapon -e /u/swapfile
: exit 0
To revert the change, run ``update-rc.d -f addswap.sh remove; rm /etc/init.d/addswap.sh; sync; reboot``.

0 comments on commit e6a6f74

Please sign in to comment.