Skip to content

Commit

Permalink
Merge branch 'akpm' (patches from Andrew)
Browse files Browse the repository at this point in the history
Merge more updates from Andrew Morton:
 "147 patches, based on 7d2a07b.

  Subsystems affected by this patch series: mm (memory-hotplug, rmap,
  ioremap, highmem, cleanups, secretmem, kfence, damon, and vmscan),
  alpha, percpu, procfs, misc, core-kernel, MAINTAINERS, lib,
  checkpatch, epoll, init, nilfs2, coredump, fork, pids, criu, kconfig,
  selftests, ipc, and scripts"

* emailed patches from Andrew Morton <akpm@linux-foundation.org>: (94 commits)
  scripts: check_extable: fix typo in user error message
  mm/workingset: correct kernel-doc notations
  ipc: replace costly bailout check in sysvipc_find_ipc()
  selftests/memfd: remove unused variable
  Kconfig.debug: drop selecting non-existing HARDLOCKUP_DETECTOR_ARCH
  configs: remove the obsolete CONFIG_INPUT_POLLDEV
  prctl: allow to setup brk for et_dyn executables
  pid: cleanup the stale comment mentioning pidmap_init().
  kernel/fork.c: unexport get_{mm,task}_exe_file
  coredump: fix memleak in dump_vma_snapshot()
  fs/coredump.c: log if a core dump is aborted due to changed file permissions
  nilfs2: use refcount_dec_and_lock() to fix potential UAF
  nilfs2: fix memory leak in nilfs_sysfs_delete_snapshot_group
  nilfs2: fix memory leak in nilfs_sysfs_create_snapshot_group
  nilfs2: fix memory leak in nilfs_sysfs_delete_##name##_group
  nilfs2: fix memory leak in nilfs_sysfs_create_##name##_group
  nilfs2: fix NULL pointer in nilfs_##name##_attr_release
  nilfs2: fix memory leak in nilfs_sysfs_create_device_group
  trap: cleanup trap_init()
  init: move usermodehelper_enable() to populate_rootfs()
  ...
  • Loading branch information
torvalds committed Sep 8, 2021
2 parents cc09ee8 + b285437 commit 2d33820
Show file tree
Hide file tree
Showing 149 changed files with 5,341 additions and 930 deletions.
15 changes: 15 additions & 0 deletions Documentation/admin-guide/mm/damon/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
.. SPDX-License-Identifier: GPL-2.0
========================
Monitoring Data Accesses
========================

:doc:`DAMON </vm/damon/index>` allows light-weight data access monitoring.
Using DAMON, users can analyze the memory access patterns of their systems and
optimize those.

.. toctree::
:maxdepth: 2

start
usage
114 changes: 114 additions & 0 deletions Documentation/admin-guide/mm/damon/start.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
.. SPDX-License-Identifier: GPL-2.0
===============
Getting Started
===============

This document briefly describes how you can use DAMON by demonstrating its
default user space tool. Please note that this document describes only a part
of its features for brevity. Please refer to :doc:`usage` for more details.


TL; DR
======

Follow the commands below to monitor and visualize the memory access pattern of
your workload. ::

# # build the kernel with CONFIG_DAMON_*=y, install it, and reboot
# mount -t debugfs none /sys/kernel/debug/
# git clone https://github.com/awslabs/damo
# ./damo/damo record $(pidof <your workload>)
# ./damo/damo report heat --plot_ascii

The final command draws the access heatmap of ``<your workload>``. The heatmap
shows which memory region (x-axis) is accessed when (y-axis) and how frequently
(number; the higher the more accesses have been observed). ::

111111111111111111111111111111111111111111111111111111110000
111121111111111111111111111111211111111111111111111111110000
000000000000000000000000000000000000000000000000001555552000
000000000000000000000000000000000000000000000222223555552000
000000000000000000000000000000000000000011111677775000000000
000000000000000000000000000000000000000488888000000000000000
000000000000000000000000000000000177888400000000000000000000
000000000000000000000000000046666522222100000000000000000000
000000000000000000000014444344444300000000000000000000000000
000000000000000002222245555510000000000000000000000000000000
# access_frequency: 0 1 2 3 4 5 6 7 8 9
# x-axis: space (140286319947776-140286426374096: 101.496 MiB)
# y-axis: time (605442256436361-605479951866441: 37.695430s)
# resolution: 60x10 (1.692 MiB and 3.770s for each character)


Prerequisites
=============

Kernel
------

You should first ensure your system is running on a kernel built with
``CONFIG_DAMON_*=y``.


User Space Tool
---------------

For the demonstration, we will use the default user space tool for DAMON,
called DAMON Operator (DAMO). It is available at
https://github.com/awslabs/damo. The examples below assume that ``damo`` is on
your ``$PATH``. It's not mandatory, though.

Because DAMO is using the debugfs interface (refer to :doc:`usage` for the
detail) of DAMON, you should ensure debugfs is mounted. Mount it manually as
below::

# mount -t debugfs none /sys/kernel/debug/

or append the following line to your ``/etc/fstab`` file so that your system
can automatically mount debugfs upon booting::

debugfs /sys/kernel/debug debugfs defaults 0 0


Recording Data Access Patterns
==============================

The commands below record the memory access patterns of a program and save the
monitoring results to a file. ::

$ git clone https://github.com/sjp38/masim
$ cd masim; make; ./masim ./configs/zigzag.cfg &
$ sudo damo record -o damon.data $(pidof masim)

The first two lines of the commands download an artificial memory access
generator program and run it in the background. The generator will repeatedly
access two 100 MiB sized memory regions one by one. You can substitute this
with your real workload. The last line asks ``damo`` to record the access
pattern in the ``damon.data`` file.


Visualizing Recorded Patterns
=============================

The following three commands visualize the recorded access patterns and save
the results as separate image files. ::

$ damo report heats --heatmap access_pattern_heatmap.png
$ damo report wss --range 0 101 1 --plot wss_dist.png
$ damo report wss --range 0 101 1 --sortby time --plot wss_chron_change.png

- ``access_pattern_heatmap.png`` will visualize the data access pattern in a
heatmap, showing which memory region (y-axis) got accessed when (x-axis)
and how frequently (color).
- ``wss_dist.png`` will show the distribution of the working set size.
- ``wss_chron_change.png`` will show how the working set size has
chronologically changed.

You can view the visualizations of this example workload at [1]_.
Visualizations of other realistic workloads are available at [2]_ [3]_ [4]_.

.. [1] https://damonitor.github.io/doc/html/v17/admin-guide/mm/damon/start.html#visualizing-recorded-patterns
.. [2] https://damonitor.github.io/test/result/visual/latest/rec.heatmap.1.png.html
.. [3] https://damonitor.github.io/test/result/visual/latest/rec.wss_sz.png.html
.. [4] https://damonitor.github.io/test/result/visual/latest/rec.wss_time.png.html
112 changes: 112 additions & 0 deletions Documentation/admin-guide/mm/damon/usage.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
.. SPDX-License-Identifier: GPL-2.0
===============
Detailed Usages
===============

DAMON provides below three interfaces for different users.

- *DAMON user space tool.*
This is for privileged people such as system administrators who want a
just-working human-friendly interface. Using this, users can use the DAMON’s
major features in a human-friendly way. It may not be highly tuned for
special cases, though. It supports only virtual address spaces monitoring.
- *debugfs interface.*
This is for privileged user space programmers who want more optimized use of
DAMON. Using this, users can use DAMON’s major features by reading
from and writing to special debugfs files. Therefore, you can write and use
your personalized DAMON debugfs wrapper programs that reads/writes the
debugfs files instead of you. The DAMON user space tool is also a reference
implementation of such programs. It supports only virtual address spaces
monitoring.
- *Kernel Space Programming Interface.*
This is for kernel space programmers. Using this, users can utilize every
feature of DAMON most flexibly and efficiently by writing kernel space
DAMON application programs for you. You can even extend DAMON for various
address spaces.

Nevertheless, you could write your own user space tool using the debugfs
interface. A reference implementation is available at
https://github.com/awslabs/damo. If you are a kernel programmer, you could
refer to :doc:`/vm/damon/api` for the kernel space programming interface. For
the reason, this document describes only the debugfs interface

debugfs Interface
=================

DAMON exports three files, ``attrs``, ``target_ids``, and ``monitor_on`` under
its debugfs directory, ``<debugfs>/damon/``.


Attributes
----------

Users can get and set the ``sampling interval``, ``aggregation interval``,
``regions update interval``, and min/max number of monitoring target regions by
reading from and writing to the ``attrs`` file. To know about the monitoring
attributes in detail, please refer to the :doc:`/vm/damon/design`. For
example, below commands set those values to 5 ms, 100 ms, 1,000 ms, 10 and
1000, and then check it again::

# cd <debugfs>/damon
# echo 5000 100000 1000000 10 1000 > attrs
# cat attrs
5000 100000 1000000 10 1000


Target IDs
----------

Some types of address spaces supports multiple monitoring target. For example,
the virtual memory address spaces monitoring can have multiple processes as the
monitoring targets. Users can set the targets by writing relevant id values of
the targets to, and get the ids of the current targets by reading from the
``target_ids`` file. In case of the virtual address spaces monitoring, the
values should be pids of the monitoring target processes. For example, below
commands set processes having pids 42 and 4242 as the monitoring targets and
check it again::

# cd <debugfs>/damon
# echo 42 4242 > target_ids
# cat target_ids
42 4242

Note that setting the target ids doesn't start the monitoring.


Turning On/Off
--------------

Setting the files as described above doesn't incur effect unless you explicitly
start the monitoring. You can start, stop, and check the current status of the
monitoring by writing to and reading from the ``monitor_on`` file. Writing
``on`` to the file starts the monitoring of the targets with the attributes.
Writing ``off`` to the file stops those. DAMON also stops if every target
process is terminated. Below example commands turn on, off, and check the
status of DAMON::

# cd <debugfs>/damon
# echo on > monitor_on
# echo off > monitor_on
# cat monitor_on
off

Please note that you cannot write to the above-mentioned debugfs files while
the monitoring is turned on. If you write to the files while DAMON is running,
an error code such as ``-EBUSY`` will be returned.


Tracepoint for Monitoring Results
=================================

DAMON provides the monitoring results via a tracepoint,
``damon:damon_aggregated``. While the monitoring is turned on, you could
record the tracepoint events and show results using tracepoint supporting tools
like ``perf``. For example::

# echo on > monitor_on
# perf record -e damon:damon_aggregated &
# sleep 5
# kill 9 $(pidof perf)
# echo off > monitor_on
# perf script
1 change: 1 addition & 0 deletions Documentation/admin-guide/mm/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ the Linux memory management.

concepts
cma_debugfs
damon/index
hugetlbpage
idle_page_tracking
ksm
Expand Down
Loading

0 comments on commit 2d33820

Please sign in to comment.