Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---
title: Increasing Linux Kernel Page Size on Arm
title: Explore Performance Gains by Increasing the Linux Kernel Page Size on Arm

draft: true
cascade:
draft: true

minutes_to_complete: 30

who_is_this_for: This Learning Path is for developers who want to modify the Linux kernel page size on Arm-based systems to improve performance for memory-intensive workloads.
who_is_this_for: This is an introductory topic for developers who want to modify the Linux kernel page size on Arm-based systems to improve performance for memory-intensive workloads.

learning_objectives:
- Verify the current page size on your system.
Expand All @@ -16,20 +16,12 @@ learning_objectives:
- Revert to the default 4K page size kernel (optional).

prerequisites:
- Arm-based Linux system
- Ubuntu [20.04 LTS or newer](https://releases.ubuntu.com/20.04/)
- Debian [11 “Bullseye” or newer](https://www.debian.org/releases/bullseye/)
- CentOS [9 or newer](https://www.centos.org/download/)
- An Arm-based Linux system running Ubuntu, Debian, or CentOS.

author:
- Geremy Cohen

layout: learning-path
author: Geremy Cohen

skill_level: Intermediate
skill_level: Introductory
subjects: Performance and Architecture
cloud_service_providers: Google Cloud

armips:
- Neoverse
Expand All @@ -40,23 +32,27 @@ operatingsystems:
tools_software_languages:
- bash

further reading:
- resource:
title: Page (computer memory) – Wikipedia
link: https://en.wikipedia.org/wiki/Page_(computer_memory)
type: documentation
- resource:
title: Debian Kernel Source Guide
link: https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_kernel_source
type: documentation
- resource:
title: Ubuntu Kernel Build Docs
link: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
type: documentation
- resource:
title: CentOS Kernel Modules Guide
link: https://docs.centos.org/en-US/centos/install-guide/kernel-modules/
type: documentation
further_reading:
- resource:
title: Understanding Memory Page Sizes on Arm64
link: https://amperecomputing.com/tuning-guides/understanding-memory-page-sizes-on-arm64
type: documentation
- resource:
title: Page (computer memory) – Wikipedia
link: https://en.wikipedia.org/wiki/Page_(computer_memory)
type: documentation
- resource:
title: Debian Kernel Source Guide
link: https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_kernel_source
type: documentation
- resource:
title: Ubuntu Kernel Build Docs
link: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
type: documentation
- resource:
title: CentOS Documentation
link: https://docs.centos.org/
type: documentation

### FIXED, DO NOT MODIFY
# ================================================================================
Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
---
title: Centos Page Size Modification
title: Change page size on CentOS
weight: 5
### FIXED, DO NOT MODIFY
layout: learningpathall
---

### Verify current page size
Verify you’re on a 4 KB pagesize kernel by entering the following commands:
Follow the steps below to install a 64K page size kernel on [CentOS 9 or newer](https://www.centos.org/download/).

## Verify the current page size

Verify you’re using a 4 KB pagesize kernel by entering the following commands:

```bash
getconf PAGESIZE
uname -r
```
The output should be similar to below -- the full kernel name may vary, but the first line should always be **4096**:

The output should be similar to below. The kernel flavor (the string after the version number) may vary, but the first line should always be 4096.

```output
4096
Expand All @@ -21,15 +25,15 @@ The output should be similar to below -- the full kernel name may vary, but the

The 4096 indicates the current page size is 4KB. If you see a value that is different, you are already using a page size other than 4096 (4K). On Arm systems, the valid options are 4K, 16K, and 64K.

### Install the kernel-64k package:
## Install the 64k kernel package:

Enter the below `dnf` command to install the 64k kernel:
Enter the command below to install the 64k kernel:

```bash
sudo dnf -y install kernel-64k
```
```bash
sudo dnf -y install kernel-64k
```

You should see a page or so of similar output ending with:
You should see a page of output ending with:

```output
...
Expand All @@ -40,32 +44,31 @@ Installed:
Complete!
```

### Set the kernel-64k as the default kernel and reboot

Enter the following to set the newly installed 64K kernel as default and reboot:
Enter the following to configure the 64K kernel as default and reboot:

```bash
k=$(echo /boot/vmlinuz*64k)
sudo grubby --set-default "$k" --update-kernel "$k"
sudo reboot
```

### Verify the page size and kernel version:
## Verify the page size and kernel version:

Upon reboot, check the kernel page size and name once again to confirm the changes:

```bash
getconf PAGESIZE
uname -r
```

The output should be similar to below -- like before, the full kernel name may vary, but the first line should always be **65536**:
The output shows the 64k kernel is running:

```output
65536
5.14.0-583.el9.aarch64+64k
```

## Reverting back to the original 4K kernel on CentOS
## Revert back to the 4K kernel

To revert to the original 4K kernel, enter the following:

Expand Down Expand Up @@ -103,7 +106,8 @@ Upon reboot, verify you’re on a 4 KB pagesize kernel by entering the following
getconf PAGESIZE
uname -r
```
The output should be similar to below -- the full kernel name may vary, but the first line should always be **4096**:

The output shows the 4k kernel is running:

```output
4096
Expand Down
Original file line number Diff line number Diff line change
@@ -1,20 +1,30 @@
---
title: Debian Page Size Modification
title: Change page size on Debian
weight: 4
### FIXED, DO NOT MODIFY
layout: learningpathall
---

Debian does not provide a 64K kernel via apt, so you will need to compile it from source. There are two ways to do this: 1) download the source from the kernel.org website, or 2) use the Debian source package. This guide will use the Debian source package.
Follow the steps below to install a 64K page size kernel on [Debian 11 “Bullseye” or newer](https://www.debian.org/releases/bullseye/).

### Verify current page size
Verify you’re on a 4 KB pagesize kernel by entering the following commands:
Debian does not provide a 64K kernel package, so you will need to compile it from source.

There are two ways to do this:
- Download the source from kernel.org.
- Use the Debian source package.

The instructions below use the Debian source package.

## Verify the current page size

Verify you’re using a 4 KB pagesize kernel by entering the following commands:

```bash
getconf PAGESIZE
uname -r
```
The output should be similar to below -- the full kernel name may vary, but the first line should always be **4096**:

The output should be similar to below. The kernel flavor (the string after the version number) may vary, but the first line should always be 4096.

```output
4096
Expand All @@ -23,20 +33,20 @@ The output should be similar to below -- the full kernel name may vary, but the

The 4096 indicates the current page size is 4KB. If you see a value that is different, you are already using a page size other than 4096 (4K). On Arm systems, the valid options are 4K, 16K, and 64K.

### Install from Debian Source Package
## Install the Debian kernel source package

To install a 64K page size kernel via package manager, follow these steps:
Follow the steps below to install a 64K kernel using the Debian kernel source package.

First, update, and install dependencies:
First, update, and install the required software:

```bash
sudo apt-get -y update
sudo apt-get -y install git build-essential autoconf automake libtool libncurses-dev bison flex libssl-dev libelf-dev bc debhelper-compat rsync
```

Download the kernel and cd to its directory:
```bash
Download the kernel source and cd to its directory:

```bash
# Fetch the actual kernel source
apt source linux
# Change to kernel source dir
Expand All @@ -46,6 +56,7 @@ cd -- linux*/
## Build and install the kernel

Now that you have the kernel source, follow these steps to build and install the kernel:

```bash
# Use running config as template for new config
cp /boot/config-$(uname -r) .config
Expand All @@ -70,25 +81,28 @@ sudo dpkg -i linux-image-*64k*.deb linux-headers-*64k*.deb
```

The system is now ready to reboot:

```bash
sudo reboot
```

Upon reboot, check the kernel page size and name once again to confirm the changes:

```bash
getconf PAGESIZE
uname -r
```

The output should be similar to below -- like before, the full kernel name may vary, but the first line should always be **65536**:
The output shows the 64k kernel is running:

```output
65536
6.12.22-64k
```

This indicates the current page size is 64K, and you are using the new custom made 64k kernel.

## Reverting back to the original 4K kernel
## Revert back to the 4K kernel

To revert back to the kernel we started with, enter:

Expand All @@ -105,6 +119,7 @@ Upon reboot, verify you’re on a 4 KB pagesize kernel by entering the following
getconf PAGESIZE
uname -r
```

The output should be similar to below -- the full kernel name may vary, but the first line should always be **4096**:

```output
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,32 @@ weight: 2
layout: learningpathall
---

## How the CPU Locates Your Data
## How does the CPU locate data in memory?

When your program asks for a memory address, the CPU doesn’t directly reach into RAM or swap space for it; that would be slow, unsafe, and inefficient.

Instead, it goes through the **virtual memory** system, where it asks for a specific chunk of memory called a **page**. Pages map virtual memory location to physical memory locations in RAM or swap space.
Instead, it goes through the virtual memory system, where it asks for a specific chunk of memory called a page. Pages map virtual memory locations to physical memory locations in RAM or swap space.

## What’s a Memory “Page”?
## What’s a memory page?

Think of your computer’s memory like a big sheet of graph paper. Each **page** is one square on that sheet. The **page table** is the legend that tells the computer which square (virtual address) maps to which spot in real RAM. On x86, 4K is the only pagesize option, but Arm-based systems allow you to set 4K, 16K, or 64K page sizes to fine tune performance of your applications. This tutorial only covers switching between 4K and 64K page sizes.
Think of your computer’s memory like a big sheet of graph paper. Each page is one square on that sheet. The page table is the legend that identifies which square (virtual address) maps to which spot in physical RAM. On x86, 4K is the only page size option, but Arm-based systems allow you to use 4K, 16K, or 64K page sizes to fine tune the performance of your applications.

This Learning Path explains how to switch between 4K and 64K pages on different Linux distributions.

## When to Choose Which
## How should I select the memory page size?

- **4 KB pages** are the safe, default choice. They let you use memory in small slices and keep waste low. Since they are smaller, you need more of them when handling larger memory footprint applications. This creates more overhead for the CPU to manage, but may be worth it for the flexibility. They are great for applications that need to access small bits of data frequently, like web servers or databases with lots of small transactions.
Points to consider when thinking about page size:

- **64 KB pages** shine when you work with big, continuous data—like video frames or large database caches—because they cut down on management overhead. They can waste more memory if you don’t use the whole page, but they can also speed up access times for large data sets.
- **4K pages** are the safe, default choice. They let you use memory in small slices and keep waste low. Since they are smaller, you need more of them when handling larger memory footprint applications. This creates more overhead for the operating system to manage, but it may be worth it for the flexibility. They are great for applications that need to access small bits of data frequently, like web servers or databases with lots of small transactions.

When rightsizing your page size, its important to **try both** under real-world benchmarking conditions, as it will depend on the data size and retrieval patterns of the data you are working with. In addition, the page size may need to be adjusted over time as the application, usage patterns, and data size changes. Althoug this tutorial only covers switching between 4K and 64K page sizes, you can see from the below table that in some cases, you may find a sweet spot in between 4K and 64K at a 16K page size:
- **64K pages** shine when you work with large, continuous data such as video frames or large database caches because they cut down on management overhead. They will use more memory if you don’t use the whole page, but they can also speed up access times for large data sets.

When selecting your page size, it's important to try both options under real-world conditions, as it will depend on the data size and retrieval patterns of the data you are working with.

In addition, the page size may need to be reviewed over time as the application, memory usage patterns, and data sizes may change.


### Summary of page size differences

| Aspect | 4K Pages | 64K Pages |
|-----------------|--------------------------------------|----------------------------------------|
Expand All @@ -32,18 +39,18 @@ When rightsizing your page size, its important to **try both** under real-world
| **Efficiency** | Needs more entries (more bookkeeping) | Needs fewer entries (less bookkeeping) |
| **Waste** | At most 4 KB unused per page | Up to 64 KB unused if not fully used |

##### Summary of page size differences
This Learning Path covers switching between 4K and 64K page sizes because these are supported by most Arm Linux distributions. In some cases, you may find that 16K page size is a sweet spot for your application, but Linux kernel, hardware, and software support is limited. One example of 16k page size is [Asahi Linux](https://asahilinux.org/).

## Experiment to see which works best for your workload

The best way to determine the impact of page size on application performance is to experiment with both options.

{{% notice Do not test on Production%}}
Modifying the Linux kernel page size can lead to system instability or failure. Perform testing in a non-production environment before applying to production systems.
{{% /notice %}}

This learning path will guide you how to change (and revert back) the page size, so you can begin experimenting to see which fits best. The steps to install the 64K page size kernel are different for each OS, so be sure to select the correct one.

Select the Arm Linux distribution you are using to find out how to install the 64K page size kernel.

- [Ubuntu](../ubuntu)
- [Debian](../debian)
- [CentOS](../centos)

---
- [Ubuntu](/learning-paths/servers-and-cloud-computing/arm_linux_page_size/ubuntu/)
- [Debian](/learning-paths/servers-and-cloud-computing/arm_linux_page_size/debian/)
- [CentOS](/learning-paths/servers-and-cloud-computing/arm_linux_page_size/centos/)
Loading