F33 feature/change proposal SwapOnZRAM by default #509
Comments
|
It's in the change proposal, but just to put a fine point on it here: when the /dev/zram0 device is created with whatever size, this is not a preallocation. It doesn't actually consume memory right away. There is about 0.1% overhead to create it, but otherwise the memory is dynamically allocated and deallocated based on demand. |
|
Note that Kubernetes explicitly fails if swap is enabled: kubernetes/kubernetes#53533 Of course this swap isn't really the same as other swap, particularly if you're doing swap on any kind of rotational storage (but hopefully no one is doing that anymore). Personally, I think zram is a convenient and cheap approach in some scenarios, but what we really want is to make the operating system behave more like iOS/Android by default and actively evict (i.e. kill) applications (and yes, this requires intelligence in the frame work and apps). Scheduling applications more intelligently is basically what Kubernetes is doing. |
|
From my reading, Android is uses swap-on-zram already for a while, as well as Chromium/Chrome OS. It's not consistently deployed by OEMs I guess. |
|
"Swap can make a system slower to OOM kill". I don't know if this concern is why cloud environments tend to not have swap configured. But oomd2 and likely future systemd-oomd depends on various PSI metrics including swap pressure, i.e. swap needs to exist to do this. I think using zram based swap for this is an open question; I did ask some upstream kernel cgroupsv2 folks about it, and they kinda shrugged and said it all depends, and may even need to be made dynamic based on the workload. |
|
We covered this in the last meeting. There were several thumbs-up on both not having swap by default (i.e. current status) and allowing users to opt-in swap-on-zram (i.e. the F33 way, minus the always-on default). The general flow that we could be targeting is:
For this to work, we assume that FCOS can generally just follow vanilla Fedora approach here. The only point of contention/customization would be around vendor defaults. @cmurf I didn't see the change-ticket for this F33 feature, but IMHO it would be nice to bring up systemd/zram-generator#29 as a soft-blocker there. PS: I'm deliberately ignoring the whole "is swap default better on or off" discussion here. I'd like to keep this ticket focused on the swap-on-zram topic. |
|
@lucab The feature is still brand new, so it doesn't have a change tracking bug yet and hasn't yet been approved by FESCo. I agree with the approch in zram-generator#29 but leave it up to CoreOS folks to decide whether to ship a missing |
|
For reference, the discussion here brought to light CVE-2020-10781 (local DoS, fix upcoming). |
|
Nice work Luca! |
|
systemd/zram-generator#33 added support for configuration fragments. If/once this proposal land in Fedora, we just need to add a vendor fragment to disable the default configuration, and then document how people can opt-in again into that. |
lucab
added
area/docs
kind/new-feature
fallout/f33
and removed
meeting
|
I think what you'd do is install |
|
Now that I know a little bit more about how swap on zram works i think we could consider enabling it by default after getting some real world experience with it. It could also be something we do at a later time (opt in for now, default to later). It would be nice if it could give us some wins in environments with less resources. |
|
Fedora IoT has been enabling it since the start. These defaults are a bit more conservative considering the 4G cap, which are subject to change. I've talked to a few kernel fs/storage/mm/cgroups folks about it and it's definitely better than no swap. Eviction at 50% efficacy (based on compression ratio estimate) is not as good as 100% efficacy using disk-based swap; but is still better than 0% which causes anonymous pages to be pinned to memory, and increases the chance of otherwise unnecessary reclaim. So even without swap you can get "swap like" behavior, and repetitive reclaim is expensive. |
|
A bit more information: The configuration for the zram-generator has a setting: So hosts with more than The real question is:
|
|
@dustymabe I'd like to answer "yes/maybe" here, but my understanding is that all higher level orchestration systems (e.g. k8s, nomad, etc.) basically assume a "no". |
|
I think what you're saying is reasonable. |
|
ok so it seems like we are leaning towards included but not enabled by default. We have two options for that that I see:
This means in order to enable the defaults the user just deletes the
In order to enable zram you'd need to create a file at |
Though one thing we could do is create our own FCOS config with |
It seems like it could surprise users if we enable a potentially Kubernetes-breaking feature only on machines with certain amounts of RAM. |
|
Maybe ask Kubernetes users if noswap actually manifests well in practice? It's leaving a lot on the table to either insist on significant overprovision of memory to avoid both the need for page eviction and reclaim, or suffer with reclaim which can be worse than incidental paging, especially when using a memory based swap. The noswap by default policy translates into an expectation to throw more memory at such setups (and pay for it). I think it's better to optimize for the general purpose use cases CoreOS is targeting, rather than papering over Kubernetes design oversight. That is, they fail on swap because they haven't worked out how swap should look with the semantics of guaranteed pods, not because there's some real technical limitation of swap. |
|
We discussed this in the meeting today. |
This was part of a F33 proposed change. We'll include the generator but not the defaults subpackage because we don't want it enabled by default just yet. We'll add docs for users instructing them how to enable it by dropping down a file via Ignition. Closes coreos/fedora-coreos-tracker#509
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
|
The fix for this went into testing stream release |
dustymabe
added
the
status/pending-stable-release
|
The fix for this went into stable stream release |
dustymabe
removed
the
status/pending-stable-release
This was part of a F33 proposed change. We'll include the generator
but not the defaults subpackage because we don't want it enabled by
default just yet. We'll add docs for users instructing them how to
enable it by dropping down a file via Ignition/FCCT. The following
snippet is an example:
```
variant: fcos
version: 1.1.0
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa AAAAB user@example.com
storage:
files:
- path: /etc/systemd/zram-generator.conf
mode: 0644
contents:
inline: |
# This config file enables a /dev/zram0 device with the default settings
[zram0]
```
Closes coreos/fedora-coreos-tracker#509
This will be proposed for Fedora 33 all editions and spins
https://fedoraproject.org/wiki/Changes/SwapOnZRAM
I'd like to make the case that some swap, especially if it's fast, is better than no swap. In the no swap case, if the system comes under any memory pressure, it means the system must resort to reclaim of file pages because it's not possible to evict even inactive anonymous pages due to lack of swap. Such a system starts to do a kind of swap thrashing, which is really not at all swap related since it doesn't exist, but it's this churn of reading file pages on demand, and then almost immediately they get dropped out of memory. Whereas if there were a swap-on-ZRAM device, those inactive anonymous pages would get evicted, compressed, and free up memory to avoid reclaiming file pages.
Other than totally opting out, I think there are two options Fedora CoreOS could consider:
include zram-generator, but not include a configuration file. Without a configuration file, there is no setup of a swap-on-zram device.
include both zram-generator and configuration; but you could have a different configuration than the default if you want to go even more minimalist than proposed. e.g. maybe use a fraction of 20% RAM instead of 50%.
Note that Fedora IoT has been using swap-on-ZRAM for some time and they're defaulting to 50% RAM which is the same as this proposal.
The text was updated successfully, but these errors were encountered: