Unable to start swap-create@ service with zstd compressor

[ElXreno]

Could you create a new release? There is a commit in the master branch that fixed the zram creation with the zstd compressor.
In version 0.2.0 you cannot create zram with zstd, the error is as follows:

Jan 10 13:34:34 laptop.local zram-generator[64345]: Error: Failed to configure disk size into /sys/block/zram0/disksize
Jan 10 13:34:34 laptop.local zram-generator[64345]: Caused by:
Jan 10 13:34:34 laptop.local zram-generator[64345]:     Cannot allocate memory (os error 12)

Additional info:
System: Fedora 33 (KDE)
SystemD version: 246.7-2.fc33
zram-generator version: 0.2.0-4.fc33

[ignatenkobrain]

Just curious, which commit?

[ElXreno]

Apparently, there is no fix in the master branch, it's just the configurator works from my console, but the service refuses to start.
Sorry about the mistake. I will rename this issue.

Could you check if this problem is reproduced on your system @ignatenkobrain?

[polarathene]

it's just the configurator works from my console, but the service refuses to start.

I assume you checked the closed issue on zstd with same error message, but if not, you might want to look at this.

[ElXreno]

Thanks for the reply! modprobe -r zram && modprobe zram didn't work for me, but modprobe zstd did!
Maybe we should detect this error, and show a possible solution instead of displaying Cannot allocate memory?

[ElXreno]

This is a very interesting problem, because if you manually run /usr/lib/systemd/system-generators/zram-generator, everything works without problems and without modprobe zstd. This problem only affects the systemd service.

[keszybz]

Do you have selinux enabled? Any avcs?

[keszybz]

Hmm, in my tests, the zstd module is automatically loaded when the device with zstd compression is initialized. We could execute modprobe <compression> for any compression type that is used, but I'd like to understand why it is failing before doing that.

[ElXreno]

Do you have selinux enabled?

Of course. In Fedora SELinux enabled by default.

[polarathene]

Not an issue on new Ubuntu 20.10 VPS instance:

cat /proc/modules | grep -c zstd
# 0

apt-get update && apt install --yes build-essential pkg-config ronn
curl https://sh.rustup.rs -sSf | sh -s -- -y
source $HOME/.cargo/env
git clone https://github.com/systemd/zram-generator
cd zram-generator && make install
cp zram-generator.conf.example /etc/systemd/zram-generator.conf
sed -i 's/compression-algorithm.*/compression-algorithm = zstd/' /etc/systemd/zram-generator.conf
systemctl daemon-reload

$ cat /sys/block/zram0/comp_algorithm
lzo lzo-rle lz4 lz4hc 842 [zstd]

On a new Fedora 33 instance, I add zstd to the default config:

cat /proc/modules | grep -c zstd
# 0

echo "compression-algorithm = zstd" >> /usr/lib/systemd/zram-generator.conf
systemctl restart swap-create@zram0
# Job for swap-create@zram0.service failed because the control process exited with error code.
# See "systemctl status swap-create@zram0.service" and "journalctl -xe" for details.
modprobe zstd
systemctl restart swap-create@zram0

$ cat /sys/block/zram0/comp_algorithm
lzo lzo-rle lz4 lz4hc 842 [zstd]

I noticed that with the Ubuntu setup which is installing from git, that if I initialized with the example config and didn't change to zstd on a fresh install, I could change to zstd and later and restart the service without error. I also added a [zram1] with no config, unloaded module with modprobe -r zstd and after initializing that with no config just defaults, added the zstd compressor to config and restarted the service, no problems.

When stopping the swap-create@zram0 service on Fedora, unloading the zstd module and restarting the service, I get failure again. SELinux is the cause? I also tried installing from git on Fedora, no improvement.

[keszybz]

I raised selinux, but actually I don't think it can be related, since there's no explicit modprobe involved. (And even if an explicit modprobe is done, the generator is running in init_t context, so it should have privs to do modprobe.)

I think a race condition, i.e. a kernel bug, is the most likely explanation. The PR would resolve the issue in that case.

[polarathene]

I think a race condition, i.e. a kernel bug, is the most likely explanation.

I don't get how it would be a race condition? Results are consistent and reproducible between Fedora and Ubuntu as described above. On Fedora zstd is never loaded unless done so explicitly, while Ubuntu was loading it when zstd was not loaded but later configured to.

[keszybz]

I don't get how it would be a race condition?

At the moment when echo zstd >/sys/block/zram0/comp_algorithm returns, I expect zstd module to be loaded by the kernel. This is what happens on my machines, but it clearly doesn't always work this way.

[polarathene]

@keszybz I cloned your branch onto a fresh Fedora 33 instance, installed it (and just to make sure I was running your build, added a println!() statement to view in the log, which it showed up as I restarted the service).

Jan 13 09:59:39 vultr.guest systemd[1]: Starting Create swap on /dev/zram3...
Jan 13 09:59:39 vultr.guest zram-generator[997]: Hellooooooooooooooo
Jan 13 09:59:39 vultr.guest kernel: Can't allocate a compression stream
Jan 13 09:59:39 vultr.guest kernel: zram: Cannot initialise zstd compressing backend
Jan 13 09:59:39 vultr.guest zram-generator[997]: Error: Failed to configure disk size into /sys/block/zram3/disksize
Jan 13 09:59:39 vultr.guest zram-generator[997]: Caused by:
Jan 13 09:59:39 vultr.guest zram-generator[997]:     Cannot allocate memory (os error 12)
Jan 13 09:59:39 vultr.guest systemd[1]: swap-create@zram3.service: Main process exited, code=exited, status=1/FAILURE
Jan 13 09:59:39 vultr.guest systemd[1]: swap-create@zram3.service: Failed with result 'exit-code'.
Jan 13 09:59:39 vultr.guest systemd[1]: Failed to start Create swap on /dev/zram3.
Jan 13 09:59:39 vultr.guest systemd[1]: Dependency failed for Compressed swap on /dev/zram3.
Jan 13 09:59:39 vultr.guest systemd[1]: dev-zram3.swap: Job dev-zram3.swap/start failed with result 'dependency'.

I've verified the branch I cloned does have your commits, so I'm not sure that it fixes that issue with Fedora. modprobe zstd is still required to make it work.

[keszybz]

Jan 13 09:59:39 vultr.guest kernel: Can't allocate a compression stream
Jan 13 09:59:39 vultr.guest kernel: zram: Cannot initialise zstd compressing backend
...
Jan 13 09:59:39 vultr.guest zram-generator[997]:     Cannot allocate memory (os error 12)

OK, so the issue is not because zstd is not loaded, but because kzalloc fails.
The error message is from

	comp = zcomp_create(zram->compressor);
	if (IS_ERR(comp)) {
		pr_err("Cannot initialise %s compressing backend\n",
				zram->compressor);
		err = PTR_ERR(comp);
		goto out_free_meta;
	}

which calls

struct zcomp *zcomp_create(const char *compress)
{
	struct zcomp *comp;
	int error;

	if (!zcomp_available_algorithm(compress))
		return ERR_PTR(-EINVAL);

	comp = kzalloc(sizeof(struct zcomp), GFP_KERNEL);
	if (!comp)
		return ERR_PTR(-ENOMEM);

	comp->name = compress;
	error = zcomp_init(comp);
	if (error) {
		kfree(comp);
		return ERR_PTR(error);
	}
	return comp;
}

If the compression wasn't available, EINVAL would be returned. We get ENOMEM either from kzalloc() or from zcomp_init(). Can you report a kernel bugzilla with your kernel version and other details?

[polarathene]

uname -r:

• Fedora 33: 5.8.15-301.fc33.x86_64
• Ubuntu 20.10: 5.8.0-29-generic

So Fedora has a slightly more updated 5.8 kernel? Presumably if it was kernel related it'd be something to do with Fedora's kernel config more than the kernel version compared to Ubuntu?

Can you report a kernel bugzilla with your kernel version and other details?

Not exactly my forte :)

I don't personally use Fedora, I only created an instance on Vultr to more easily test zram-generator, and then verify reproducibility of the issue here and test your PR to provide feedback. It does seem like it'd be a bug for reporting to Fedora rather than the kernel bugzilla no?

My understanding is that zram considers zstd a supported/available compressor, but that it's knowledge of that support is unrelated to if the zstd module is loaded. It assumes it's available, but doesn't do whatever Ubuntu is doing to detect and load it when needed?

[keszybz]

It doesn't really matter if it's Fedora or Ubuntu: everybody is using the same upstream kernel.
Please sign up under https://bugzilla.kernel.org/show_bug.cgi?id=211173 if you can, this will help the bug get resolved.
I'll close this here, since it doesn't seem to be a problem with the generator itself.

[polarathene]

It doesn't really matter if it's Fedora or Ubuntu: everybody is using the same upstream kernel.

My point is I can't reproduce it on Ubuntu. I'm not sure if Fedora is configured the same or what differences it has such as SELinux which I don't believe the Ubuntu system has enabled.

I'll create an account and submit the reproduction info if it's helpful, it's 1am here, so I'll take care of that after I've rested :)

I'm still skeptical that it's kernel issue, since modprobe zstd fixes the problem. That'd suggest that your rust approach to issue the command wasn't working in the same manner/sequence (perhaps it needed to enforce some delay?). You deleted the relevant commit and I destroyed the VPS that cloned it, so I'm unable to investigate that further.

If your code was loading the module in the same manner as modprobe zstd (modprobe crypto-zstd works too as your code was doing instead), then when I check for the module after the failure, it should have still been loaded...? That'd suggest that the module was not loaded by your rust PR no?

[keszybz]

It's called a "heisenbug" — seemingly unrelated changes cause the bug go away. In this particular case, I assume the kernel is trying to allocate a few pages of contiguous memory, and depending on what else is happening on the machine, sometimes this fails. So the fact that it happens when you do an additional modprobe call one way, but not the other, doesn't really mean anything, because tiny differences in timing can affect reproducibility.

[polarathene]

TL;DR

As suspected, your code was faulty. It made incorrect assumptions, which is why it didn't work when I tested it against Fedora 33. It's not a kernel bug, your code just never loaded the module in the first place, which I tried to raise concern about as a possibility.

The correct solution, is to check /proc/crypto, if the configured module is not listed there, then it should be loaded. Use your commit and replace the get_known_compressions() method with the logic at the bottom of this comment.

You should relay this information to your bug report and close it.

---

@keszybz Morning from NZ :)

I tested out my thoughts from last night, that the problem was related to your code and not the kernel (Ubuntu and Fedora behaviour differences to autoload zstd aside). I removed zram-generator package and built from source (which was initially problematic as all of a sudden my commands weren't working due to new commits on both the Makefile and service names).

An issue with documentation for me is I have no idea where zram-generator <directory> is called/triggered, just that --setup-device and --reset-device are in the systemd generator service. The directory variant is the one that actually runs the run_generators() logic, of which I can't seem to get stdout from println!() statements logging to journalctl in generator.rs like main.rs seems to... I have noticed that it appears to run when I issue systemctl daemon-reload, but as I needed the log output for my naive debugging, I just ran the command directly.

I avoided your logic (which seems to depend on getting the available compressors from an existing initialized zram device in the first place? Explicitly zram0, I haven't checked if that can cause any issues if that were the device you wanted to initialize with zstd, you probably have a better idea there), instead I just called the command to modprobe zstd in the run_generators() method. This loads zstd module as expected (and works with the service upon systemctl daemon-reload).

So I suspected there was an issue in your implementation.

for compression in compressions.into_iter().filter(|c| !known.contains(c)) {}

Here you're filtering out any compression a device has been configured to use against whatever compressors zram0 (if available at this point) reports back as having. As you can see from the reports at the top of this issue and the related one, zstd module can be unloaded but still be reported as a supported compressor for zram, it's not representative of what modules have actually been loaded, just what the zram module is supporting...

Correct me if I'm wrong, but run_generators()(generator.rs) may run before any zram device is initialized and configured? Which run_device_setup(setup.rs) performs the initialization and compression algorithm setting in zram-generator --setup-device call from the service?

If you want to check for loaded kernel modules, we can check /proc/modules like so:

let path = Path::new("/proc/modules");

let content = fs::read_to_string(path).with_context(|| format!("Failed to read {:?}", path))?;

let loaded_modules = content.lines().into_iter().flat_map(|m|{
  m.split_whitespace().next()
}).map(String::from).collect();

Ok(loaded_modules)

This works in a sense, we can know if zstd is loaded or not, but compressors that aren't configured as kernel modules and built into the kernel itself (always available) such as lzo won't show up here... so not ideal. Instead, we can look up the compressors in /proc/crypto:

let path = Path::new("/proc/crypto");

let content = fs::read_to_string(path).with_context(|| format!("Failed to read {:?}", path))?;

let available_crypto: Vec<String> = content.lines().into_iter()
    .filter(|line| line.starts_with("driver"))
    .flat_map(|m|{
      let s = m.rsplit(':').next().expect("should have `:` delimiter");
      s.trim().strip_suffix("-generic")
    })
    .map(String::from)
    .collect();

Ok(available_crypto)

This works well! lzo and others will appear here (including zstd if you've built a kernel that doesn't treat it as a module), if a kernel module like zstd isn't loaded it won't appear here, but when it is loaded, it'll appear in the results 👍

Quick breakdown, cat /proc/crypto will have some output like:

name         : lzo
driver       : lzo-scomp
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : scomp

name         : lzo
driver       : lzo-generic
module       : kernel
priority     : 0
refcnt       : 3
selftest     : passed
internal     : no
type         : compression

You can see there are multiple entries for different types, so you could filter on name and dedupe, or notice they all use -generic and just match the driver name (this pattern works with all current compressors). There might be an existing rust crate if you want something less "hacky", note the amount of fields varies, so parsing the output into individual entries may be more work when we don't need it.

I'm not much of a rust dev, so feel free to improve on the code snippet. strip_suffix() returns an option or None on the given input, so the driver entries without -generic get filtered out via the flat_map() and we get a vec of strings returned as desired, just with all the extra crypto modules that aren't related to compressors (not ideal but this is the most reliable way I could find without complicating logic to associate the compression type).

Here's what returned vec may look like:

["zstd", "ghash", "842", "xxhash64", "lzo-rle", "lzo", "crct10dif", "crc32c", "deflate", "aes", "blake2b-512", "blake2b-384", "blake2b-256", "blake2b-160", "sha384", "sha512", "sha224", "sha256", "sha1", "md5", "digest_null", "compress_null", "cipher_null", "rsa", "dh"]

This approach will also work for lz4 and lz4hc which zram also supports and Fedora doesn't load by default.

[polarathene]

PR submitted.

[keszybz]

Thank you for the extensive investigation.
You are right, my code was checking the wrong thing.

This doesn't mean that there is no kernel bug — the kernel does load the crypt module when necessary, so none of this should be necessary. But if it helps, then that's good. Let's continue the discussion in the pull request.