Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multithreaded "xz" when creating tarballs #1415

Merged
merged 5 commits into from Mar 20, 2020

Conversation

lbrpdx
Copy link

@lbrpdx lbrpdx commented Mar 17, 2020

Changing a little bit the options for "tar" saves me precious minutes on a build.
On my 2-core/4-thread CPU, I typically save about 20 minutes on a x86_64 build.

When I time the creation of boot.tar.xz:

  • tar -cJf (where xz uses only 1 thread):
    real 32m48.042s
    user 32m44.317s
    sys 0m22.580s
  • tar -I 'xz -T0' -cf (as many threads as available cores):
    real 13m53.472s
    user 13m10.625s
    sys 0m19.628s

With CPUs with more cores, the win might even be well higher.

@tcamargo
Copy link

Fine by me. What about also changing squash fs compression to zstd? Decompression is really fast.

@lbrpdx
Copy link
Author

lbrpdx commented Mar 18, 2020

Where is that done, @tcamargo?

@tcamargo
Copy link

Where is that done, @tcamargo?

Add BR2_TARGET_ROOTFS_SQUASHFS4_ZSTD=y to your defconfig.

@lbrpdx
Copy link
Author

lbrpdx commented Mar 18, 2020

Couldn't make before/after measurement, but Zstd is faster for sure. There is an interesting paragraph about the benefits of Zstd for SquashFS on https://engineering.fb.com/core-data/zstandard/

@tcamargo
Copy link

I love when someone remove items. from my todo list. ;)

@lbrpdx
Copy link
Author

lbrpdx commented Mar 18, 2020

Unfortunately, doesn't boot any more with Zstd

@mistepien
Copy link

mistepien commented Mar 18, 2020

I think this is a problem with kernel. squashfs in normal kernel is not compiled with zstd.

@tcamargo tcamargo merged commit 4bd60e5 into batocera-linux:master Mar 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants