Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs #82718

Merged
merged 1 commit into from Jul 2, 2020

Conversation

@misuzu
Copy link
Contributor

misuzu commented Mar 16, 2020

Motivation for this change

When building big sd-image on armv7l cptofs is failing with this error:

can't open fsimg /nix/store/aal5c4ia7x587kmy0x9swna8bhqd8g92-ext4-fs.img: Value too large for defined data type
21119 Segmentation fault      cptofs -t ext4 -i $out nix-path-registration /

Fix this issue by dropping cptofs from script and using -d argument of mkfs.ext4 instead as lkl is not maintained and takes a very long time to build.

Also make image creation more reproducible.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc @samueldr @grahamc

@misuzu
Copy link
Contributor Author

misuzu commented May 4, 2020

This should also fix building on x86_64 using boot.binfmt.emulatedSystems = [ "aarch64-linux" ];:

nix-build nixos -I nixos-config=nixos/modules/installer/cd-dvd/sd-image-aarch64.nix -A config.system.build.sdImage --argstr system aarch64-linux

Before:

building '/nix/store/0ls2xbc6na607bh6lhdnq7zx6j2g95y2-ext4-fs.img.zst.drv'...
Creating an EXT4 image of 2773430272 bytes (numInodes=94009, numDataBlocks=489089)
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mke2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Discarding device blocks: done
Creating filesystem with 677107 4k blocks and 169344 inodes
Filesystem UUID: 44444444-4444-4444-8888-888888888888
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

error while reading directory .: Cannot allocate memory
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
error while reading directory ./nix/store: Cannot allocate memory
error while reading directory ./nix: Cannot allocate memory
error while reading directory .: Cannot allocate memory
copying store paths to image...
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib/lib/named: Cannot allocate memory
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib/lib: Cannot allocate memory
error while reading directory /nix/store/03nsv3a6a81ms8dsak74gb2jaqrn5llf-bind-9.14.11-lib: Cannot allocate memory
error while reading directory /nix/store: Cannot allocate memory
....
etc, etc

After:

building '/nix/store/0jlr2r3cxjcixp22ii3lgmywzwlc25pb-ext4-fs.img.zst.drv'...
copying store paths for image...
copying files for image...
Creating an EXT4 image of 2681835520 bytes (numInodes=94013, numDataBlocks=466719)
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object 'libfakeroot.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mke2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Discarding device blocks: done
Creating filesystem with 654745 4k blocks and 163840 inodes
Filesystem UUID: 44444444-4444-4444-8888-888888888888
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Copying files into the device: done
Writing superblocks and filesystem accounting information: done

e2fsck 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
NIXOS_SD: 93759/163840 files (0.2% non-contiguous), 526170/654745 blocks
Resizing to minimum allowed size
resize2fs 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Resizing the filesystem on temp.img to 630560 (4k) blocks.
The filesystem on temp.img is now 630560 (4k) blocks long.

e2fsck 1.45.5 (07-Jan-2020)
Unsupported ioctl: cmd=0xffffffff80200204
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
NIXOS_SD: 93759/163840 files (0.2% non-contiguous), 526170/630560 blocks
Compressing image
*** zstd command line interface 64-bits v1.4.4, by Yann Collet ***
./temp.img           : 23.13%   (2582773760 => 597319969 bytes, /nix/store/9hf8pz2k4r3b6flh4k6jfsn4q5w41km0-ext4-fs.img.zst)
building '/nix/store/rpx12maw8rqawm1847dja791yfzv1m65-nixos-sd-image-20.09pre-git-aarch64-linux.img.drv'...
Decompressing rootfs image
/nix/store/9hf8pz2k4r3b6flh4k6jfsn4q5w41km0-ext4-fs.img.zst: 2582773760 bytes
Checking that no-one is using this disk right now ... OK

Disk /nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img: 2.45 GiB, 2622619648 bytes, 5122304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x2178694e.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img1: Created a new partition 1 of type 'W95 FAT32' and of size 30 MiB
.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img2: Created a new partition 2 of type 'Linux' and of size 2.4 GiB.
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img3: Done.

New situation:
Disklabel type: dos
Disk identifier: 0x2178694e

Device                                                                                                                                            Boot Start     End Sectors  Size Id Type
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img1      16384   77823   61440   30M  b W95
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img/sd-image/nixos-sd-image-20.09pre-git-aarch64-linux.img2 *    77824 5122303 5044480  2.4G 83 Linu

The partition table has been altered.
Syncing disks.
5044480+0 records in
5044480+0 records out
2582773760 bytes (2.6 GB, 2.4 GiB) copied, 25.4228 s, 102 MB/s
ERROR: ld.so: object '/nix/store/b8spi94bjg0hhdj9mnqsvc5dydcb4w9g-libfaketime-0.9.8/lib/libfaketime.so.1' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
mkfs.fat 4.1 (2017-01-24)
Copying bootcode.bin
Copying config.txt
Copying fixup.dat
Copying fixup4.dat
Copying fixup4cd.dat
Copying fixup4db.dat
Copying fixup4x.dat
Copying fixup_cd.dat
Copying fixup_db.dat
Copying fixup_x.dat
Copying start.elf
Copying start4.elf
Copying start4cd.elf
Copying start4db.elf
Copying start4x.elf
Copying start_cd.elf
Copying start_db.elf
Copying start_x.elf
Copying u-boot-rpi3.bin
fsck.fat 4.1 (2017-01-24)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "mkfs.fat"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
      2048 bytes per cluster
         4 reserved sectors
First FAT starts at byte 2048 (sector 4)
         2 FATs, 16 bit entries
     30720 bytes per FAT (= 60 sectors)
Root directory starts at byte 63488 (sector 124)
       512 root directory entries
Data area starts at byte 79872 (sector 156)
     15321 data clusters (31377408 bytes)
32 sectors/track, 64 heads
         0 hidden sectors
     61440 sectors total
Checking for unused clusters.
firmware_part.img: 20 files, 12115/15321 clusters
61440+0 records in
61440+0 records out
31457280 bytes (31 MB, 30 MiB) copied, 0.306261 s, 103 MB/s
/nix/store/nks2rj2zigpbblr0ar1gwa4zpaw4093m-nixos-sd-image-20.09pre-git-aarch64-linux.img
@misuzu misuzu requested a review from lovesegfault May 4, 2020
@Renha Renha mentioned this pull request May 4, 2020
nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
@misuzu misuzu force-pushed the misuzu:armv7l-ext4-fs-fix branch from f37d063 to f192676 May 4, 2020
Copy link
Contributor

lovesegfault left a comment

Awesome :)

@misuzu misuzu changed the title ext4-fs: fix image creation on armv7l when image is bigger than 2G nixos/lib/make-ext4-fs: use mkfs.ext4 instead of cptofs May 4, 2020
@mebubo
Copy link

mebubo commented May 21, 2020

I have successfully used this in conjunction with #86291 to cross-build an aarch64 image for a raspberry 3 via https://github.com/nix-community/nixos-generators (+ enabling binfmt.emulatedSystems = [ "aarch64-linux" ]; on the host x86_64 system):

nixos-generate --format sd-aarch64 --system aarch64-linux -I nixpkgs=[path-to-nixpkgs] --configuration configuration.nix

Can confirm that this change gets rid of the error while reading directory ... Cannot allocate memory errors.

nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
@misuzu misuzu force-pushed the misuzu:armv7l-ext4-fs-fix branch from f192676 to fd0fe4b May 21, 2020
@nixos-discourse
Copy link

nixos-discourse commented Jun 3, 2020

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/26

Copy link
Contributor

jonringer left a comment

diff LGTM

would like to see what @samueldr has to say, especially with this work making mobile NixOS.

@nixos-discourse
Copy link

nixos-discourse commented Jun 23, 2020

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/cant-boot-into-new-configuration-after-nix-rebuild-raspberry-pi3-b/7812/1

@misuzu misuzu force-pushed the misuzu:armv7l-ext4-fs-fix branch from fd0fe4b to ff6354f Jun 23, 2020
nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
@c00w
c00w approved these changes Jun 29, 2020
@misuzu misuzu force-pushed the misuzu:armv7l-ext4-fs-fix branch from ff6354f to 40ffbce Jun 29, 2020
@c00w
Copy link
Contributor

c00w commented Jun 30, 2020

I poked @samueldr on IRC about reviewing this, and they said they would take a look

Copy link
Member

samueldr left a comment

One minor change and imo this can be merged to unstable then.

nixos/lib/make-ext4-fs.nix Outdated Show resolved Hide resolved
This fixes image creation on armv7l when image is bigger than 2G.
Also fix some reproducibility issues and other cptofs issues.
@misuzu misuzu force-pushed the misuzu:armv7l-ext4-fs-fix branch from 40ffbce to 9ac1ab1 Jul 1, 2020
@misuzu misuzu requested a review from samueldr Jul 1, 2020
@samueldr samueldr merged commit 736c7ca into NixOS:master Jul 2, 2020
13 checks passed
13 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="9ac1ab1"; rev="9ac1ab10c963a86457c2c9b1edf31542ff3737cd"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@blitz blitz mentioned this pull request Jul 25, 2020
1 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants
You can’t perform that action at this time.