Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] cannot run 'apt-get install' #350

Closed
smoser opened this issue Dec 6, 2022 · 12 comments
Closed

[Bug] cannot run 'apt-get install' #350

smoser opened this issue Dec 6, 2022 · 12 comments
Labels

Comments

@smoser
Copy link
Contributor

smoser commented Dec 6, 2022

stacker version

v0.30.1-5-g1346183 liblxc 074b9fe66304fb05e07554d01353c8d2e53e392a

Describe the bug

No response

To reproduce

Given this stacker.yaml

$ cat stacker.yaml
smtest:
  from:
    type: docker
    url: "docker://aci-zot.cisco.com:5000/c3/minbase:1.0.32${{LAYER_SUFFIX}}"
  run: |
    id
    touch /var/lib/dpkg/myfile && echo "/var/lib/dpkg is writable"
    ls -l /var/lib/dpkg/
    apt-get install hello

This will work fine:

* `stacker build --layer-type=tar --substitute=LAYER_SUFFIX="" `

But attempting squashfs will fail as shown:

$ ./stacker --debug build --layer-type=squashfs --substitute=LAYER_SUFFIX=-squashfs
stacker version v0.30.1-5-g1346183
usernsexec-ing [u 0 1001 1 1 231073 65535 g 0 1001 1 1 231073 65535 -- /tmp/stacker --internal-userns --debug build --layer-type=squashfs --substitute=LAYER_SUFFIX=-squashfs]
stacker version v0.30.1-5-g1346183
initializing stacker recipe: stacker.yaml
substituting $LAYER_SUFFIX to -squashfs
substituting $STACKER_ROOTFS_DIR to /tmp/t1/roots
substituting $STACKER_STACKER_DIR to /tmp/t1/.stacker
substituting $STACKER_OCI_DIR to /tmp/t1/oci
stacker build order:
0 build /tmp/t1/stacker.yaml: requires: []
building: 0 /tmp/t1/stacker.yaml
substituting $LAYER_SUFFIX to -squashfs
substituting $STACKER_ROOTFS_DIR to /tmp/t1/roots
substituting $STACKER_STACKER_DIR to /tmp/t1/.stacker
substituting $STACKER_OCI_DIR to /tmp/t1/oci
Dependency Order [smtest]
preparing image smtest...
loading docker://aci-zot.cisco.com:5000/c3/minbase:1.0.32-squashfs
Copying blob 2789aef1e35a skipped: already exists
Copying blob ca96295927a1 skipped: already exists
Copying config 00d5e3dcbb done
Writing manifest to image destination
Storing signatures
cached: types.Layer{From:types.ImageSource{Type:"docker", Url:"docker://aci-zot.cisco.com:5000/c3/minbase:1.0.32", Tag:"", Insecure:false}, Imports:types.Imports(nil), OverlayDirs:[]types.OverlayDir(nil), Run:types.StringList{"id\ntouch /var/lib/dpkg/myfile && echo \"/var/lib/dpkg is writable\"\nls -l /var/lib/dpkg/\napt-get install hello\n"}, Cmd:types.Command(nil), Entrypoint:types.Command(nil), FullCommand:types.Command(nil), BuildEnvPt:[]string(nil), BuildEnv:map[string]string(nil), Environment:map[string]string(nil), Volumes:[]string(nil), Labels:map[string]string(nil), GenerateLabels:types.StringList(nil), WorkingDir:"", BuildOnly:false, Binds:types.Binds(nil), RuntimeUser:"", Annotations:map[string]string(nil), OS:(*string)(0xc00044a7d0), Arch:(*string)(0xc00044a7e0)}
new: types.Layer{From:types.ImageSource{Type:"docker", Url:"docker://aci-zot.cisco.com:5000/c3/minbase:1.0.32-squashfs", Tag:"", Insecure:false}, Imports:types.Imports(nil), OverlayDirs:[]types.OverlayDir(nil), Run:types.StringList{"id\ntouch /var/lib/dpkg/myfile && echo \"/var/lib/dpkg is writable\"\nls -l /var/lib/dpkg/\napt-get install hello\n"}, Cmd:types.Command(nil), Entrypoint:types.Command(nil), FullCommand:types.Command(nil), BuildEnvPt:[]string(nil), BuildEnv:map[string]string(nil), Environment:map[string]string(nil), Volumes:[]string(nil), Labels:map[string]string(nil), GenerateLabels:types.StringList(nil), WorkingDir:"", BuildOnly:false, Binds:types.Binds(nil), RuntimeUser:"", Annotations:map[string]string(nil), OS:(*string)(0xc00044a190), Arch:(*string)(0xc00044a1a0)}
cache miss because layer definition was changed
unpacking to /tmp/t1/roots/smtest
Extracting /tmp/t1/.stacker/layer-bases/oci/blobs/sha256/ca96295927a1655df6869aa6dc4d5d6e4dbf9980f775b0611eb3793b7b6ff25d -> /tmp/t1/roots/sha256_ca96295927a1655df6869aa6dc4d5d6e4dbf9980f775b0611eb3793b7b6ff25d/overlay with squashfuse [/tmp/t1/roots/sha256_ca96295927a1655df6869aa6dc4d5d6e4dbf9980f775b0611eb3793b7b6ff25d/.overlay-squashfuse.log]
Extracting /tmp/t1/.stacker/layer-bases/oci/blobs/sha256/2789aef1e35ac561943dc3fec9c80fdd866e67171041b74d6665dc67c9f5cc63 -> /tmp/t1/roots/sha256_2789aef1e35ac561943dc3fec9c80fdd866e67171041b74d6665dc67c9f5cc63/overlay with squashfuse [/tmp/t1/roots/sha256_2789aef1e35ac561943dc3fec9c80fdd866e67171041b74d6665dc67c9f5cc63/.overlay-squashfuse.log]
lxc rootfs overlay arg overlayfs:/tmp/t1/roots/sha256_ca96295927a1655df6869aa6dc4d5d6e4dbf9980f775b0611eb3793b7b6ff25d/overlay:/tmp/t1/roots/sha256_2789aef1e35ac561943dc3fec9c80fdd866e67171041b74d6665dc67c9f5cc63/overlay:/tmp/t1/roots/smtest/overlay
stacker version v0.30.1-5-g1346183
stacker subcommand: [/tmp/stacker --oci-dir /tmp/t1/oci --roots-dir /tmp/t1/roots --stacker-dir /tmp/t1/.stacker --storage-type overlay --internal-userns --debug internal-go check-aa-profile lxc-container-default-cgns]
bind mounting /tmp/t1/.stacker/imports/smtest into container
+ id
uid=0(root) gid=0(root) groups=0(root)
+ touch /var/lib/dpkg/myfile
+ echo /var/lib/dpkg is writable
/var/lib/dpkg is writable
+ ls -l /var/lib/dpkg/
total 257
drwxr-xr-x 2 root root     0 Nov  4 20:57 alternatives
-rw-r--r-- 1 root root 82696 Nov  4 20:57 available
-rw-r--r-- 1 root root     8 Nov  4 20:57 cmethopt
-rw-r--r-- 1 root root    98 Nov  4 20:57 diversions
-rw-r--r-- 1 root root    29 Nov  4 20:57 diversions-old
drwxr-xr-x 1 root root     0 Dec  5 18:41 info
-rw-r----- 1 root root     0 Dec  5 18:41 lock
-rw-r----- 1 root root     0 Nov  4 20:57 lock-frontend
-rw-rw-r-- 1 root root     0 Dec  6 17:04 myfile
drwxr-xr-x 2 root root     0 Mar 23  2020 parts
-rw-r--r-- 1 root root     0 Nov  4 20:57 statoverride
-rw-r--r-- 1 root root 90129 Dec  5 18:41 status
-rw-r--r-- 1 root root 90139 Dec  5 18:41 status-old
drwxr-xr-x 1 root root     0 Dec  5 18:41 triggers
drwxr-xr-x 1 root root     0 Dec  5 18:41 updates
+ apt-get install hello
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (38: Function not implemented)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
error: run commands failed: execute failed: exit status 100
stackerbuild.io/stacker.(*Builder).build
        /stacker-tree/build.go:449
stackerbuild.io/stacker.(*Builder).BuildMultiple
        /stacker-tree/build.go:546
main.doBuild
        /stacker-tree/cmd/build.go:112
github.com/urfave/cli.HandleAction
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:524
github.com/urfave/cli.Command.Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.10/command.go:173
github.com/urfave/cli.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:277
main.main
        /stacker-tree/cmd/main.go:304
runtime.main
        /usr/lib/go/src/runtime/proc.go:250
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1594
error: exit status 1
stackerbuild.io/stacker/container.MaybeRunInNamespace
        /stacker-tree/container/userns.go:102
main.main.func3
        /stacker-tree/cmd/main.go:299
github.com/urfave/cli.(*App).Run
        /stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.10/app.go:264
main.main
        /stacker-tree/cmd/main.go:304
runtime.main
        /usr/lib/go/src/runtime/proc.go:250
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:1594

Expected behavior

Expected squash to work as tar.

Screenshots

No response

Additional context

I don't have an easy way to give a reference docker container. Those on zothub are not built with new enough stacker to have xattrs removed.

can probably attach a tarball.

@smoser smoser added the bug label Dec 6, 2022
@smoser
Copy link
Contributor Author

smoser commented Dec 6, 2022

Running strace as @hallyn suggested left me with output like this:

getdents64(4, /* 0 entries */, 131072)  = 0
close(4)                                = 0
openat(AT_FDCWD, "/var/lib/dpkg/lock-frontend", O_RDWR|O_CREAT|O_NOFOLLOW, 0640) = -1 ENOSYS (Function not implemented)
write(2, "E", 1E)                        = 1
write(2, ": ", 2: )                       = 2
write(2, "Could not open lock file /var/li"..., 90Could not open lock file /var/lib/dpkg/lock-frontend - open (38: Function not implemented)) = 90
write(2, "\n", 1
)                       = 1
write(2, "E", 1E)                        = 1
write(2, ": ", 2: )                       = 2
write(2, "Unable to acquire the dpkg front"..., 85Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?) = 85
write(2, "\n", 1

I can reproduce the problem with a simple test:

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char* argv[]) {
    int fd = 0, errors = 0;
    for (int i=1; i<argc;i++) {
        fd = openat(AT_FDCWD, argv[i], O_RDWR|O_CREAT|O_NOFOLLOW, 0640);
        if (fd < 0) {
            printf("%s: %d: %s\n", argv[i], fd, strerror(errno));
            errors++;
        } else {
            printf("%s: %d\n", argv[i], fd);
        }
        close(fd);
    }

    return errors;
}

I can run that via stacker.yaml with /stacker/test-me /var/lib/dpkg/lock-frontend /etc/passwd /tmp/foo /etc/bar
And see output of:

+ /stacker/test-me /var/lib/dpkg/lock-frontend /etc/passwd /tmp/foo /etc/bar
/var/lib/dpkg/lock-frontend: -1: Function not implemented
/etc/passwd: -1: Function not implemented
/tmp/foo: 3
/etc/bar: 3

@smoser
Copy link
Contributor Author

smoser commented Dec 6, 2022

I can fix this for real root by supporting kernel mounts of squashfs like this:

diff --git a/squashfs/squashfs.go b/squashfs/squashfs.go
index 17ded43..d0a1241 100644
--- a/squashfs/squashfs.go
+++ b/squashfs/squashfs.go
@@ -168,7 +168,15 @@ func ExtractSingleSquash(squashFile string, extractDir string, storageType strin
 		return err
 	}
 
-	if p := which("squashfuse"); p != "" {
+	if realuid := os.Getenv("STACKER_REAL_UID"); realuid == "0" {
+		log.Debugf("Extracting %s -> %s with kernel mount", squashFile, extractDir)
+		cmd := exec.Command("mount", "-oloop,ro", squashFile, extractDir)
+		cmd.Stdout = os.Stdout
+		cmd.Stderr = os.Stderr
+		cmd.Stdin = nil
+		return cmd.Run()
+		return cmd.Start()
+	} else if p := which("squashfuse"); p != "" {
 		// given extractDir of path/to/some/dir[/], log to path/to/some/.dir-squashfs.log
 		extractDir := strings.TrimSuffix(extractDir, "/")

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

openat(AT_FDCWD, "/var/lib/dpkg/lock-frontend", O_RDWR|O_CREAT|O_NOFOLLOW, 0640) = -1 ENOSYS (Function not implemented)

Can you show the squashfuse log for that?

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

I compiled the following program:

#include <stdio.h>
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>

int main(int argc, char *argv[]) {
	int fd;
	char *filename = NULL;

	if (argc < 2) {
		printf("usage: %s filename\n", argv[0]);
		exit(0);
	}

	filename = argv[1];

	fd = openat(AT_FDCWD, filename, O_RDWR|O_CREAT|O_NOFOLLOW, 0640);
	printf("filename %s, fd %d, errno %m\n", filename, fd);
	if (fd >= 0) {
		int n = write(fd, "hi", 3);
		printf("write returned %d %m\n", n);
		close(fd);
	}
}

I created a squashfs file containing:

serge@jerom ~/delme/squashfuse$ find squash-src/
squash-src/
squash-src/a
squash-src/a/ab

I created a userns and mntns, mounted the s.squashfs (built from
squash-src) at l, mounted the overlay under dest, and was able to
do the openat:

root@jerom ~/delme/squashfuse$ squashfuse s.squashfs l
root@jerom ~/delme/squashfuse$ ls l
a
root@jerom ~/delme/squashfuse$ mount -t overlay -o lowerdir=l,upperdir=u,workdir=w,index=on l dest
root@jerom ~/delme/squashfuse$ ~/test/openat dest/x
filename dest/x, fd 3, errno Success
write returned 3 Success
root@jerom ~/delme/squashfuse$ cat u/x
hiroot@jerom ~/delme/squashfuse$ ~/test/openat dest/a/y
filename dest/a/y, fd 3, errno Success
write returned 3 Success

This is on 5.15.0-56-generic .

I wonder what's going on with the stacker build!

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

Oh, duh - it's what happens when the file exists:

root@jerom ~/delme/squashfuse$ ~/test/openat dest/a/ab
filename dest/a/ab, fd -1, errno Function not implemented

So, why?

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

Just O_CREATE works, but O_CREATE|O_RDWR does not.

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

running squashfuse with -o allow_root does not help. Debug mode gives me:

unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 42, pid: 278341
LOOKUP /a
getattr /a
   NODEID: 2
   unique: 6, success, outsize: 144
unique: 8, opcode: LOOKUP (1), nodeid: 2, insize: 43, pid: 278341
LOOKUP /a/ab
getattr /a/ab
   NODEID: 3
   unique: 8, success, outsize: 144
unique: 10, opcode: OPEN (14), nodeid: 3, insize: 48, pid: 278341
open flags: 0x48000 /a/ab
   open[93843198916512] flags: 0x48000 /a/ab
   unique: 10, success, outsize: 32
unique: 12, opcode: ??? (46), nodeid: 3, insize: 64, pid: 278341
   unique: 12, error: -38 (Function not implemented), outsize: 16
unique: 14, opcode: READ (15), nodeid: 3, insize: 80, pid: 278341
read[93843198916512] 4096 bytes from 0 flags: 0x48000
   read[93843198916512] 3 bytes from 0
   unique: 14, success, outsize: 19
unique: 16, opcode: LISTXATTR (23), nodeid: 3, insize: 48, pid: 278341
listxattr /a/ab 0
   unique: 16, success, outsize: 24
unique: 18, opcode: OPEN (14), nodeid: 3, insize: 48, pid: 278341
open flags: 0x0 /a/ab
   open[93843198916880] flags: 0x0 /a/ab
   unique: 18, success, outsize: 32
unique: 20, opcode: OPEN (14), nodeid: 3, insize: 48, pid: 278341
open flags: 0x0 /a/ab
   open[93843198917104] flags: 0x0 /a/ab
   unique: 20, success, outsize: 32
unique: 22, opcode: IOCTL (39), nodeid: 3, insize: 72, pid: 278341
   unique: 22, success, outsize: 36
unique: 24, opcode: RELEASE (18), nodeid: 3, insize: 64, pid: 0
release[93843198917104] flags: 0x0
   unique: 24, success, outsize: 16
unique: 26, opcode: RELEASE (18), nodeid: 3, insize: 64, pid: 0
release[93843198916880] flags: 0x0
   unique: 26, success, outsize: 16
unique: 28, opcode: RELEASE (18), nodeid: 3, insize: 64, pid: 0
release[93843198916512] flags: 0x48000
   unique: 28, success, outsize: 16

@smoser smoser changed the title [Bug]: cannot run 'apt-get install' [Bug] cannot run 'apt-get install' Dec 6, 2022
@smoser
Copy link
Contributor Author

smoser commented Dec 6, 2022

I opened a squashfuse issue with a recreate vasi/squashfuse#80

@hallyn
Copy link
Contributor

hallyn commented Dec 6, 2022

I opened a squashfuse issue with a recreate vasi/squashfuse#80

Thanks - i was still looking through the closed issue for a hint.

I'm thinking I'll try and reproduce this in a lucid ubuntu image with mainline kernel. It does seem just as likely that the villain here is overlay, not directly squashfuse.

@hallyn
Copy link
Contributor

hallyn commented Dec 7, 2022

I've been misreading my squashfuse log file. The open arguments did not cause the 'Function not implemented'. It is an opcode 46 that is done immediately after:

unique: 12, opcode: ??? (46), nodeid: 3, insize: 64, pid: 278341
   unique: 12, error: -38 (Function not implemented), outsize: 16

I had been thinking that was part of the previous open operation.

@smoser
Copy link
Contributor Author

smoser commented Dec 7, 2022

tldr; This is fixed in squashfuse and libfuse3 packages available in ppa:puzzleos/dev.

Some status here.

I found that

  1. fuse has a fix for this in its repository somewhere between Ubuntu 22.04's version (3.10.5-1build1) and most recent tag 3.12.0
  2. squashfuse in all current debian and ubuntu packages is linked against fuse2 , not fuse3. fuse2 is effectively dead.

I filed debian bug #1025706 requesting it be built against fuse3.

I then:

  • built and published a new version of squashfuse with upstream 0.1.105 linking against libfuse3 0.1.105-0ubuntu122.04.1ppa0
  • built and published a no change backport of debian/unstable (3.12.0-1) to 22.04 as version 3.12.0-122.04.1ppa0

Binaries are available in ppa:puzzleos/dev

An upload to debian or ubuntu (lunar) that linked against fuse3 will fix this (fuse3 in lunar is sufficient, but probably not in kinetic and definitely not in jammy).

@smoser
Copy link
Contributor Author

smoser commented Mar 2, 2023

closing this, it is fixed see comment above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants