Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container doesn't start because of missing /media possibly due to failed RPM transaction #539

Closed
bam80 opened this issue Aug 25, 2020 · 18 comments
Labels

Comments

@bam80
Copy link

@bam80 bam80 commented Aug 25, 2020

[bam@host ~]$ toolbox --verbose  enter --container kde.f33
DEBU Running as real user ID 1001                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user bam 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /home/bam/.config/toolbox 
DEBU Current Podman version is 2.0.4              
DEBU Old Podman version is 2.0.4                  
DEBU Migration not needed: Podman version 2.0.4 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'kde.f33'                         
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'kde.f33'                         
DEBU Image: 'fedora-toolbox:32'                   
DEBU Release: '32'                                
DEBU Checking if container kde.f33 exists         
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU Starting container kde.f33                   
DEBU Inspecting entry point of container kde.f33  
DEBU Entry point PID is a float64                 
DEBU Entry point of container kde.f33 is toolbox (PID=0) 
Error: invalid entry point PID of container kde.f33
[bam@host ~]$ toolbox --version
toolbox version 0.0.93

[bam@host ~]$ podman start --attach kde.f33
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1001"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Path /run/host/monitor exists"
level=debug msg="Redirecting /media to /run/media"
Error: failed to redirect /media to /run/media

I don't see that /media line at all when attaching to f32 container:

[bam@host ~]$ podman --log-level debug start --attach kde.f32
....                    
level=debug msg="Running as real user ID 0"
level=debug msg="Resolved absolute path to the executable as /usr/bin/toolbox"
level=debug msg="TOOLBOX_PATH is /usr/bin/toolbox"
level=debug msg="XDG_RUNTIME_DIR is unset"
level=debug msg="XDG_RUNTIME_DIR set to /run/user/1001"
level=debug msg="Creating /run/.toolboxenv"
level=debug msg="Monitoring host"
level=debug msg="Path /run/host/etc exists"
level=debug msg="Binding /etc/machine-id to /run/host/etc/machine-id"
level=debug msg="Creating /run/systemd/journal"
level=debug msg="Binding /run/systemd/journal to /run/host/run/systemd/journal"
level=debug msg="Creating /var/lib/flatpak"
level=debug msg="Binding /var/lib/flatpak to /run/host/var/lib/flatpak"
level=debug msg="Creating /var/log/journal"
level=debug msg="Binding /var/log/journal to /run/host/var/log/journal"
level=debug msg="Creating /var/mnt"
level=debug msg="Binding /var/mnt to /run/host/var/mnt"
level=debug msg="Creating /sys/fs/selinux"
level=debug msg="Binding /sys/fs/selinux to /usr/share/empty"
level=debug msg="Path /run/host/monitor exists"
level=debug msg="Finished initializing container"
level=debug msg="Creating runtime directory /run/user/1001/toolbox"
level=debug msg="Creating initialization stamp /run/user/1001/toolbox/container-initialized-17636"
level=debug msg="Going to sleep"
DEBU[0137] Sending signal 28 to container 3cda239851f1cb2706cc312490e6b75b568eb1e1f92c56f52bd1ea924674d547 
@debarshiray
Copy link
Member

@debarshiray debarshiray commented Aug 26, 2020

Could you please try with Toolbox 0.0.94?

Specifically, this error should be more detailed now:

[bam@host ~]$ podman start --attach kde.f33
...
level=debug msg="Redirecting /media to /run/media"
Error: failed to redirect /media to /run/media
@bam80
Copy link
Author

@bam80 bam80 commented Aug 26, 2020

I doubt how I can try Toolbox 0.0.94 on Fedora Silverblue 32.
However, I could probably temporary upgrade to SB 33, not sure if it is already accessible..

@bam80
Copy link
Author

@bam80 bam80 commented Aug 26, 2020

Could you please try with Toolbox 0.0.94?

Can I run Toolbox from within Toolbox? :)

@debarshiray
Copy link
Member

@debarshiray debarshiray commented Aug 26, 2020

I doubt how I can try Toolbox 0.0.94 on Fedora Silverblue 32.

I created an update for Fedora 32 just now. You can use rpm-ostree to override the toolbox package that's in your OS image.

Can I run Toolbox from within Toolbox? :)

Yes, but it will still be the version that's on your host.

@bam80
Copy link
Author

@bam80 bam80 commented Aug 26, 2020

@debarshiray thanks! For now I just manually installed new Toolbox in custom dir, hope it should work OK from there.
Here is the results:

[bam@host ~]$ toolbox/toolbox -v enter --container kde.f33
toolbox: running as real user ID 1001
toolbox: resolved absolute path for toolbox/toolbox to /var/home/bam/toolbox/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user bam
toolbox: TOOLBOX_PATH is /var/home/bam/toolbox/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 2.0.4
toolbox: migration not needed: Podman version 2.0.4 is unchanged
toolbox: Fedora generational core is f32
toolbox: base image is fedora-toolbox:32
toolbox: container is kde.f33
toolbox: checking if container kde.f33 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container kde.f33

Error: error opening container config: open /var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/config.json: permission denied
toolbox: /etc/profile.d/toolbox.sh not mounted in container kde.f33
toolbox: copying /etc/profile.d/toolbox.sh to /run/user/1001/toolbox
Error: unable to start container "f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b": cannot chown /var/home/bam/.local/share/containers/storage/overlay/f5bf2ffa2b7a0ae07934fb025889b64ffa9aa25dd0e9e58d3ed620bdaf169bdb/merged to 1:1: chown /var/home/bam/.local/share/containers/storage/overlay/f5bf2ffa2b7a0ae07934fb025889b64ffa9aa25dd0e9e58d3ed620bdaf169bdb/merged: invalid argument
toolbox: failed to start container kde.f33

[bam@host ~]$ ll /var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/config.json
ls: cannot access '/var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/config.json': Permission denied
[bam@host ~]$ ll /var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/
total 4
drwx------. 14 100000 100000 4096 Aug 26 16:34 userdata

[bam@host ~]$ ll /var/home/bam/.local/share/containers/storage/overlay/f5bf2ffa2b7a0ae07934fb025889b64ffa9aa25dd0e9e58d3ed620bdaf169bdb/merged
total 0

It doesn't help if I manually chown the userdata directory to normal user: : invalid argument error persists

@bam80
Copy link
Author

@bam80 bam80 commented Aug 26, 2020

@debarshiray thanks! For now I just manually installed new Toolbox in custom dir, hope it should work OK from there.

UPDATE:
@debarshiray eventually I had to return to your newly created .rpm since custom-installed one didn't help in case of podman --attach, thank you for this!

And here we have an update!:

[bam@host ~]$ podman --log-level debug start --attach kde.f33
...
level=debug msg="Redirecting /media to /run/media"
Error: failed to redirect /media to /run/media: remove /media: no such file or directory

Should I create /media in the container? How, considering I can't start it?

@bam80
Copy link
Author

@bam80 bam80 commented Aug 27, 2020

I've compared working and non-working containers based on the same image. Please see if it could be helpful. I have run out of ideas..

[bam@host ~]$ diff <(podman inspect   fedora-toolbox-33) <(podman inspect kde.f33)
3,4c3,4
<         "Id": "323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090",
<         "Created": "2020-08-26T14:51:37.743975407+03:00",
---
>         "Id": "f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b",
>         "Created": "2020-04-22T03:06:10.24892153+03:00",
23c23
<             "OciVersion": "1.0.2-dev",
---
>             "OciVersion": "1.0.1-dev",
31c31
<             "ExitCode": 130,
---
>             "ExitCode": 1,
33,34c33,34
<             "StartedAt": "2020-08-27T05:55:25.011942018+03:00",
<             "FinishedAt": "2020-08-27T05:56:10.104726968+03:00",
---
>             "StartedAt": "2020-08-27T06:00:56.807335389+03:00",
>             "FinishedAt": "2020-08-27T06:00:57.03796755+03:00",
46c46
<         "HostnamePath": "/run/user/1001/containers/overlay-containers/323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090/userdata/hostname",
---
>         "HostnamePath": "/run/user/1001/containers/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/hostname",
48,49c48,49
<         "StaticDir": "/var/home/bam/.local/share/containers/storage/overlay-containers/323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090/userdata",
<         "OCIConfigPath": "/var/home/bam/.local/share/containers/storage/overlay-containers/323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090/userdata/config.json",
---
>         "StaticDir": "/var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata",
>         "OCIConfigPath": "/var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/config.json",
51c51
<         "LogPath": "/var/home/bam/.local/share/containers/storage/overlay-containers/323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090/userdata/ctr.log",
---
>         "LogPath": "/var/home/bam/.local/share/containers/storage/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/ctr.log",
53,54c53,54
<         "ConmonPidFile": "/run/user/1001/containers/overlay-containers/323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090/userdata/conmon.pid",
<         "Name": "fedora-toolbox-33",
---
>         "ConmonPidFile": "/run/user/1001/containers/overlay-containers/f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b/userdata/conmon.pid",
>         "Name": "kde.f33",
57c57
<         "MountLabel": "system_u:object_r:container_file_t:s0:c480,c754",
---
>         "MountLabel": "system_u:object_r:container_file_t:s0:c67,c836",
106,107c106,107
<                 "UpperDir": "/var/home/bam/.local/share/containers/storage/overlay/345d26bdd6981c7c79d18618a6faf48cdf58328f26ca0cda07c232dfe88e882c/diff",
<                 "WorkDir": "/var/home/bam/.local/share/containers/storage/overlay/345d26bdd6981c7c79d18618a6faf48cdf58328f26ca0cda07c232dfe88e882c/work"
---
>                 "UpperDir": "/var/home/bam/.local/share/containers/storage/overlay/f5bf2ffa2b7a0ae07934fb025889b64ffa9aa25dd0e9e58d3ed620bdaf169bdb/diff",
>                 "WorkDir": "/var/home/bam/.local/share/containers/storage/overlay/f5bf2ffa2b7a0ae07934fb025889b64ffa9aa25dd0e9e58d3ed620bdaf169bdb/work"
114,115c114,115
<                 "Source": "/run/.heim_org.h5l.kcm-socket",
<                 "Destination": "/run/.heim_org.h5l.kcm-socket",
---
>                 "Source": "/var",
>                 "Destination": "/run/host/var",
119,120d118
<                     "nosuid",
<                     "nodev",
124c122
<                 "Propagation": "rprivate"
---
>                 "Propagation": "rslave"
129,130c127,128
<                 "Source": "/etc",
<                 "Destination": "/run/host/etc",
---
>                 "Source": "/tmp",
>                 "Destination": "/run/host/tmp",
133a132,133
>                     "nosuid",
>                     "nodev",
137c137
<                 "Propagation": "rprivate"
---
>                 "Propagation": "rslave"
142,143c142,143
<                 "Source": "/dev",
<                 "Destination": "/dev",
---
>                 "Source": "/run/user/1001",
>                 "Destination": "/run/user/1001",
147a148
>                     "nodev",
151c152
<                 "Propagation": "rslave"
---
>                 "Propagation": "rprivate"
156,157c157,158
<                 "Source": "/var/home/bam",
<                 "Destination": "/var/home/bam",
---
>                 "Source": "/usr",
>                 "Destination": "/run/host/usr",
163c164
<                 "RW": true,
---
>                 "RW": false,
169,170c170,171
<                 "Source": "/run/user/1001",
<                 "Destination": "/run/user/1001",
---
>                 "Source": "/run/dbus/system_bus_socket",
>                 "Destination": "/run/dbus/system_bus_socket",
184,185c185,186
<                 "Source": "/etc/profile.d/toolbox.sh",
<                 "Destination": "/etc/profile.d/toolbox.sh",
---
>                 "Source": "/dev",
>                 "Destination": "/dev",
188a190
>                     "nosuid",
191,192c193,194
<                 "RW": false,
<                 "Propagation": "rprivate"
---
>                 "RW": true,
>                 "Propagation": "rslave"
197,198c199,200
<                 "Source": "/var",
<                 "Destination": "/run/host/var",
---
>                 "Source": "/run/media",
>                 "Destination": "/run/media",
201a204,205
>                     "nosuid",
>                     "nodev",
210,211c214,215
<                 "Source": "/usr/bin/toolbox",
<                 "Destination": "/usr/bin/toolbox",
---
>                 "Source": "/run/.heim_org.h5l.kcm-socket",
>                 "Destination": "/run/.heim_org.h5l.kcm-socket",
214a219,220
>                     "nosuid",
>                     "nodev",
217c223
<                 "RW": false,
---
>                 "RW": true,
223,224c229,230
<                 "Source": "/run/user/1001/.flatpak-helper/monitor",
<                 "Destination": "/run/host/monitor",
---
>                 "Source": "/run",
>                 "Destination": "/run/host/run",
233c239
<                 "Propagation": "rprivate"
---
>                 "Propagation": "rslave"
238,239c244,245
<                 "Source": "/run/dbus/system_bus_socket",
<                 "Destination": "/run/dbus/system_bus_socket",
---
>                 "Source": "/run/user/1001/.flatpak-helper/monitor",
>                 "Destination": "/run/host/monitor",
253,254c259,260
<                 "Source": "/usr",
<                 "Destination": "/run/host/usr",
---
>                 "Source": "/var/home/bam",
>                 "Destination": "/var/home/bam",
260c266
<                 "RW": false,
---
>                 "RW": true,
266,267c272,273
<                 "Source": "/run/media",
<                 "Destination": "/run/media",
---
>                 "Source": "/etc",
>                 "Destination": "/run/host/etc",
271,272d276
<                     "nosuid",
<                     "nodev",
276c280
<                 "Propagation": "rslave"
---
>                 "Propagation": "rprivate"
281,282c285,286
<                 "Source": "/tmp",
<                 "Destination": "/run/host/tmp",
---
>                 "Source": "/etc/profile.d/toolbox.sh",
>                 "Destination": "/etc/profile.d/toolbox.sh",
286,287d289
<                     "nosuid",
<                     "nodev",
290,291c292,293
<                 "RW": true,
<                 "Propagation": "rslave"
---
>                 "RW": false,
>                 "Propagation": "rprivate"
296,297c298,299
<                 "Source": "/run",
<                 "Destination": "/run/host/run",
---
>                 "Source": "/usr/bin/toolbox",
>                 "Destination": "/usr/bin/toolbox",
301,302d302
<                     "nosuid",
<                     "nodev",
305,306c305,306
<                 "RW": true,
<                 "Propagation": "rslave"
---
>                 "RW": false,
>                 "Propagation": "rprivate"
348c348
<             "file",
---
>             "journald",
351c351
<             "323c52fecd296730cb0a9475d8e469e8a89516144f7de4a956e6f0d2502b5090"
---
>             "f80dca917199ae7d82f10379502fa3ae4bf1f9518a1ae60678a49ab670aba59b"
368,370c368
<                 "container=podman",
<                 "NAME=fedora-toolbox",
<                 "VERSION=33",
---
>                 "container=oci",
372a371,372
>                 "NAME=fedora-toolbox",
>                 "VERSION=33",
421c421
<                 "io.kubernetes.cri-o.Created": "2020-08-26T14:51:37.743975407+03:00",
---
>                 "io.kubernetes.cri-o.Created": "2020-04-22T03:06:10.24892153+03:00",
433,434d432
<                 "--log-level",
<                 "error",
451c449
<                 "fedora-toolbox-33",
---
>                 "kde.f33",
466,468c464
<                 "/etc:/run/host/etc",
<                 "--volume",
<                 "/dev:/dev:rslave",
---
>                 "/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket",
470c466
<                 "/run:/run/host/run:rslave",
---
>                 "/run/media:/run/media:rslave",
472c468
<                 "/tmp:/run/host/tmp:rslave",
---
>                 "/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro",
474c470
<                 "/var:/run/host/var:rslave",
---
>                 "/usr/bin/toolbox:/usr/bin/toolbox:ro",
476c472
<                 "/run/dbus/system_bus_socket:/run/dbus/system_bus_socket",
---
>                 "/run/user/1001:/run/user/1001",
479a476,477
>                 "/run/dbus/system_bus_socket:/run/dbus/system_bus_socket",
>                 "--volume",
482c480
<                 "/usr/bin/toolbox:/usr/bin/toolbox:ro",
---
>                 "/etc:/run/host/etc",
484c482
<                 "/usr:/run/host/usr:ro,rslave",
---
>                 "/dev:/dev:rslave",
486c484
<                 "/run/user/1001:/run/user/1001",
---
>                 "/run:/run/host/run:rslave",
488c486
<                 "/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket",
---
>                 "/tmp:/run/host/tmp:rslave",
490c488
<                 "/run/media:/run/media:rslave",
---
>                 "/usr:/run/host/usr:ro,rslave",
492c490
<                 "/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro",
---
>                 "/var:/run/host/var:rslave",
513,518d510
<                 "/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket:rw,rprivate,nosuid,nodev,rbind",
<                 "/etc:/run/host/etc:rw,rprivate,rbind",
<                 "/dev:/dev:rslave,rw,nosuid,rbind",
<                 "/var/home/bam:/var/home/bam:rslave,rw,rbind",
<                 "/run/user/1001:/run/user/1001:rw,rprivate,nosuid,nodev,rbind",
<                 "/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro,rprivate,rbind",
520,522c512,513
<                 "/usr/bin/toolbox:/usr/bin/toolbox:ro,rprivate,rbind",
<                 "/run/user/1001/.flatpak-helper/monitor:/run/host/monitor:rw,rprivate,nosuid,nodev,rbind",
<                 "/run/dbus/system_bus_socket:/run/dbus/system_bus_socket:rw,rprivate,nosuid,nodev,rbind",
---
>                 "/tmp:/run/host/tmp:rslave,rw,nosuid,nodev,rbind",
>                 "/run/user/1001:/run/user/1001:rw,rprivate,nosuid,nodev,rbind",
523a515,516
>                 "/run/dbus/system_bus_socket:/run/dbus/system_bus_socket:rw,rprivate,nosuid,nodev,rbind",
>                 "/dev:/dev:rslave,rw,nosuid,rbind",
525,526c518,524
<                 "/tmp:/run/host/tmp:rslave,rw,nosuid,nodev,rbind",
<                 "/run:/run/host/run:rslave,rw,nosuid,nodev,rbind"
---
>                 "/run/.heim_org.h5l.kcm-socket:/run/.heim_org.h5l.kcm-socket:rw,rprivate,nosuid,nodev,rbind",
>                 "/run:/run/host/run:rslave,rw,nosuid,nodev,rbind",
>                 "/run/user/1001/.flatpak-helper/monitor:/run/host/monitor:rw,rprivate,nosuid,nodev,rbind",
>                 "/var/home/bam:/var/home/bam:rslave,rw,rbind",
>                 "/etc:/run/host/etc:rw,rprivate,rbind",
>                 "/etc/profile.d/toolbox.sh:/etc/profile.d/toolbox.sh:ro,rprivate,rbind",
>                 "/usr/bin/toolbox:/usr/bin/toolbox:ro,rprivate,rbind"
595c593
<             "MemorySwappiness": 0,
---
>             "MemorySwappiness": -1
@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Sep 9, 2020

Workaround I used:

$ podman unshare /bin/bash
# cd $(podman mount fedora-toolbox-33)
# ln -s /run/media .
# exit
@HarryMichal
Copy link
Collaborator

@HarryMichal HarryMichal commented Sep 9, 2020

Thank you @owtaylor for adding the workaround!

Owen shared with me the last lines of podman logs fedora-toolbox-33:

level=debug msg="Redirecting /media to /run/media"
Error: failed to redirect /media to /run/media: remove /media: no such file or directory

I believe I know what is going on there. The redirectPath function before creating the containerPath symlink aiming to target tries to remove containerPath using os.Remove(containerPath). That function fails if:

  • there's some permission error
  • containerPath is a non-empty folder
  • containerPath does not exist

I believe we're hitting the third point here. So, the solution should either be to handle cases when containerPath does not exist or use os.RemoveAll that removes even non-empty folders. I wonder what's the correct approach. @owtaylor, @debarshiray?

@debarshiray
Copy link
Member

@debarshiray debarshiray commented Oct 1, 2020

I see. Interesting.

I'm puzzled why the containerPath doesn't exist in the first place. In this case, where containerPath is /media, I don't understand why it went missing from the image. eg., I pulled registry.fedoraproject.org/f33/fedora-toolbox:33 and it still has a /media.

The proposed solution of special casing the third point (ie., containerPath doesn't exist) makes sense to me. Knowing the root cause would give me a bit more peace of mind. :)

@debarshiray
Copy link
Member

@debarshiray debarshiray commented Oct 1, 2020

I just want to clarify this bit that came up in #silverblue on Freenode:

<otaylor> I am puzzled with redirectPath(), why the check for err is      
      inside the 'if folder' check - that looks odd

The older POSIX shell implementation used rmdir <path> && mkdir --parents ... for directories, and rm --force <path> && ... for regular files. That's why the Go code is the way it is - to try to mimic the old code as closely as possible.

Removing the --force from the rm --force would make this oddity go away, and I think that it was a historical accident. At least I can't remember any real reason. However, I didn't touch it because, you know, why change something that doesn't seem broken. :)

@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Oct 1, 2020

I'm puzzled why the containerPath doesn't exist in the first place. In this case, where containerPath is /media, I don't understand why it went missing from the image. eg., I pulled registry.fedoraproject.org/f33/fedora-toolbox:33 and it still has a /media.

I'd say it was just something weird that happened to my toolbox, except that it also happened to @bam80 - I agree that every f33 toolbox has shipped with a /media. We still have problems with the filesystem package on 'dnf update' inside a container - maybe something happened there?

The shell-like 'if [ -e /media ]' seems fine to me - it's infinitesimally less efficient than handling ENOENT, but makes the code easier to read, and there shouldn't be any risk of race conditions in this case - the toolbox init is the only thing manipulating the container.

@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Oct 1, 2020

Confirmed - did something like:

$ toolbox create -c f33-to-upgrade -i f33/fedora-toolbox:f33-4
$ toolbox enter f33-to-upgrade
$ dnf -y --enablerepo=fedora --disablerepo=rawhide update
<...>
Failed:
  filesystem-3.14-2.fc32.x86_64                                                         filesystem-3.14-3.fc33.x86_64                                                        

Error: Transaction failed
$ ls -l /media
ls: cannot access '/media': No such file or directory
@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Oct 1, 2020

The particular error upgrading filesystem is similar to what we've seen before:

  Upgrading        : filesystem-3.14-3.fc33.x86_64                                                                                                                    17/601 
Error unpacking rpm package filesystem-3.14-3.fc33.x86_64
error: unpacking of archive failed on file /dev: cpio: chown
error: filesystem-3.14-3.fc33.x86_64: install failed

I thought we were dropping in an /etc/rpm file setting %_netsharedpath either in the Fedora image or in the toolbox image, but I don't see that.

@owtaylor
Copy link
Contributor

@owtaylor owtaylor commented Oct 1, 2020

Doing: echo '%_netsharedpath /proc:/sys:/dev' > /etc/rpm/macros.toolbox - before upgrading helps some - the transaction still fails because filesystem fails verification (permissons are wrong on /proc /sys /dev), but things are unpacked correctly and the image is left with a /media directory.

debarshiray added a commit to HarryMichal/toolbox that referenced this issue Oct 1, 2020
The redirectPath function used to error out when handling directories,
if the path inside the container was initially absent. There's no real
reason for this, and recently some containers failed to start because
the /media directory was absent from them.

Therefore, it's better to not worry about the path being initially
absent, and be more forgiving and robust.

containers#539
debarshiray added a commit to HarryMichal/toolbox that referenced this issue Oct 1, 2020
The redirectPath function used to error out when handling directories,
if the path inside the container was initially absent. There's no real
reason for this, and some containers failed to start because the
/media directory was absent from them. This can happen as a
consequence of Fedora's filesystem RPM failing 'dnf update'
transactions inside containers:
  $ toolbox enter f33-to-upgrade
  $ dnf --assumeyes --enablerepo=fedora --disablerepo=rawhide update
  <...>
  Failed:
    filesystem-3.14-2.fc32.x86_64
    filesystem-3.14-3.fc33.x86_64

  Error: Transaction failed

Therefore, it's better to not worry about the path being initially
absent, and be more forgiving and robust.

containers#539
@debarshiray
Copy link
Member

@debarshiray debarshiray commented Oct 1, 2020

I'm puzzled why the containerPath doesn't exist in the first place.
In this case, where containerPath is /media, I don't understand
why it went missing from the image. eg., I pulled
registry.fedoraproject.org/f33/fedora-toolbox:33 and it still has a /media.

I'd say it was just something weird that happened to my toolbox, except that
it also happened to @bam80 - I agree that every f33 toolbox has shipped
with a /media. We still have problems with the filesystem package on
'dnf update' inside a container - maybe something happened there?

That's a good point. Thanks for the reproducer.

The shell-like 'if [ -e /media ]' seems fine to me - it's infinitesimally less
efficient than handling ENOENT, but makes the code easier to read, and
there shouldn't be any risk of race conditions in this case - the toolbox init is
the only thing manipulating the container.

True, but my OCD still compelled me to change it to handle ENOENT instead. :P

@debarshiray debarshiray changed the title can't enter fedora 33 containers: Error: failed to redirect /media to /run/media Container doesn't start because of missing /media possibly due to failed RPM transaction Oct 1, 2020
@bam80
Copy link
Author

@bam80 bam80 commented Oct 2, 2020

We still have problems with the filesystem package on 'dnf update' inside a container - maybe something happened there?

@owtaylor thanks for narrowing it down! Do we have problem with the filesystem package reported?

@bam80
Copy link
Author

@bam80 bam80 commented Oct 2, 2020

Doing: echo '%_netsharedpath /proc:/sys:/dev' > /etc/rpm/macros.toolbox - before upgrading helps some - the transaction still fails because filesystem fails verification (permissons are wrong on /proc /sys /dev), but things are unpacked correctly and the image is left with a /media directory.

After these manipulations, /media link replaced with /media dir on upgrade, not sure if it correct.

likan999 added a commit to likan999/ppa-toolbox that referenced this issue Oct 30, 2020
The redirectPath function used to error out when handling directories,
if the path inside the container was initially absent. There's no real
reason for this, and some containers failed to start because the
/media directory was absent from them. This can happen as a
consequence of Fedora's filesystem RPM failing 'dnf update'
transactions inside containers:
  $ toolbox enter f33-to-upgrade
  $ dnf --assumeyes --enablerepo=fedora --disablerepo=rawhide update
  <...>
  Failed:
    filesystem-3.14-2.fc32.x86_64
    filesystem-3.14-3.fc33.x86_64

  Error: Transaction failed

Therefore, it's better to not worry about the path being initially
absent, and be more forgiving and robust.

containers#539
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

4 participants