Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move away from nss-altfiles (was: Messed up permissions in /var) #362

Open
shdwchn10 opened this issue Sep 30, 2022 · 7 comments
Open

Move away from nss-altfiles (was: Messed up permissions in /var) #362

shdwchn10 opened this issue Sep 30, 2022 · 7 comments
Labels
enhancement New feature or request fedora-change Needs a Fedora Change kinoite Also affect Fedora Kinoite

Comments

@shdwchn10
Copy link

Describe the bug
Sometimes rpm-ostree deploys cause wrong permissions randomly in /var.

To Reproduce
Please describe the steps needed to reproduce the bug:

  1. Install tor and tang. Additional installation of cockpit can increase chances of catching this bug.
  2. Enable tor and tang: systemctl enable --now tor.service and systemctl enable --now tang.socket.
  3. Make new deploy. I was able to reproduce this bug with following commands: rpm-ostree upgrade, rpm-ostree remove and rpm-ostree rebase.

Expected behavior
Correct permissions should persist.

Some examples (reproduced in clean VMs)
System before tests:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit tang tor

Normal permissions state:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1 tang tang 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 tang tang 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root        144 Sep 29 00:50 .
drwxr-xr-x. 1 root    root        856 Sep 29 00:48 ..
-rw-------. 1 toranon toranon   20442 Sep 29 00:49 cached-certs
-rw-------. 1 toranon toranon 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 toranon toranon 8790462 Sep 29 00:50 cached-microdescs.new
drwx------. 1 toranon toranon       0 Sep 29 00:49 keys
-rw-------. 1 toranon toranon       0 Sep 29 00:49 lock
-rw-------. 1 toranon toranon    3466 Sep 29 00:50 state

Some results of reproducing this bug:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 11036
drwxr-x---. 1 toranon root     178 Sep 29 13:42 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2245089 Sep 29 13:38 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Sep 29 13:38 cached-microdescs
-rw-------. 1     962  962  228806 Sep 29 13:39 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Sep 29 13:38 lock
-rw-------. 1     962  962    4325 Sep 29 13:42 state

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 11036
drwxr-x---. 1 toranon root     178 Sep 29 13:51 .
drwxr-xr-x. 1 root    root     962 Sep 29 19:24 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2245089 Sep 29 13:38 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Sep 29 13:38 cached-microdescs
-rw-------. 1 tang    tang  228806 Sep 29 13:39 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Sep 29 13:44 lock
-rw-------. 1 tang    tang    4806 Sep 29 13:51 state

Additional context
Related to coreos/rpm-ostree#3179
@shdwchn10 shdwchn10 added the bug Something isn't working label Sep 30, 2022
@travier
Copy link
Member

travier commented Sep 30, 2022

As a short term workaround, I recommend writing a tmpfiles.d config that sets the right ownership at boot.
Medium term, converting to those packages to sysusers configs helps make them more reproduce-able.
Long term, we'll have to move away from nss-altfiles. See coreos/fedora-coreos-tracker#155 for more details (for Fedora CoreOS but it's likely the same issue).

@travier travier added enhancement New feature or request f36 Related to Fedora 36 and removed bug Something isn't working labels Sep 30, 2022
@travier
Copy link
Member

travier commented Sep 30, 2022

If you can compare the content of /etc/passwd, /usr/etc/passwd & same for group between each deployments that would be useful.

@shdwchn10
Copy link
Author

I did some retests and saw no difference difference in /etc/{passwd,group} and /usr/etc/{passwd,group} between deployments:

Before:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1 tang tang 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 tang tang 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root        144 Sep 29 00:50 .
drwxr-xr-x. 1 root    root        856 Sep 29 00:48 ..
-rw-------. 1 toranon toranon   20442 Sep 29 00:49 cached-certs
-rw-------. 1 toranon toranon 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 toranon toranon 8790462 Sep 29 00:50 cached-microdescs.new
drwx------. 1 toranon toranon       0 Sep 29 00:49 keys
-rw-------. 1 toranon toranon       0 Sep 29 00:49 lock
-rw-------. 1 toranon toranon    3466 Sep 29 00:50 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group 
root:x:0:
wheel:x:10:

1st test — $ rpm-ostree remove cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct  5 14:33 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Oct  5 14:26 cached-microdescs
-rw-------. 1     962  962       0 Oct  5 14:26 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Oct  5 14:26 lock
-rw-------. 1     962  962    3955 Oct  5 14:33 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group 
root:x:0:
wheel:x:10:

2nd test — $ rpm-ostree rebase fedora:fedora/$(($(rpm -E %fedora) + 1))/x86_64/silverblue:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/37/x86_64/silverblue
                  Version: 37.20221004.n.0 (2022-10-04T08:04:42Z)
               BaseCommit: 0d0cbfffdf49b08c94f66d97c86b41098fa144e724c5547a92c13e6097f99fe3
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: cockpit tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct  5 14:33 .
drwxr-xr-x. 1 root    root     926 Oct  5 15:31 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Oct  5 14:26 cached-microdescs
-rw-------. 1 tang    tang       0 Oct  5 14:26 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Oct  5 14:26 lock
-rw-------. 1 tang    tang    3955 Oct  5 14:33 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group
root:x:0:
wheel:x:10:

@travier
Copy link
Member

travier commented Oct 5, 2022

Sorry I meant /usr/lib/passwd & /usr/lib/group.

@travier travier added the need-info Further information is requested label Oct 5, 2022
@shdwchn10
Copy link
Author

Before is the same state as in the previous comment plus:
/usr/lib/passwd: https://pastebin.com/eaC3G6jn
/usr/lib/group: https://pastebin.com/EhjEzUKP

1st test — $ rpm-ostree remove cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct 24 17:35 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Oct 24 17:25 cached-microdescs
-rw-------. 1     962  962       0 Oct 24 17:25 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Oct 24 17:25 lock
-rw-------. 1     962  962    3956 Oct 24 17:35 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ sudo chown toranon:toranon -R /var/lib/tor
$ sudo chown tang:tang -R /var/db/tang

/usr/lib/passwd: https://pastebin.com/X6kcUkVX (diff from previous deploy: https://pastebin.com/D1edXzWW)
/usr/lib/group: https://pastebin.com/9CFQrc1T (diff from previous deploy: https://pastebin.com/ffZLgj6c)

2nd test — $ rpm-ostree rebase fedora:fedora/$(($(rpm -E %fedora) + 1))/x86_64/silverblue:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct 24 17:35 .
drwxr-xr-x. 1 root    root     926 Oct 24 17:48 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Oct 24 17:25 cached-microdescs
-rw-------. 1 tang    tang       0 Oct 24 17:25 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Oct 24 17:25 lock
-rw-------. 1 tang    tang    3956 Oct 24 17:35 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

/usr/lib/passwd: https://pastebin.com/nDKBXyTd (diff from previous deploy: https://pastebin.com/0BvAsS69)
/usr/lib/group: https://pastebin.com/fVnDUHF2 (diff from previous deploy: https://pastebin.com/HCqac2nA)

@travier travier added f37 Related to Fedora 37 rawhide f38 Related to Fedora 38 and removed need-info Further information is requested labels Oct 26, 2022
@travier
Copy link
Member

travier commented Oct 26, 2022 •
edited

The toranon user and group have a different IDs in each one of those deployments:

The easiest workaround for this issue is to make a tpmfiles.d config to make sure the directories are chown'ed on boot to the correct UIDs/GIDs:

$ cat /etc/tmpfiles.d/tor.conf
Z /var/lib/tor - toranon toranon - -

$ cat /etc/tmpfiles.d/tang.conf
Z /var/lib/tang - tang tang - -

@travier travier changed the title Messed up permissions in /var Move away from nss-altfiles (was: Messed up permissions in /var) Oct 26, 2022
@travier travier added the fedora-change Needs a Fedora Change label Oct 26, 2022
@travier travier removed f37 Related to Fedora 37 f36 Related to Fedora 36 f38 Related to Fedora 38 labels Nov 28, 2022
@travier
Copy link
Member

travier commented Dec 8, 2022 •
edited

Documentation: https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_unable_to_add_user_to_group

Other references:

@travier travier added the kinoite Also affect Fedora Kinoite label Dec 8, 2022
@travier travier removed the rawhide label May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request fedora-change Needs a Fedora Change kinoite Also affect Fedora Kinoite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@travier @shdwchn10