New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed Units: 1 systemd-tmpfiles-setup.service #1763

Closed
Knight1 opened this Issue Jan 15, 2017 · 11 comments

Comments

Projects
None yet
5 participants
@Knight1

Knight1 commented Jan 15, 2017

Issue Report

Container Linux by CoreOS beta (1248.4.0)
Failed Units: 1
systemd-tmpfiles-setup.service

CoreOS Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1248.4.0
VERSION_ID=1248.4.0
BUILD_ID=2017-01-10-0554
PRETTY_NAME="Container Linux by CoreOS 1248.4.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Environment

What hardware/cloud provider/hypervisor is being used to run CoreOS?

Expected Behavior

Actual Behavior

core@service-failure-test ~ $ journalctl -xe
Jan 15 21:22:32 service-failure-test systemd-tmpfiles[1560]: Failed to copy files to /var/lib/nfs/etab: No such file or directory
Jan 15 21:22:32 service-failure-test systemd-tmpfiles[1560]: Failed to copy files to /var/lib/nfs/rmtab: No such file or directory
Jan 15 21:22:32 service-failure-test systemd-tmpfiles[1560]: Failed to copy files to /var/lib/nfs/state: No such file or directory
Jan 15 21:22:32 service-failure-test systemd-tmpfiles[1560]: Failed to copy files to /var/lib/nfs/xtab: No such file or directory
Jan 15 21:22:32 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 21:22:32 service-failure-test systemd[1]: Failed to start Create Volatile Files and Directories.
-- Subject: Unit systemd-tmpfiles-setup.service has failed

core@service-failure-test ~ $ systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2017-01-15 21:22:32 UTC; 2min 37s ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
 Main PID: 1560 (code=exited, status=1/FAILURE)

Jan 15 21:22:32 service-failure-test systemd[1]: Starting Create Volatile Files and Directories...
Jan 15 21:22:32 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jan 15 21:22:32 service-failure-test systemd[1]: Failed to start Create Volatile Files and Directories.
Jan 15 21:22:32 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Jan 15 21:22:32 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.

Reproduction Steps

  1. installing CoreOS beta (1235.2.0) on packet.net
  2. update_engine_client -update
Broadcast message from locksmithd at 2017-01-15 21:15:04.929281 +0000 UTC:
System reboot in 5 minutes!

LAST_CHECKED_TIME=1484514809
PROGRESS=0.000000
CURRENT_OP=UPDATE_STATUS_UPDATED_NEED_REBOOT
NEW_VERSION=0.0.0.0
NEW_SIZE=206162545
I0115 21:15:05.419867  2635 update_engine_client.cc:194] Update succeeded -- reboot needed.
core@service-failure-test ~ $ reboot

Other Information

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run CoreOS?

ARMv8 on bare metal 96-core server at packet.net

Desired Feature

No error message 😅

Other Information

Thanks for any help!

@glevand

This comment has been minimized.

Show comment
Hide comment
@glevand

glevand Jan 17, 2017

I found this:

# egrep etab /usr/lib/tmpfiles.d/nfs-utils.conf
C /var/lib/nfs/etab     -       -       -       -       /usr/lib/nfs/etab

# ls -l /var/lib/nfs/etab /usr/lib/nfs
ls: cannot access '/usr/lib/nfs': No such file or directory
-rw-r--r-- 1 root root 0 Dec 14 23:19 /var/lib/nfs/etab

I'll look at it some more tomorrow.

glevand commented Jan 17, 2017

I found this:

# egrep etab /usr/lib/tmpfiles.d/nfs-utils.conf
C /var/lib/nfs/etab     -       -       -       -       /usr/lib/nfs/etab

# ls -l /var/lib/nfs/etab /usr/lib/nfs
ls: cannot access '/usr/lib/nfs': No such file or directory
-rw-r--r-- 1 root root 0 Dec 14 23:19 /var/lib/nfs/etab

I'll look at it some more tomorrow.

@vielmetti

This comment has been minimized.

Show comment
Hide comment
@vielmetti

vielmetti commented Jan 17, 2017

This commit by @dm0- looks relevant

coreos/coreos-overlay@585e907

@vielmetti

This comment has been minimized.

Show comment
Hide comment
@vielmetti

vielmetti Jan 17, 2017

and this PR looks relevant as well coreos/coreos-overlay#2284

vielmetti commented Jan 17, 2017

and this PR looks relevant as well coreos/coreos-overlay#2284

@glevand

This comment has been minimized.

Show comment
Hide comment
@glevand

glevand commented Jan 17, 2017

Should be fixed by coreos/coreos-overlay@2bb3577.

glevand added a commit to coreos/coreos-overlay that referenced this issue Jan 17, 2017

net-fs/nfs-utils: Fix arm64 systemd-tmpfiles-setup
Update nfs-utils.conf to use /usr/lib64.

Commit "585e9070f2c1 (net-fs/nfs-utils: copy default files into /var)
made a change to nfs-utils.conf that copies files using the symbolic
link '/usr/lib'.  This link is not available on arm64, so broke the
systemd-tmpfiles-setup service on arm64 systems.

Fixes coreos/bugs#1763 (Failed Units: 1
systemd-tmpfiles-setup.service)

Signed-off-by: Geoff Levand <geoff@infradead.org>

@crawford crawford closed this Jan 17, 2017

@Knight1

This comment has been minimized.

Show comment
Hide comment
@Knight1

Knight1 Jan 27, 2017

Hi,

upgraded to 1248.4.0 and the bug still exists and the errors in the logs are the same.

core@service-failure-test ~ $ systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-01-27 20:59:53 UTC; 19min ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
 Main PID: 1510 (code=exited, status=1/FAILURE)

Jan 27 20:59:53 service-failure-test systemd[1]: Starting Create Volatile Files and Directories...
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/etab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/rmtab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/state: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/xtab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jan 27 20:59:53 service-failure-test systemd[1]: Failed to start Create Volatile Files and Directories.
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
core@service-failure-test ~ $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1248.4.0
VERSION_ID=1248.4.0
BUILD_ID=2017-01-10-0554
PRETTY_NAME="Container Linux by CoreOS 1248.4.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Thanks

Knight1 commented Jan 27, 2017

Hi,

upgraded to 1248.4.0 and the bug still exists and the errors in the logs are the same.

core@service-failure-test ~ $ systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2017-01-27 20:59:53 UTC; 19min ago
     Docs: man:tmpfiles.d(5)
           man:systemd-tmpfiles(8)
 Main PID: 1510 (code=exited, status=1/FAILURE)

Jan 27 20:59:53 service-failure-test systemd[1]: Starting Create Volatile Files and Directories...
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/etab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/rmtab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/state: No such file or directory
Jan 27 20:59:53 service-failure-test systemd-tmpfiles[1510]: Failed to copy files to /var/lib/nfs/xtab: No such file or directory
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jan 27 20:59:53 service-failure-test systemd[1]: Failed to start Create Volatile Files and Directories.
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Jan 27 20:59:53 service-failure-test systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
core@service-failure-test ~ $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1248.4.0
VERSION_ID=1248.4.0
BUILD_ID=2017-01-10-0554
PRETTY_NAME="Container Linux by CoreOS 1248.4.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Thanks

@crawford

This comment has been minimized.

Show comment
Hide comment
@crawford

crawford Feb 7, 2017

Member

@Knight1 the fix made it into 1298.1.0.

Member

crawford commented Feb 7, 2017

@Knight1 the fix made it into 1298.1.0.

@vielmetti

This comment has been minimized.

Show comment
Hide comment
@vielmetti

vielmetti Mar 13, 2017

Any chance this fix can be backported into stable? I'm seeing this:

Container Linux by CoreOS stable (1235.9.0)
Failed Units: 1
  systemd-tmpfiles-setup.service

on the same machine type @Knight1 is looking at.

vielmetti commented Mar 13, 2017

Any chance this fix can be backported into stable? I'm seeing this:

Container Linux by CoreOS stable (1235.9.0)
Failed Units: 1
  systemd-tmpfiles-setup.service

on the same machine type @Knight1 is looking at.

@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Mar 13, 2017

Member

Stable is currently at 1298.5.0 (https://stable.release.core-os.net/amd64-usr/current/version.txt), so it should be fixed in all channels.

Member

dm0- commented Mar 13, 2017

Stable is currently at 1298.5.0 (https://stable.release.core-os.net/amd64-usr/current/version.txt), so it should be fixed in all channels.

@vielmetti

This comment has been minimized.

Show comment
Hide comment
@vielmetti

vielmetti Mar 13, 2017

Hm. arm64 is at 1235.9.0 (https://stable.release.core-os.net/arm64-usr/current/version.txt) if I read it right, so it's lagging behind stable for amd64?

vielmetti commented Mar 13, 2017

Hm. arm64 is at 1235.9.0 (https://stable.release.core-os.net/arm64-usr/current/version.txt) if I read it right, so it's lagging behind stable for amd64?

@dm0-

This comment has been minimized.

Show comment
Hide comment
@dm0-

dm0- Mar 13, 2017

Member

Oh, I don't think the arm64 stable channel is supported or officially exists (i.e. there is no auto-update group for it), but some releases accidentally got published for it there. A new stable version should go out around Wednesday, and I could release that for arm64 stable, but you should probably switch any ARM systems to the beta channel if you want to get automatic updates.

Member

dm0- commented Mar 13, 2017

Oh, I don't think the arm64 stable channel is supported or officially exists (i.e. there is no auto-update group for it), but some releases accidentally got published for it there. A new stable version should go out around Wednesday, and I could release that for arm64 stable, but you should probably switch any ARM systems to the beta channel if you want to get automatic updates.

@vielmetti

This comment has been minimized.

Show comment
Hide comment
@vielmetti

vielmetti Mar 13, 2017

Thanks @dm0- that explains the situation.

vielmetti commented Mar 13, 2017

Thanks @dm0- that explains the situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment