At shutdown: Shells loosing history and group based services killed immediately

[bitstreamout]

Several users report a similar behaviour. That is at shutdown it happens very often that the used shell looses its history and sometimes the full history get lost. An other problem which seems also caused by killing immediately are group based services with e service section like

[Service]
Type=simple
User=java
Group=users
ExecStart=/usr/bin/java_app

after debugging the commit 743970d has been identified as the cause of this behaviour. After searching it looks like this problem is well known:

https://bugzilla.redhat.com/show_bug.cgi?id=1141137
https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=experimental&id=6068e872da

What is the problem with
"... for nspawn we actually can't wait for cgroups running empty since systemd will get exactly zero notifications about it .."
and can this be solved in an other way instead immediately killing shell and group based services?

[michich]

Correct RH BZ link: https://bugzilla.redhat.com/show_bug.cgi?id=1141137

[bitstreamout]

Thanks for spotting!

[poettering]

@bitstreamout hmm, there's some confusion here. Commit 743970d is only relevant for units for which no main/control PID is known. That's only scope units, and service units that use Type=forking without PIDFile= and without GuessMainPID=, or where guessing the main PID fails. In your example service snippet this hence does not apply, because it uses Type=simple where the main PID is always known.

Yes, the issue applies to login sessions (which are exposed as scopes), but for them only at final shutdown, where the PAM session is not cleanly ended. For sessions that close during normal runtime the SIGKILL will not happen, because session scopes actually are left around by default until their last process ended.

The right fix for the whole mess is to port systemd to the unified cgroup logic, since we'll have proper notifications there for the first time, and can even wait for foreign processes going away nicely.

[bitstreamout]

I do not have the customers service unit here. I know that this java which is started by a bash wrapper script has its own sub processes. Maybe here pam drops in. Nevertheless the customer reports that reverting the specific commit 743970d does solve his problem. The problem of loosing the shell history I see my self every now and then on all systems at reboot.

[jamespharvey20]

Should 743970d be reverted until it can come back simultaneously with the "right fix" porting systemd to the unified cgroup logic?

Debian did so at http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=00aaecd71
Ubuntu did so at https://lists.ubuntu.com/archives/vivid-changes/2015-May/008558.html
Looks like RedHat may be doing so soon, at https://bugzilla.redhat.com/show_bug.cgi?id=1170765 and https://bugzilla.redhat.com/show_bug.cgi?id=1141137

Just filed a pull request to revert it.

[poettering]

@bitstreamout PAM has no place in starting services, that should really be fixed. The PAM session logic should be used for user sessions, but not for running arbitrary system processes.

[bitstreamout]

I'm aware but this is a foreign bash wrapper script I've never seen and AFAIK it is not documented that command like su/sudo or any other pam based tools should be avoided. On the other hand old SysVinit boot scripts had used su/sudo to switch user and group and if a third party vendor ports his complex scripts to systemd it might be that such a script becomes a wrapper script.

[bitstreamout]

Btw: How difficult it would become to detect in login/pam_systemd.c that it is within the chain of a service unit for boot instead of systemd login manager? For compatibility with SysV ;^)

[arvidjaar]

How difficult it would become to detect in login/pam_systemd.c that it is within the chain of a service > unit for boot instead of systemd login manager?

I suggested something similar as part of bnc#906900 (https://bugzilla.novell.com/show_bug.cgi?id=906900#c55) but nobody seemed to be interested (bug was "fixed" by rewriting sysvinit script). See https://build.opensuse.org/package/view_file/home:arvidjaar:bnc:906900/systemd/session-class-service.patch?expand=0

[bitstreamout]

In the menawhile a message from the customer had reached me that they could not wait any longer and they found workaround for this. Now they are using ExecStartPre= to read all options from /etc/sysconfig/ for their application and then they are writing this to /var/run/aem_author/env. This file is read by systemd as EnvironmentFile=. Later in ExecStart= they are just calling /usr/bin/java with EnvironmentFile as the variable parameter. In such configuration java is the main process and everything is working as should. Also he had found some ungracious about systemd!

@ arvidjaar : I had missed that patch, sorry about.
@ poettering : What do you think about this patch? This would allow to port complex SysVinit boot
scripts without workarounds. IMHO this would increase the acceptance for systemd a lot
if such ports would work without trouble.

Beside this the problem with loosing history data of the shells remains even with this patch.

[arvidjaar]

I had missed that patch, sorry about.

Second complementary patch is https://build.opensuse.org/package/view_file/home:arvidjaar:bnc:906900/systemd/set-session-class-service-for-sysv.patch?expand=0
Of course it is outdated and should be ported to generator but this is the idea.

[shaneforsythe]

There are currently 2 open TTS tickets for Fedora regarding this issue
https://bugzilla.redhat.com/show_bug.cgi?id=1141137
https://bugzilla.redhat.com/show_bug.cgi?id=1170765

And one here for Centos7.2
https://bugzilla.redhat.com/show_bug.cgi?id=1199644

(also see: https://bugs.launchpad.net/ubuntu/+source/mosh/+bug/1446982 )

Ubuntu,Debian, and Centos7.2 have now all reverted the commit in question to to data-integrity/data-loss.

Fedora is stuck because they rely on systemd as both upstream provider and maintainer, and from what I understand will not specifically "back port" for Fedora as they use the main upstream repo as well.

Reverting this commit may re-introduce another bug, but as I understand it that bug is a fairly specific edge-case and does not effect everyone, nor does it cause data loss.

This current bug effects everyone.

[bitstreamout]

@ poettering : is there any solution from upstream to avoid the method of jumping from the frying pan into the fire (aka `den Teufel mit dem Beelzebub austreiben')? Most customers/users do not like data loss.