Serial console does not show automatic recovery scripts #1999

Signum · 2018-12-06T15:55:34Z

ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.4 / 2018-06-21
OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"): Debian 9
ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
```
OUTPUT=ISO
GRUB_RESCUE=y
BACKUP=BORG
…
```
Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR): Occurs on a Virtualbox VM and on an HP DL 380 G7.
System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): AMD64
Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): BIOS/GRUB
Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): Local, virtualized, /dev/sda.
Description of the issue (ideally so that others can reproduce it):
- Attach a serial console.
- Create a recovery boot medium as ISO.
- Boot from the ISO.
- Choose "Automatic recovery" from the GRUB menu.
- The VGA console shows the beginning of the automatic recovery. However the serial console stops the output right after the kernel messages.

Yes, this sounds a bit like #878 but either it's something different or a regression.

And this only happens when booting from the ISO image. It does not occur when using the GRUB menu entry "Relax-and-Recover" that can be added by GRUB_RESCUE=y.

/etc/default/grub has these options set:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200"
GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

Please let me know how I can further track that down.

Workaround, if any: None known. Only the VGA console can be used.
Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

Output on the serial console:

serial-console.txt

Output on the VGA console:

The text was updated successfully, but these errors were encountered:

gozora · 2018-12-06T18:53:20Z

Hello @Signum,

When you are in boot menu of your ISO (ReaR recovery system), assuming your are booting with isolinux (syslinux) press F2 for edit boot entry you wish to boot and try to manually add console=tty1 console=ttyS0,115200 maybe it will help ...

V.

jsmeix · 2018-12-07T09:06:53Z

@Signum
I do not use serial console myself but I guess you need to play around with
the USE_SERIAL_CONSOLE and KERNEL_CMDLINE config variables
to get it work in your particular case.

Every now and then we get issue reports related to the serial console
in the recovery system (which usually "just works") but as far as I noticed
things could get really weird if the serial console does not "just work".
I think often the root cause is not actually inside the ReaR recovery system
but the hardware or virtual machine setup that is related to serial console.

For details what really happens you need to inspect the scripts
that deal with the USE_SERIAL_CONSOLE config variable:
usr/share/rear/rescue/GNU/Linux/400_use_serial_console.sh
usr/share/rear/prep/GNU/Linux/200_include_serial_console.sh
usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh
usr/share/rear/lib/bootloader-functions.sh

Run rear -s mkrescue to see which of those scripts are run
in your particular case.

Signum · 2018-12-07T13:20:46Z

@gozora

When you are in boot menu of your ISO (ReaR recovery system), assuming your are booting with isolinux (syslinux) press F2 for edit boot entry you wish to boot and try to manually add console=tty1 console=ttyS0,115200 maybe it will help ...

It seems to be "Tab" rather than "F2". But, no, that doesn't work. The default grub commandline is:

┌──────────────────────────────────────────────────────────┐
│                  Relax-and-Recover v2.4                  │
├──────────────────────────────────────────────────────────┤
│ Recover reartest                                         │
│ Automatic Recover reartest                               │
│                                                          │
│ Other actions                                            │
│ Help for Relax-and-Recover                               │
│ Boot First Local disk (hd0)                              │
│ Boot Second Local disk (hd1)                             │
│ Boot Next device                                         │
│ Hardware Detection Tool                                  │
│ ReBoot system                                            │
│                                                          │
│                                                          │
└──────────────────────────────────────────────────────────┘

> kernel initrd=initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,115200 console=tty0 auto_recover

gozora · 2018-12-07T13:24:44Z

can you try to change to onsole=ttyS1,115200 console=tty1 virtual console console on your Proliant might be configured on COM1 ...

V.

Signum · 2018-12-07T13:42:54Z

@gozora
Thanks for the suggestions. But COM1/ttyS0 is the correct port. After all I get the GRUB menu and all kernel boot messages. Just on automatic recovery I don't see any output from the REAR scripts that is sent to tty0 only.

jsmeix · 2018-12-07T14:15:38Z

@Signum
do you mean that with non-automatic recovery (i.e. when you
select Recover reartest instead of Automatic Recover reartest)
then you see the output from the ReaR scripts on your serial console?

Signum · 2018-12-07T14:31:32Z

@jsmeix
Yes, exactly. As long as I login as root through the manual recovery all is fine. So I guess the grub/kernel start line makes login shells work. But the automatic recovery doesn't require a login and seems to start in a different way (that I have not understood yet).

Please take a look what appears on my serial console in case of manual recovery:

ISOLINUX 6.03 20171018  Copyright (C) 1994-2014 H. Peter Anvin et al

          ┌──────────────────────────────────────────────────────────┐
          │                  Relax-and-Recover v2.4                  │
          ├──────────────────────────────────────────────────────────┤
          │ Recover reartest                                         │
          │ Automatic Recover reartest                               │
          │                                                          │
          │ Other actions                                            │
          │ Help for Relax-and-Recover                               │
          │ Boot First Local disk (hd0)                              │
          │ Boot Second Local disk (hd1)                             │
          │ Boot Next device                                         │
          │ Hardware Detection Tool                                  │
          │ ReBoot system                                            │
          │                                                          │
          │                                                          │
          └──────────────────────────────────────────────────────────┘

          Press [Tab] to edit, [F2] for help, [F1] for version info



Rescue image kernel 4.9.0-8-amd64  Fri, 07 Dec 2018 14:16:02 +0100
BACKUP=REQUESTRESTORE OUTPUT=ISO 
Loading kernel... ok
Loading initrd.cgz...ok
[   000000000] Linux version 4.9.0-8-amd64((eeiinn-krree@@iists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.130-2 (2018-10-27)
[    0.000000] Command line: BOOT_IMAGE=kernel initrd=initrd.cgz root=/dev/ram0 vga=normal rw selinux=0 console=ttyS0,115200ccnnsole=tty0
[    0.000000] x8//pp::SSupporting XSAVE feature 0x001: 'x87 floating point registers'
[...]
[   16.598586] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[   16.728924] ip_tables: (C) 2000-2006 Netfilter Core Team
[   16.805222] random: systemd: uninitialized urandom read (16 bytes read)
[   16.835884] random: systemd: uninitialized urandom read (16 bytes read)
[   17.062305] systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[   17.265860] random: systemd: uninitialized urandom read (16 bytes read)



Relax-and-Recover 2.4 / 2018-06-21

Relax-and-Recover comes with ABSOLUTELY NO WARRANTY; for details see
the GNU General Public License at: http://www.gnu.org/licenses/gpl.html

Host reartest using Backup REQUESTRESTORE and Output ISO
Build date: Fri, 07 Dec 2018 14:15:35 +0100


Debian GNU/Linux 9 reartest ttyS0

reartest login: root
root

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE reartest:~ #

In automatic recovery mode the output stops right before "Relax-and-Recover 2.4 / 2018-06-21".

jsmeix · 2018-12-07T14:49:02Z

@Signum
I have absolutely no idea what the root cause could be here.

The recovery system startup script that implements
automatic (and unattended) recovery mode is
usr/share/rear/skel/default/etc/scripts/system-setup
e.g. the current one online at
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/skel/default/etc/scripts/system-setup

Note my comment therein before the sleep 1
i.e. I already had inexplicable issues with missing output.

Likely a useless attempt:
Add the kernel command line option debug as in
#1999 (comment)
which let's the recovery system startup scripts run with 'set -x'
but I assume you won't get any output on your serial console
because you get none after the last kernel message.

The difference between non-automatic recovery (the default case) and
automatic recovery is that in case of automatic (and unattended) recovery
the script usr/share/rear/skel/default/etc/scripts/system-setup
is not finished - i.e. automatic (and unattended) recovery are run
as "rear recover" calls within that script's environment and
I guess that within this environment there is no output on your
serial console - but I have no idea why.

Your
#1999 (comment)
indicates that there is no output on your serial console while
usr/share/rear/skel/default/etc/scripts/system-setup runs
(it runs as /etc/scripts/system-setup in the recovery system)
because there are none of the messages from that script like

Configuring Relax-and-Recover rescue system

(see the echo commands in that script).

jsmeix · 2018-12-07T15:35:04Z

Only a blind guess:
Perhaps something is missing to get serial console output
in the systemd unit files in the recovery system
that run /etc/scripts/system-setup like
usr/share/rear/skel/default/usr/lib/systemd/system/sysinit.service
usr/share/rear/skel/default/usr/lib/systemd/system/run-system-setup.service
or whatever script it actually is that executes /etc/scripts/system-setup
in a particular recovery system?

Signum · 2018-12-09T10:53:01Z

@jsmeix
I will look into that later today. But it appears that the systemd service is just sending output to the console (tty1) by default. systemd services can be told to use other devices, but not both tty1 and ttyS0 at the same time. Maybe if I understand how the kernel bot option "console=ttyS0" leads to also getting a getty on ttyS0 then I can copy that.

jsmeix · 2018-12-14T12:55:43Z

@Signum
are no news good news?

If you somehow solved it (or found a usable workaround)
we would appreciate feedback how you did it in your case
because that helps us to deal with similar issues in the future.

If you found a solution please provide it to us as GitHub pull request
even if your solution is perhaps not yet the "last word in wisdom"
we would appreciate to learn about any intermediate step forward.

Signum · 2018-12-14T14:27:14Z

@jsmeix Sorry for the delay. I had to shift my focus because we were facing massive troubles at work on a different topic. I hope I can come up with a proposal beginning of next week.
I was thinking about the difference of login shells (that are started on ttyS0 correctly) and the systemd-induced automatic recovery job… Would it be an option to always make the user login first and then run either "rear recover" as before or "rear autorecover" for a fully automated experience?

jsmeix · 2018-12-14T15:11:20Z

@Signum
I don't know how to automatically login as 'root' during startup
of the recovery system and how to let the auto-logged-in 'root'
automatically type rear recover and afterwards reboot
(depending whether or not if 'rear recover' was successful)
to keep the same automated experience as we have now.

I fear any scripted call of rear recover and afterwards reboot
might behave different compared to a real interactive shell
but I am not at all an expert in this area - in particular not
when systemd is involved...

gdha · 2018-12-14T15:14:26Z

@Signum perhaps have a look at https://github.com/gdha/rear-automated-testing ?

github-actions · 2020-06-28T01:33:41Z

Stale issue message

jsmeix added the support / question label Dec 7, 2018

jsmeix added the waiting for info label Dec 14, 2018

jsmeix self-assigned this Dec 14, 2018

github-actions bot added the no-issue-activity label Jun 28, 2020

github-actions bot closed this as completed Jul 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serial console does not show automatic recovery scripts #1999

Serial console does not show automatic recovery scripts #1999

Signum commented Dec 6, 2018 •

edited

gozora commented Dec 6, 2018

jsmeix commented Dec 7, 2018

Signum commented Dec 7, 2018

gozora commented Dec 7, 2018

Signum commented Dec 7, 2018

jsmeix commented Dec 7, 2018 •

edited

Signum commented Dec 7, 2018 •

edited

jsmeix commented Dec 7, 2018 •

edited

jsmeix commented Dec 7, 2018

Signum commented Dec 9, 2018

jsmeix commented Dec 14, 2018

Signum commented Dec 14, 2018

jsmeix commented Dec 14, 2018

gdha commented Dec 14, 2018

github-actions bot commented Jun 28, 2020

Serial console does not show automatic recovery scripts #1999

Serial console does not show automatic recovery scripts #1999

Comments

Signum commented Dec 6, 2018 • edited

gozora commented Dec 6, 2018

jsmeix commented Dec 7, 2018

Signum commented Dec 7, 2018

gozora commented Dec 7, 2018

Signum commented Dec 7, 2018

jsmeix commented Dec 7, 2018 • edited

Signum commented Dec 7, 2018 • edited

jsmeix commented Dec 7, 2018 • edited

jsmeix commented Dec 7, 2018

Signum commented Dec 9, 2018

jsmeix commented Dec 14, 2018

Signum commented Dec 14, 2018

jsmeix commented Dec 14, 2018

gdha commented Dec 14, 2018

github-actions bot commented Jun 28, 2020

Signum commented Dec 6, 2018 •

edited

jsmeix commented Dec 7, 2018 •

edited

Signum commented Dec 7, 2018 •

edited

jsmeix commented Dec 7, 2018 •

edited