cant ssh to any host. #125

qxmips · 2021-01-19T04:24:33Z

not sure what's happened but now I can't ssh to any host:

 ssh bastion@xxxxxxxx
~ Welcome to bastion.xxxxx.xxxx , xxxx, your last login was 00:00:08 ago (Tue 2021-01-19 04:20:43 UTC) from xxxxxxx)

185.144.208.251:39982 => qxmips@ip-xxxxxx.ec2.internal:22 => bastion@ip-xxxxx.ec2.internal:22 ...
 allowed ... log on(/home/qxmips/ttyrec/xxxxxx/2021-01-19.04-20-51.656242.c4d5847cbfe2.qxmips.bastion.1xxxxxxx.ttyrec)

 will try the following accesses you have:
  - group-member of vyos_infra with RSA-4096 key SHA256:xxxxxxxxxxxxxxxxxxxxx
  - [2021/01/14]

Connecting...
Usage: ttyrec [options] -- <command> [command options]

Usage (legacy compatibility mode): ttyrec -e <command> [options] [ttyrec file name]

Options:
  -z, --uuid UUID           specify an UUID (can be any string) that will appear in the ttyrec output file names,
                              and kept with SIGUSR1 rotations (default: own PID)
  -f, --output FILE         full path of the first ttyrec file to write to (autogenerated if omitted)
  -d, --dir FOLDER          folder where to write the ttyrec files (taken from -f if omitted,
                              defaulting to working directory if both -f and -d are omitted)
  -F, --name-format FMT     custom strftime-compatible format string to qualify the full path of the output files,
                              including the SIGUSR1 rotated ones
  -a, --append              open the ttyrec output file in append mode instead of write-clobber mode
  -Z                        enable on-the-fly compression if available, silently fallback to no compression if not
      --zstd                force on-the-fly compression of output file using zstd,
                              the resulting file will have a '.ttyrec.zst' extension
      --max-flush-time S    specify the maximum number of seconds after which we'll force zstd to flush its output buffers
                              to ensure that even somewhat quiet sessions gets regularly written out to disk, default is 15
  -l, --level LEVEL         set compression level, must be between 1 and 19 for zstd, default is 3
  -n, --count-bytes         count the number of bytes out and print it on termination (experimental)
  -t, --lock-timeout S      lock session on input timeout after S seconds
      --warn-before-lock S  warn S seconds before locking (see --lock-timeout)
  -k, --kill-timeout S      kill session on input timeout after S seconds
      --warn-before-kill S  warn S seconds before killing (see --kill-timeout)
  -C, --no-cheatcodes       disable cheat-codes (see below), this is the default
  -c, --cheatcodes          enable cheat-codes (see below)
  -p, --no-openpty          don't use openpty() even when it's available
  -T, --term MODE           MODE can be either 'never' (never allocate a pseudotty, even if stdin is a tty, and use pipes to
                              handle stdout/stderr instead), 'always' (always allocate a pseudotty, even if stdin is not a tty)
                              or 'auto' (default, allocate a pseudotty if stdin is a tty, uses pipes otherwise)
  -v, --verbose             verbose (debug) mode, use twice for more verbosity
  -V, --version             show version information
  -e, --shell-cmd CMD       enables legacy compatibility mode and specifies the command to be run under the user's $SHELL -c

Examples:
  Run some shell commands in legacy mode: ttyrec -e 'for i in a b c; do echo $i; done' outfile.ttyrec
  Run some shell commands in normal mode: ttyrec -f /tmp/normal.ttyrec -- sh -c 'for i in a b c; do echo $i; done'
  Connect to a remote machine interactively: ttyrec -t 60 -k 300 -- ssh remoteserver
  Execute a local script remotely with the default remote shell: ttyrec -- ssh remoteserver < script.sh
  Record a screen session: ttyrec screen

Handled signals:
  SIGUSR1     close current ttyrec file and reopen a new one (log rotation)
  SIGURG      lock session
  SIGUSR2     unlock session

Cheat-codes (magic keystrokes combinations):
  ^L^L^L^L^L^L^L^L   lock your session (that's 8 CTRL+L's)
  ^K^I^L^L^K^I^L^L   kill your session

Remark about session lock and session kill:
  If we don't have a tty, we can't lock, so -t will be ignored,
  whereas -k will be applied without warning, as there's no tty to output a warning to.
You specified --warn-before-kill without enabling --timeout-kill, this doesn't make sense

seems like ttyrec doesn't like some param.
how i can check what parameters is ttyrec is called with?

The text was updated successfully, but these errors were encountered:

qxmips · 2021-01-19T04:27:53Z

the version was v3.01.03 when issue appered . updated to v3.01.99-rc2 and
also re-executed /opt/bastion/bin/admin/install-ttyrec.sh -a

ttyrec v1.1.6.6 (x86_64-linux-gnu)
uses: zstd[static] isastream cfmakeraw getpt posix_openpt grantpt openpty[pty.h] (Linux)
compiler version 8.3.0 (gcc)
libzstd version 10308 (1.3.8

qxmips · 2021-01-19T04:52:26Z

enabled debug:

....
....
~ <6456:/opt/bastion/bin/shell/osh.pl> about to exec: ttyrec -f /home/qxmips/ttyrec/10.2.11.53/2021-01-19.04-44-25.873712.9a4308bf3479.qxmips.bastion.10.2.11.53.22.ttyrec -F /home/qxmips/ttyrec/10.2.11.53/%Y-%m-%d.%H-%M-%S.#usec#.9a4308bf3479.qxmips.bastion.10.2.11.53.22.ttyrec -v --warn-before-kill 30 -- /usr/bin/ssh 10.2.11.53 -l bastion -p 22 -i /home/keykeeper/keyaws_web/id_rsa4096_aws_web.1606094723 -o PreferredAuthentications=publickey
Connecting...
remaining non-parsed options argc=10

option 0: </usr/bin/ssh>

option 1: <10.2.11.53>

option 2: <-l>

option 3: <bastion>

option 4: <-p>

option 5: <22>

option 6: <-i>

option 7: </home/keykeeper/keyaws_web/id_rsa4096_aws_web.1606094723>

option 8: <-o>

option 9: <PreferredAuthentications=publickey>

Usage: ttyrec [options] -- <command> [command options]

Usage (legacy compatibility mode): ttyrec -e <command> [options] [ttyrec file name]

Options:
  -z, --uuid UUID           specify an UUID (can be any string) that will appear in the ttyrec output file names,
                              and kept with SIGUSR1 rotations (default: own PID)
  -f, --output FILE         full path of the first ttyrec file to write to (autogenerated if omitted)
  -d, --dir FOLDER          folder where to write the ttyrec files (taken from -f if omitted,
                              defaulting to working directory if both -f and -d are omitted)
  -F, --name-format FMT     custom strftime-compatible format string to qualify the full path of the output files,
                              including the SIGUSR1 rotated ones
  -a, --append              open the ttyrec output file in append mode instead of write-clobber mode
  -Z                        enable on-the-fly compression if available, silently fallback to no compression if not
      --zstd                force on-the-fly compression of output file using zstd,
                              the resulting file will have a '.ttyrec.zst' extension
      --max-flush-time S    specify the maximum number of seconds after which we'll force zstd to flush its output buffers
                              to ensure that even somewhat quiet sessions gets regularly written out to disk, default is 15
  -l, --level LEVEL         set compression level, must be between 1 and 19 for zstd, default is 3
  -n, --count-bytes         count the number of bytes out and print it on termination (experimental)
  -t, --lock-timeout S      lock session on input timeout after S seconds
      --warn-before-lock S  warn S seconds before locking (see --lock-timeout)
  -k, --kill-timeout S      kill session on input timeout after S seconds
      --warn-before-kill S  warn S seconds before killing (see --kill-timeout)
  -C, --no-cheatcodes       disable cheat-codes (see below), this is the default
  -c, --cheatcodes          enable cheat-codes (see below)
  -p, --no-openpty          don't use openpty() even when it's available
  -T, --term MODE           MODE can be either 'never' (never allocate a pseudotty, even if stdin is a tty, and use pipes to
                              handle stdout/stderr instead), 'always' (always allocate a pseudotty, even if stdin is not a tty)
                              or 'auto' (default, allocate a pseudotty if stdin is a tty, uses pipes otherwise)
  -v, --verbose             verbose (debug) mode, use twice for more verbosity
  -V, --version             show version information
  -e, --shell-cmd CMD       enables legacy compatibility mode and specifies the command to be run under the user's $SHELL -c

Examples:
  Run some shell commands in legacy mode: ttyrec -e 'for i in a b c; do echo $i; done' outfile.ttyrec
  Run some shell commands in normal mode: ttyrec -f /tmp/normal.ttyrec -- sh -c 'for i in a b c; do echo $i; done'
  Connect to a remote machine interactively: ttyrec -t 60 -k 300 -- ssh remoteserver
  Execute a local script remotely with the default remote shell: ttyrec -- ssh remoteserver < script.sh
  Record a screen session: ttyrec screen

Handled signals:
  SIGUSR1     close current ttyrec file and reopen a new one (log rotation)
  SIGURG      lock session
  SIGUSR2     unlock session

Cheat-codes (magic keystrokes combinations):
  ^L^L^L^L^L^L^L^L   lock your session (that's 8 CTRL+L's)
  ^K^I^L^L^K^I^L^L   kill your session

Remark about session lock and session kill:
  If we don't have a tty, we can't lock, so -t will be ignored,
  whereas -k will be applied without warning, as there's no tty to output a warning to.
You specified --warn-before-kill without enabling --timeout-kill, this doesn't make sense

~ <6435:/opt/bastion/bin/shell/osh.pl> about to run_cmd ['id','-G','-n']
~ <6435:/opt/bastion/bin/shell/osh.pl> waiting for child PID 6459 to complete...
~ <6435:/opt/bastion/bin/shell/osh.pl> stdout(6459): qxmips bastion-users osh-accountRevokeCommand osh-accountUnexpire osh-accountPIV osh-accountDelPersonalAccess osh-accountGeneratePassword osh-realmDelete osh-realmList osh-selfAddPersonalAccess osh-accountGrantCommand osh-accountListPasswords osh-groupCreate osh-whoHasAccessTo osh-accountListAccesses osh-selfDelPersonalAccess osh-accountMFAResetPassword osh-realmInfo osh-accountListIngressKeys osh-rootListIngressKeys osh-accountMFAResetTOTP osh-accountDelete osh-accountAddPersonalAccess osh-groupDelete osh-realmCreate osh-accountCreate osh-accountList osh-accountListEgressKeys osh-accountModify osh-accountInfo osh-auditor osh-admin keyaws_web keyaws_web-aclkeeper keyaws_web-gatekeeper keyaws_web-owner keyvyos_infra keyvyos_infra-aclkeeper keyvyos_infra-gatekeeper keyvyos_infra-owner qxmips-tty
~ <6435:/opt/bastion/bin/shell/osh.pl> all fds are EOF, waiting for pid 6459 indefinitely
~ <6435:/opt/bastion/bin/shell/osh.pl> cmd returned with status 0
~ <6435:/opt/bastion/bin/shell/osh.pl> about to run_cmd ['id','-G','-n','qxmips']
~ <6435:/opt/bastion/bin/shell/osh.pl> waiting for child PID 6460 to complete...
~ <6435:/opt/bastion/bin/shell/osh.pl> stdout(6460): qxmips osh-admin osh-auditor osh-accountInfo osh-accountModify osh-accountListEgressKeys osh-accountList osh-accountCreate osh-realmCreate osh-groupDelete osh-accountAddPersonalAccess osh-accountDelete osh-accountMFAResetTOTP osh-rootListIngressKeys osh-accountListIngressKeys osh-realmInfo osh-accountMFAResetPassword osh-selfDelPersonalAccess osh-accountListAccesses osh-whoHasAccessTo osh-groupCreate osh-accountListPasswords osh-accountGrantCommand osh-selfAddPersonalAccess osh-realmList osh-realmDelete osh-accountGeneratePassword osh-accountDelPersonalAccess osh-accountPIV osh-accountUnexpire osh-accountRevokeCommand bastion-users qxmips-tty keyaws_web keyaws_web-aclkeeper keyaws_web-gatekeeper keyaws_web-owner keyvyos_infra keyvyos_infra-aclkeeper keyvyos_infra-gatekeeper keyvyos_infra-owner
~ <6435:/opt/bastion/bin/shell/osh.pl> all fds are EOF, waiting for pid 6460 indefinitely
~ <6435:/opt/bastion/bin/shell/osh.pl> cmd returned with status 0
qxmips@bastion.aws.us.sentrium.io(master)>

speed47 · 2021-01-19T09:07:19Z

It looks like you're having an old version of ttyrec, which seems strange because the latest The Bastion version bumps didn't change the ttyrec calls parameters.
Could you check that you don't have multiple ttyrec versions installed, for example one in /usr/local/bin and another one in /usr/bin, one being an older version?

EDIT: disregard that, I missed the
You specified --warn-before-kill without enabling --timeout-kill, this doesn't make sense line, bisecting.

speed47 · 2021-01-19T09:36:14Z

There were no recent change around that portion of code, did you change your /etc/bastion/bastion.conf recently, possibly setting warnBeforeKillSeconds without setting idleKillTimeout? The warnBeforeKillSeconds setting tells ttyrec to warn the user a few seconds/minutes before the idleKillTimeout timer kicks in and terminate their session. That's why setting a non-zero value to warnBeforeKillSeconds without enabling idle session killing through idleKillTimeout doesn't make sense.

I'll propose a patch to detect this case, so that the bastion ignores the warnBeforeKillSeconds setting if idleKillTimeout is not enabled, and log an error to syslog so that the sysadmin know they should adjust their configuration, to avoid completely breaking connections as it does now.

qxmips · 2021-01-19T09:42:46Z

@speed47 yep. that's it. warnBeforeKillSeconds was set without idleKillTimeout . Thank you

Before, an inconsistency in the configuration settings of the warnBefore(Lock|Kill)Seconds and idle(Lock|Kill)Timeout could break any new connection (ttyrec refuses to launch). Now we detect this case properly, and fallback to a sane setting for warnBefore(Lock|Kill)Seconds (zero) if those were set without enabling the corresponding idle(Lock|Kill)Timeout setting. We also log an error to syslog when it happens, so that the sysadmin can fix their configuration. Added hints about how these configuration options work together in the bastion.conf.dist file. Fixes #125

speed47 mentioned this issue Jan 19, 2021

enh: config: detect warnBefore/idleTimeout misconfiguration #126

Merged

speed47 closed this as completed in #126 Jan 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cant ssh to any host. #125

cant ssh to any host. #125

qxmips commented Jan 19, 2021

qxmips commented Jan 19, 2021

qxmips commented Jan 19, 2021

speed47 commented Jan 19, 2021 •

edited

speed47 commented Jan 19, 2021

qxmips commented Jan 19, 2021

cant ssh to any host. #125

cant ssh to any host. #125

Comments

qxmips commented Jan 19, 2021

qxmips commented Jan 19, 2021

qxmips commented Jan 19, 2021

speed47 commented Jan 19, 2021 • edited

speed47 commented Jan 19, 2021

qxmips commented Jan 19, 2021

speed47 commented Jan 19, 2021 •

edited