Authentication failed for user ...

[nrocco]

We have recently upgraded to the latest version of oxidized v0.24.0, previously we were running 0.19.0.

Since we upgraded we are seeing issues with backing up EOS devices. Some EOS devices backup just fine, but most of them fail with the below debug messages:

D, [2018-09-27T13:52:56.070614 #27849] DEBUG -- : lib/oxidized/worker.rb: Added 10.29.255.13 to the job queue
D, [2018-09-27T13:52:56.071001 #27849] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
D, [2018-09-27T13:52:56.071548 #27849] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for 10.29.255.13 at 2018-09-27 13:52:56 UTC
D, [2018-09-27T13:52:56.072437 #27849] DEBUG -- : AUTH METHODS::["none", "publickey", "password"]
D, [2018-09-27T13:52:56.073006 #27849] DEBUG -- : lib/oxidized/input/ssh.rb: Connecting to 10.29.255.13
W, [2018-09-27T13:52:56.411443 #27849]  WARN -- : 10.29.255.13 raised Net::SSH::AuthenticationFailed with msg "Authentication failed for user xyz@10.29.255.13"
D, [2018-09-27T13:52:56.412078 #27849] DEBUG -- : lib/oxidized/node.rb: Oxidized::SSH failed for 10.29.255.13
D, [2018-09-27T13:52:56.412589 #27849] DEBUG -- : lib/oxidized/job.rb: Config fetched for 10.29.255.13 at 2018-09-27 13:52:56 UTC
W, [2018-09-27T13:52:57.072063 #27849]  WARN -- : 10.29.255.13 status no_connection, retries exhausted, giving up

But when manually ssh'ing from the cli to the above network device we are able to connect and run commands like show running-config.

Also we tried using the net/ssh gem directly which also works just fine

require 'net/ssh'

Net::SSH.start('10.29.255.13', 'xyz', password: "xxxxxx") do |ssh|
  output = ssh.exec!("show running-config | no-more | exclude ! Time:")
  puts output
end

The above ruby code works as expected.

Using oxidized-scripts we get

oxidized@server:~$ time oxs --verbose --debug 10.29.255.13 'sh ver'
unable to connect
/var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/lib/oxidized/script/script.rb:113:in `connect': unable to connect (Oxidized::Script::NoConnection)
	from /var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/lib/oxidized/script/script.rb:92:in `initialize'
	from /var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/lib/oxidized/script/cli.rb:122:in `new'
	from /var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/lib/oxidized/script/cli.rb:122:in `connect'
	from /var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/lib/oxidized/script/cli.rb:37:in `run'
	from /var/lib/gems/2.3.0/gems/oxidized-script-0.5.1/bin/oxs:5:in `<top (required)>'
	from /usr/local/bin/oxs:23:in `load'
	from /usr/local/bin/oxs:23:in `<main>'

real	0m0.608s
user	0m0.280s
sys	0m0.048s

Our oxidized configuration:

oxidized@app-oxidized:~$ cat config/oxidized/config
---
interval: 3600
log: /var/log/oxidized/oxidized.log
debug: true
threads: 30
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
vars:
  remove_secret: true
groups: {}
rest: 127.0.0.1:8888
input:
  default: ssh
  debug: true
  ssh:
    secure: false
output:
  default: git
  git:
    user: "Oxidized CI"
    email: "xxx@yyy.com"
    repo: "/var/lib/oxidized/config/oxidized/oxidized.git"
hooks:
  push_to_remote:
    type: githubrepo
    events: [post_store]
    remote_repo: git@xxx.yyy.com/infra/oxidized.git
    publickey: /var/lib/oxidized/.ssh/id_rsa.pub
    privatekey: /var/lib/oxidized/.ssh/id_rsa
source:
  default: csv
  csv:
    file: /var/lib/oxidized/config/oxidized/router.db
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
    vars_map:
      ssh_proxy: 4
      enable: 5
model_map:
  cisco: ios
  juniper: junos
  arista: eos
  hp: procurve
  a10: acos

Can you give any pointers how to troubleshoot the above issue correctly?

The password for the above network device only contains uppercase, lowercase and digits. So no special characters that could conflict with the separator value in router.db.

[wk]

Have you reviewed the model notes regarding authentication?

[nrocco]

@wk I cannot believe how I missed that documentation. Setting auth methods to

  auth_methods: [ "none", "publickey", "password", "keyboard-interactive" ]

seems to make backups for most of our EOS devices work again.

We are currently rolling this out but for now I will assume that this is a non issue and close it.

Thanks for the quick help.