Connection with two factor authentication failing

[djw8605]

I am using ssh2 to connect to a machine that runs Duo two factor authentication. When it attempts to connect, it says: ECONNRESET.

ssh2 version: ssh2@0.3.6
node version: v0.10.22 (node-webkit 0.8.6)

The order of operations seems to be:

1. Call connect with username / password.
2. Receive Keyboard Interactive event with a prompt asking for the password again.
3. Receive another Keyboard Interactive, this time with the Duo prompt.
4. After successful duo authentication, ECONNRESET.

A gist with the debug output is here: https://gist.github.com/djw8605/90c30d3c7d3a168fe3e4

It should be noted, that ssh2 works just fine when Duo is not involved, on the same server version (SSH-2.0-OpenSSH_5.3).

Any help would be greatly appreciated.

[mscdex]

Are you able to get debug output (turned all the way up) from OpenSSH on the server side as well? I don't see anything wrong protocol-wise in the client-side debug log you posted.

[mscdex]

Also, are you using the Duo Unix setup or are you using their "Native SSH"?

[djw8605]

We are using Duo Unix.

I was able to set the debug level for SSHD up to debug3. I created a gist with 3 files:

1. The failed sshd debugging output.
2. The failed ssh2js debug output.
3. A successful sshd debugging output (using command line ssh from a Mac)

I tried to sanitize the output somewhat. I'm hoping I did enough.
https://gist.github.com/djw8605/e6e2a99869496e179c90

From my reading of the debug ouptut, I'm not seeing anything obvious. It's the PAM layer that isn't allowing me in, but then it shouldn't allow me in with the command line either then. There must be a protocol difference somewhere.

[mscdex]

I just tested ssh2 with Duo Unix and OpenSSH 6.6 and it works for me. Here's the contents of my /etc/pam.d/common-auth on the test Ubuntu 14.04 VM I used:

auth    requisite      pam_unix.so nullok_secure
auth    [success=1 default=ignore]      pam_duo.so
auth    requisite                       pam_deny.so
auth    required                        pam_permit.so

I also made sure to set these in /etc/ssh/sshd_config:

UsePAM yes
ChallengeResponseAuthentication yes
UseDNS no

And here is the test code I used:

var Connection = require('ssh2');
var readline = require('readline');
var conn = new Connection();

var rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout
});

conn.on('keyboard-interactive', function redo(name, instructions, instructionsLang, prompts, finish, answers) {
  answers = answers || [];
  if (answers.length < prompts.length) {
    rl.question(prompts[answers.length].prompt, function(answer) {
      answers.push(answer);
      redo(name, instructions, instructionsLang, prompts, finish, answers)
    });
  } else
    finish(answers);
});
conn.on('ready', function() {
  rl.close();
  conn.exec('uptime', function(err, stream) {
    if (err) throw err;
    stream.stdout.pipe(process.stdout);
    stream.stderr.pipe(process.stderr);
    stream.on('exit', function() {
      conn.end();
    });
  });
});

conn.connect({
  host: '192.168.10.5',
  port: 22,
  username: 'foo',
  tryKeyboard: true,
  readyTimeout: 99999999 // this gives us more than enough time to manually answer prompts
});

If you are still having problems, you might check /var/log/syslog for possibly more PAM debug output. That helped me when I had it working initially but then it suddenly stopped working. The syslog showed that Duo had locked out my account while I was debugging. After I had unlocked my account through the Duo admin site, it was working again.

[djw8605]

Thanks @mscdex for the debugging help.

I tested your code, and it worked immediately. I re-examined my code, and the only difference was that I was providing the password in the connect call. So I added the password to your code, and it did not work. Removing the password from the connect call in my code immediately made it work. I'm not entirely sure why providing a password makes duo connections fail?

FWIW, on the CentOS 5 machine that is running this in production (I know, ancient. But production is production...), the file /etc/pam.d/system-auth contains the line:

auth        sufficient    pam_duo.so

Otherwise, the configurations match.

[mscdex]

Are you prompted for a password and then the duo auth prompt using the code (with no password set in the ssh2 config) and configuration I provided?

[djw8605]

Yes. I am prompted a password then the Duo Auth prompt with your exact code, with no password set.

I am also prompted for a password and then the Duo auth when I provide a password in the connect call.

[mscdex]

Well that part makes sense. I'm not sure why it would be failing when password is set though. I will try to reproduce it tonight.

[djw8605]

Hi @mscdex, where you able to replicate my issue with password being set?

Thanks

[mscdex]

Unfortunately I haven't been able to reproduce it yet.

[zhulduz]

I have the same problem. If I remove password from connect it works fine for me. But with password it doesn't work.

[zhulduz]

I guess this information could be useful.

var Client = require('ssh2').Client;

var conn = new Client();
conn.on('ready', function () {
    console.log('Client :: ready');

    conn.shell(function (err, stream) {
        if (err)
            console.log(err);
        stream.on('close', function () {
            console.log('Stream :: close');
            conn.end();
        }).on('data', function (data) {
            console.log('STDOUT: ' + data.toString());
        }).stderr.on('data', function (data) {
                console.log('STDERR: ' + data.toString());
            });
        stream.write('exit\n');
    });

}).on('keyboard-interactive', function(name,instructions,lang,prompts,finish) {
    if(prompts[0].prompt.indexOf("Password:") != -1) {
        finish(["password"])
    } else {
        finish(["verification_code"])
    };

}).on('error', function(err) {
    console.log("Error", err.message);
}).connect({
    host: 'address',
    port: 2124,
    username: 'user',
    tryKeyboard: true,
    password: "password",
    debug: console.log
    }
);

Also debug logs https://gist.github.com/zhulduz/88d14479c1cae81f3658

[mscdex]

@zhulduz Can you provide debug logs (with debug level turned all the way up) from the sshd process running on the server?

@djw8605 Looking at your sshd error log again, I noticed that something in PAM failed. You may need to add (at least) something like auth required pam_unix_cred.so.1 to /etc/pam.d/sshd or similar. It seems like the appropriate PAM module(s) are not getting properly loaded when OpenSSH is interacting with PAM.

[zhulduz]

Of course

Dec  1 11:44:39 SAJSDev sshd(pam_google_authenticator)[31422]: Invalid verification code
Dec  1 11:44:40 SAJSDev sshd[31422]: Failed password for 'user' from 'address' port 50355 ssh2
Dec  1 11:44:40 SAJSDev sshd[31422]: Accepted keyboard-interactive/pam for 'user' from 'address' port 50355 ssh2
Dec  1 11:44:40 SAJSDev sshd[31422]: fatal: PAM: pam_setcred(): Permission denied

[mscdex]

@zhulduz Is PermitEmptyPassword yes set in /etc/ssh/sshd_config on the server?

[zhulduz]

I have checked PermitEmptyPasswords parameter with yes and no values. It doesn't work yet.

[mscdex]

@zhulduz Can you try with the last version of ssh2 and let me know if this still happens?

[zhulduz]

@mscdex it is ok now) Thanks

[mscdex]

Closing this for now. If this is still an issue with the most recent version of ssh2 (and ssh2-streams), let me know if there is a way for me to reproduce the issue locally.

[InputOutputZ]

Of course

Dec  1 11:44:39 SAJSDev sshd(pam_google_authenticator)[31422]: Invalid verification code
Dec  1 11:44:40 SAJSDev sshd[31422]: Failed password for 'user' from 'address' port 50355 ssh2
Dec  1 11:44:40 SAJSDev sshd[31422]: Accepted keyboard-interactive/pam for 'user' from 'address' port 50355 ssh2
Dec  1 11:44:40 SAJSDev sshd[31422]: fatal: PAM: pam_setcred(): Permission denied

Set in sshd_config

UsePAM yes
AuthenticationMethods keyboard-interactive

This will fix the pam_setcred permission denied issue.