ruby-net-ssh: can't add a new key into hash during iteration during ssh.exec

[martin21]

I am currently packaging our own distkeys key distribution ruby script
(see http://bugs.debian.org/712787 RFS: distkeys/1.0 -- distribute SSH keys).

However it only works with Ruby 1.8 for now, as with Ruby 1.9 I get a error
back from ruby-net-ssh when trying to add or remove a key:

./distkeys -K somekey.pub -H somehost add

Host: somehost
Connecting to host somehost (user: ms, port: 9999)...
Opening SFTP session...
Key somekey added.
Creating a backup to .ssh/authorized_keys-2013-07-01.bak if not already done today...
Uploading keys to .ssh/authorized_keys-new...
File does exist and has correct size, moving to .ssh/authorized_keys...
/var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:299:in `[]=': can't add a new key into hash during iteration (RuntimeError)
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:299:in `open_channel'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:323:in `exec'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:357:in `exec!'
        from ./distkeys:206:in `block in commit'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `call'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `respond_to'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:948:in `dispatch_request'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:911:in `when_channel_polled'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/channel.rb:311:in `call'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/channel.rb:311:in `process'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:217:in `block in preprocess'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:217:in `each'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:217:in `preprocess'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:200:in `process'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `block in loop'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `loop'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `loop'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:72:in `wait'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:842:in `wait_for'
        from /var/lib/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:320:in `lstat!'
        from ./distkeys:200:in `commit'
        from ./distkeys:571:in `handle_host'
        from ./distkeys:677:in `block in handle_gwhost'
        from ./distkeys:660:in `each'
        from ./distkeys:660:in `handle_gwhost'
        from ./distkeys:692:in `loop'
        from ./distkeys:797:in `<main>'</code>

I tried with newest gems:

mango:~# gem install net-sftp net-ssh net-ssh-gateway
Fetching: net-ssh-2.6.7.gem (100%)
Fetching: net-sftp-2.1.2.gem (100%)
Successfully installed net-ssh-2.6.7
Successfully installed net-sftp-2.1.2
Successfully installed net-ssh-2.6.7

With Ruby 1.8 this works.

The code where this happens in distkeys as of commit
bf13f12e8ca3846998cc1cb610403ad958979377 (tag debian-bug-714606) to the git repo:

    request = @sftp.lstat!(newauthkeyfile) do | response |
        if response.ok?
            # File size okay?
            if response[:attrs].size >= wantedsize
                puts "File does exist and has correct size, moving to #{@authkeyfile}..."
                # Move the new keyfile over the old one
                @ssh.exec!(  "mv  #{newauthkeyfile} #{@authkeyfile}" )

                # We saved the changes, so no unsaved changes anymore
                @changed = false
            end
        end
    end
    #@sftp.loop

URL to git repo is:

git://oss.teamix.org/distkeys.git

I worked around this issue by using sftp.rename now. But this isn't optimal as openssh does not yet support sftp protocol version 5 which always for flag 0x0001 to overwrite a file and ruby-net-sftp does not support openssh vendor extension posix-rename. See:

Bug 2123 - does not seem to support sftp protocol version 5 rename flag for overwriting files
https://bugzilla.mindrot.org/show_bug.cgi?id=2123

Unless there is some programming mistake in distkeys that Ruby 1.9 brings
to light I bet this is an upstream bug.

Thanks,
Martin

[delano]

Thanks for the report. Ditto regarding maintenance mode but happy to pull in a patch!

[martin21]

Am Samstag, 6. Juli 2013 schrieb Delano Mandelbaum:

Thanks for the report. Ditto regarding maintenance mode but happy to
pull in a patch!

Thanks for notice. About a patch: I currently have no idea on why it fails
here and what that code does. Maybe after a closer look.

Martin Steigerwald - teamix GmbH - http://www.teamix.de
gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90

[delano]

Sounds good.

[mfazekas]

The issue comes from:
https://github.com/net-ssh/net-ssh/blob/master/lib/net/ssh/connection/session.rb#L223

channels.each { |id, channel| channel.process unless channel.local_closed? }

and we might be indirectly calling this from process:

channels[local_id] = channel

https://github.com/net-ssh/net-ssh/blob/master/lib/net/ssh/connection/session.rb#L306

We probably better to clone the channels before iteration, so in case we modify it there is no conflict in iteration and insertion.