Unable to notify remote exec command of an EOF of the stdin

[tsingakbar]

I'm trying to use sshj to implement something like cat foo.tar.gz | ssh user@remote tar xzf -, and I used the Command.getOutputStream() to feed these bytes from a local file named foo.tar.gz, but when all bytes is fed, I can't find a way to shutdown the process gracefully.
Invoking the close() method on the stream won't work.

[shikhar]

it's possible this got broken by d955865

[tsingakbar]

So if I uncomment it back in my project, will there be any problem?

[nstogner]

Did this fix the issue? I ran into the same thing

[hierynomus]

@tsingakbar @nstogner Could one of you verify that that fixes the problem, and write a unit test that shows this solves it? Then I'll readd the line. I don't know why it was removed, the commit message isn't really saying much... @shikhar Why was it removed?

[shikhar]

I was trying to further simplify teardown flow (as a follow-up to 5ee2f0a), and thought sending the EOF for stdout was redundant since in typical usage the channel would be closed soon after. This was probably a bad idea.

[hierynomus]

Commit d955865 has been reverted in commit 703a0df. This should fix this issue.

[hierynomus]

I'm commenting the line out again, just ran into the following:

2015-01-20 15:22:36.578 [TaskExecutionEngine-akka.actor.default-dispatcher-3]  DEBUG n.s.sshj.connection.ConnectionImpl - Attaching `session` channel (#0)
2015-01-20 15:22:36.731 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Initialized - < session channel: id=0, recipient=0, localWin=[winSize=2097152], remoteWin=[winSize=0] >
2015-01-20 15:22:36.731 [TaskExecutionEngine-akka.actor.default-dispatcher-3]  DEBUG n.s.s.c.c.direct.SessionChannel - Will request to exec `ls -ld /tmp/ot-20150120T152236244`
2015-01-20 15:22:36.731 [TaskExecutionEngine-akka.actor.default-dispatcher-3]  DEBUG n.s.s.c.c.direct.SessionChannel - Sending channel request for `exec`
2015-01-20 15:22:36.732 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Received window adjustment for 2097152 bytes
2015-01-20 15:22:36.733 [reader]  DEBUG n.s.s.c.channel.Window$Remote - Increasing by 2097152 up to 2097152
2015-01-20 15:22:36.736 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Got chan request for `exit-status`
2015-01-20 15:22:36.736 [reader]  DEBUG n.s.s.c.channel.Window$Local - Consuming by 72 down to 2097080
2015-01-20 15:22:36.736 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Got EOF
2015-01-20 15:22:36.736 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Got close
2015-01-20 15:22:36.737 [reader]  DEBUG n.s.s.c.c.direct.SessionChannel - Sending close
2015-01-20 15:22:36.737 [reader]  DEBUG n.s.sshj.connection.ConnectionImpl - Forgetting `session` channel (#0)
.... snip ...
2015-01-20 15:22:36.869 [reader]  INFO  n.s.sshj.transport.TransportImpl - Received SSH_MSG_DISCONNECT (reason=PROTOCOL_ERROR, msg=Received ieof for nonexistent channel 0.)
2015-01-20 15:22:36.873 [reader] 
net.schmizz.sshj.transport.TransportException: Received ieof for nonexistent channel 0.
    at net.schmizz.sshj.transport.TransportImpl.gotDisconnect(TransportImpl.java:534) ~[sshj-0.11.0-SNAPSHOT.jar:na]
    at net.schmizz.sshj.transport.TransportImpl.handle(TransportImpl.java:489) ~[sshj-0.11.0-SNAPSHOT.jar:na]
    at net.schmizz.sshj.transport.Decoder.decode(Decoder.java:107) ~[sshj-0.11.0-SNAPSHOT.jar:na]
    at net.schmizz.sshj.transport.Decoder.received(Decoder.java:175) ~[sshj-0.11.0-SNAPSHOT.jar:na]
    at net.schmizz.sshj.transport.Reader.run(Reader.java:61) ~[sshj-0.11.0-SNAPSHOT.jar:na]

[nicksellen]

I just ran into this issue when needing to send data to stdin of ssh script. I had to commit a deadly sin to work around it:

public static void writeEOF(OutputStream o) throws Exception {
  if (!(o instanceof ChannelOutputStream)) return;
  ChannelOutputStream out = (ChannelOutputStream) o;

  Field f1 = ChannelOutputStream.class.getDeclaredField("trans");
  f1.setAccessible(true);
  Transport trans = (Transport) f1.get(out);

  Field f2 = ChannelOutputStream.class.getDeclaredField("chan");
  f2.setAccessible(true);
  Channel chan = (Channel) f2.get(out);

  trans.write(new SSHPacket(Message.CHANNEL_EOF).putUInt32(chan.getRecipient()));

}

I am writing env via stdin as I ran into #128

Nice library otherwise :)

[asgeirn]

Also got bitten by this. This is how I wrote a workaround, only using public methods:

ssh.getTransport().write(new SSHPacket(Message.CHANNEL_EOF).putUInt32(session.getRecipient()));

Could we make this an option on the channel, so people have the option of controlling whether or not to send EOF?

[vladimirlagunov]

All IntelliJ-based IDEs use SSHJ by default since 2019.2 version, with a similar workaround on the IDE side that sends EOF-packets on close of every exec channel. (Recently the same behaviour has been added to tcpip-forwardings and shell channels, but it's not released yet. Let's have a look if there is still some doubt.)

For year and a half we've received myriad of stacktraces and bug reports, but none of them have been related to this race. Even though I remember I managed to catch the mentioned "Received something for nonexistent channel" error once or twice while I was working on the new SSH backend for 2019.2.

Instead of avoiding sending EOF packets, I'd consider changing reactions on receiving packets from unexpected channels. ConnectionImpl and ChannelInputStream can merely ignore wrong packets. I read the RFC 4254 section 5 and didn't find any obligation regarding handling packets from unexpected channel. And finally, if I'm not mistaken, both OpenSSH client and server ignore such packets.

[aphyr]

I've been trying to use SSHJ for Jepsen as well, and hit this issue too. It looks like I can work around it by following @asgeirn's approach, though I'm not sure if I'm setting myself up for race conditions as a result. It'd be great if there were some kind of official documentation for "how not to leave commands hanging forever"! :-)