Implement proper streaming for passive commands and add tests #31

merged 21 commits into from Nov 20, 2012


2 participants

sergi commented Nov 20, 2012

This fix gives proper Nodejs streaming capabilities to FTP. Until now, jsftp relied on buffering, although it had streaming methods ready to be used (getPutSocket and getGetSocket). However, upon some testing with vfs-ftp I found out that piping wouldn't work, and it would never do because of the way that PASV commands were handled at the moment. That is, sequentially.

Passive command handling methods have been now split into two, getPasvSocket and orderedPasv. The latter uses the former, and now the user of the API can choose what API to use (the streaming API is less straightforward). In this way, getPutSocket and getGetSocket can directly use the methods that allow them to do multiple requests and deal with the returned sockets (streams). The implementation is cleaner now.

In case you were wondering why not get rid of the sequential API, it is because the sequential API is useful for when the user wants to ensure that an FTP command will always be executed after another, but still write it in procedural style (not callback-style). That was needed for the way jsDAV calles different fs functions, which necessarily doesn't compose them and behave independently. Check out the test "test multiple concurrent pasvs" as an example.

A bit more (and a bit duplicated) information
This is a big rehaul of how streaming passive requests are handled.


  • Before, streams were not properly resumed and paused. Resulting on timeouts in piping streams, for example.

  • Split passive function into two. getPasvSocket and orderedPasv. The former retrieves the passive socket and the latter is for PASV calls that have to be synchronous, like in

    ftp.put("file1", callback1);
    ftp.put("file2", callback2);
  • VFS will be using only streaming functions, of course.

  • The PASV call now goes through the normal flow that includes initialisation, auth, etc. Before a PASV call on connecting to the Ftp would fail because auth didn't happen first.

  • Other small but important changes.

The diff should be pretty self-explanatory and I of course added new unit tests for the changes.

fjakobs and others added some commits Mar 7, 2012

don't use submodules
fix unit tests
Merge pull request #2 from ajaxorg/fix_continuation
Changed code to continuation-style
Merge pull request #3 from ajaxorg/fix_onconnect
Fix execution of the custom `onConnect` handler
Fix streaming commands
Now it pauses and resumes properly so no data is possibly lost, and pasv commands also use `_enqueueCmd` instead of `enqueue` so that they go through the traditional auth flow.
Further improvements:
 - Created a method `_getStreamableSocket` that extends the pause/resume behaviour of a socket to workaround Nodejs 0.6 bugs
 - Tests now use `concatStream` form Ftp, not their own. That uncovered a potential bug.
Merge pull request #4 from ajaxorg/fix_streaming
Implement proper streaming capabilities for passive commands

sergi pushed a commit that referenced this pull request Nov 20, 2012

Merge pull request #31 from ajaxorg/master
Implement proper streaming for passive commands and add tests

@sergi sergi merged commit d13451b into sergi:master Nov 20, 2012

1 check passed

default The Travis build passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment