New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

module-ssh2: streaming from SFTP server impossible without user re-implementing SftpPoller methods #1557

Closed
pavelkveton opened this Issue Jan 10, 2017 · 1 comment

Comments

Projects
None yet
2 participants
@pavelkveton
Contributor

pavelkveton commented Jan 10, 2017

SftpPoller.qm:

the SftpPoller works by running its private run() method, which regularly calls (public) runOnce().

The documentation to runOnce() is not available, so I guess it should normally not be overloaded by child classes. The documentation in the code says:

#! runs a single poll (useful for checking for errors inline before starting a background thread)

In runOnce(), we check for files and for each file we

  • get file's content (using getFile() or getTextFile()) and store it in memory (!!!) in a hash member
  • call abstract singleFileEvent() with the hash generated earlier (carrying the entire file content)

The functions getFile() and/or getTextFile() do not use any kind of streaming - they just try to get all the file in as large blocks as possible.

The function singleFileEvent() is abstract and intended to be overloaded by the child classes - but even if this function uses streaming (i.e. processing with max-read-block-size) to process the files, the entire idea of streaming is already spoiled due to previous getFile() call.

The situation can be worked-around by redefining of runOnce() in the child class, but this function is entirely undocumented (and IMHO not intended to be redefined).

@pavelkveton pavelkveton added the bug label Jan 10, 2017

@pavelkveton pavelkveton added this to the 0.8.12.6 milestone Jan 10, 2017

@pavelkveton

This comment has been minimized.

Contributor

pavelkveton commented Feb 1, 2017

the problem is that we cannot fix this issue without breaking backward compatibility. We will probably have to fix it by introducing a flag into SftpPoller options...

davidnich added a commit to qorelanguage/module-ssh2 that referenced this issue Feb 2, 2017

@omusil24 omusil24 added the fixed label Feb 2, 2017

@omusil24 omusil24 closed this Feb 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment