Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix download with pattern issue in rooted filesystem #88

Closed

Conversation

The-Yoda
Copy link
Contributor

@The-Yoda The-Yoda commented Feb 10, 2019

Right now, download with pattern doesn't work when the underlying
filesystem is virtual filesystem. Also, the tests were masking the
behaviour by providing working location as rooted filesystem path.

This patch fixes the issue, by providing absolute path to retrieve files
matching the pattern.

cc @lgoldstein @thefourtheye @sramki

public void testScpVirtualOnDirWithPattern() throws Exception {
Path remoteDir = Files.createTempDirectory("remote");
sshd.setFileSystemFactory(new VirtualFileSystemFactory(remoteDir));

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use getTempTargetRelativeFile (method inherited from JUnitTestSupport) - it makes sure that the file(s) reside under target folder which is cleaned when mvn clean is invoked. Your code leaves "orphan" directories and files in the user's /tmp folder.

Copy link
Contributor Author

@The-Yoda The-Yoda Feb 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm cleaning that at the end of the test. But this is more cleaner.

@Test
public void testScpVirtualOnDirWithPattern() throws Exception {
Path remoteDir = Files.createTempDirectory("remote");
sshd.setFileSystemFactory(new VirtualFileSystemFactory(remoteDir));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that sshd is a shared instance - therefore you need to restore it to its original state:

@Test
public void testScpVirtualOnDirWithPattern() throws Exception {
    try {
      ...
       sshd.setFileSystemFactory(...test factory...);
      ...
    } finally {
        sshd.setFileSystemFactory(fileSystemFactory);    // the one created in the test constructor
    }
}

Copy link
Contributor Author

@The-Yoda The-Yoda Feb 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is being taken care in Before block which gets executed before each test.

   @Before
    public void setUp() throws Exception {
        sshd.setFileSystemFactory(fileSystemFactory);
    }

Iterable<String> included = opener.getMatchingFilesToSend(session, basedir, pattern);
Path basePath = resolveLocalPath(basedir);
Iterable<String> included = opener.getMatchingFilesToSend(session, basePath.toFile().getAbsolutePath(), pattern);

for (String path : included) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use toFile - if you want the absolute path then use basePath.toAbsolutePath().toString().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

basePath.toAbsolutePath().toString() won't give absolute path of the file when the underlying filesystem is virtual filesystem. I couldn't find any other cleaner way other than toFile

@lgoldstein
Copy link
Contributor

I assume you have run the unit test before the patch and saw that if fails....

@lgoldstein
Copy link
Contributor

BWT, if you rename the issue [SSHD-893] SCP download with pattern does not work download with virtual filesystem I believe the ASF Github BOT will automatically link it to the issue...

@lgoldstein
Copy link
Contributor

I have tried your code with the fixes and the unit test fails:

2019-02-11 14:11:14,458 | WARN  | 4]-nio2-thread-3 | AcceptAllServerKeyVerifier       | org.apache.sshd.client.keyverifier.StaticServerKeyVerifier         59 | Server at /127.0.0.1:45293 presented unverified RSA key: SHA256:w1ctRmSyLM2y2ypPpDdKQxy9WXz35Ug34iHdfIv/nJI
2019-02-11 14:11:14,580 | INFO  | 0)-nio2-thread-1 | ServerUserAuthService            | org.apache.sshd.server.session.ServerUserAuthService              322 | Session testScpVirtualOnDirWithPattern@/127.0.0.1:50362 authenticated
2019-02-11 14:11:15,440 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: opt unknown file type
2019-02-11 14:11:15,479 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: sbin unknown file type
2019-02-11 14:11:15,480 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: mnt unknown file type
2019-02-11 14:11:15,481 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: lib unknown file type
2019-02-11 14:11:15,482 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: lost+found unknown file type
2019-02-11 14:11:15,485 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: dev unknown file type
2019-02-11 14:11:15,485 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: srv unknown file type
2019-02-11 14:11:15,486 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: dev-share unknown file type
2019-02-11 14:11:15,486 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: lib64 unknown file type
2019-02-11 14:11:15,487 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: .esmtp_queue unknown file type
2019-02-11 14:11:15,487 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: tmp unknown file type
2019-02-11 14:11:15,488 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: sys unknown file type
2019-02-11 14:11:15,490 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: boot unknown file type
2019-02-11 14:11:15,490 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: run unknown file type
2019-02-11 14:11:15,492 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: .rnd unknown file type
2019-02-11 14:11:15,493 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: var unknown file type
2019-02-11 14:11:15,494 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: usr unknown file type
2019-02-11 14:11:15,494 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: etc unknown file type
2019-02-11 14:11:15,495 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: proc unknown file type
2019-02-11 14:11:15,496 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: root unknown file type
2019-02-11 14:11:15,497 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: bin unknown file type
2019-02-11 14:11:15,499 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: media unknown file type
2019-02-11 14:11:15,500 | WARN  | main             | ScpHelper                        | org.apache.sshd.common.scp.ScpHelper                              832 | readAck(ScpHelper[ClientSessionImpl[testScpVirtualOnDirWithPattern@/127.0.0.1:45293]])[EOF=true] - Received warning: home unknown file type

Right now, download with pattern doesn't work when the underlying
filesystem is virtual filesystem. Also, the tests were masking the
behaviour by providing working location as rooted filesystem path.

This patch fixes the issue, by providing absolute path to retrieve files
matching the pattern.
@The-Yoda The-Yoda force-pushed the fix-multifile-pattern-download-issue branch from 3dd2f78 to f4883f2 Compare February 11, 2019 12:19
@The-Yoda
Copy link
Contributor Author

Anything specific I need to do to run tests?

It passes for me.

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.sshd.client.scp.ScpTest
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.458 s - in org.apache.sshd.client.scp.ScpTest
[INFO] Running org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.085 s - in org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Running org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.638 s - in org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 23, Failures: 0, Errors: 0, Skipped: 1
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (mina) @ sshd-scp ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.sshd.client.scp.ScpTest
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.689 s - in org.apache.sshd.client.scp.ScpTest
[INFO] Running org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.233 s - in org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Running org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.74 s - in org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 23, Failures: 0, Errors: 0, Skipped: 1
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.22.1:test (netty) @ sshd-scp ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.87 s - in org.apache.sshd.client.scp.SimpleScpClientTest
[INFO] Running org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.412 s - in org.apache.sshd.server.scp.ScpCommandFactoryTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 40.408 s
[INFO] Finished at: 2019-02-11T17:52:35+05:30
[INFO] Final Memory: 57M/588M
[INFO] ------------------------------------------------------------------------

@lgoldstein
Copy link
Contributor

No - but have you tried using the relative-temp-file path ?

@lgoldstein
Copy link
Contributor

I believe I found a better way - see https://github.com/lgoldstein/mina-sshd/tree/SSHD-893 (based on your code). Can you review it and let me know if this is what you meant + run some tests on it ?

@lgoldstein
Copy link
Contributor

See also #89

@The-Yoda
Copy link
Contributor Author

Sure. Let me check.

@The-Yoda
Copy link
Contributor Author

@lgoldstein #89 covers the fix. Shall I close this PR?

@lgoldstein
Copy link
Contributor

Yes, please close it and we'll discuss #89

@The-Yoda The-Yoda closed this Feb 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants