OC5.0.6 - External Storage - FTP - unable to mount share

[j-ed]

Expected behaviour

If you've created a FTP mount configuration the share should automatically be mounted.

Actual behaviour

The configured FTP share is not being mounted if I select the share BilderHome from the files tab and the message "Cloud not found, i //BilderHome" is shown instead. It might be important to mention that the mount point BilderHomeis displayed as a file and not as a folder.

Steps to reproduce

1. I've installed a local FTP-Server and tested access to it by using a standard ftp client. The server is reachable from the local server as from the local LAN and a login is also possible without any problems.

2.1) At first I've created the following mount.php file to test access to the server:

   <?php
    return array(
      'user' => array(
        'jed' => array(
          '/$user/files/BilderHome'=>array('class'=>'OC_Filestorage_FTP','options'=>array('host'=>'ftp.privat.lan','user'=>'jed','password'=>'test','root'=>'/',' secure'=>false)),
        ),
      ),
    );

2.2) In addition I've also tried to create the same ftp mount using the GUI to access the server. As a result the following mount.json file was created:

{"user":{"jed":{"\/$user\/files\/BilderHome":{"class":"\\OC\\Files\\Storage\\FTP","options":{"host":"ftp.privat.lan","user":"jed","password":"test","root":"","secure":"false"}}}}

3. If I select the displayed share BilderHome, which is not displayed as folder but as a file, the message "Cloud not found, i //BilderHome" is displayed.

4. If I select the Administrator menu, the following PHP error is shown on the screen for a short moment:

    `Warning: stat():  stat failed for ftp://jed:test@ftp.privat.lan/ in .../owncloud/apps/files_external/lib/streamwrapper.php on line 81`

Environment

Server Configuration

OS: Linux 2.6.24-24
Web server: Apache/ 2.2.22
Database: MySQL
PHP version: 5.3.13
Pure-FTPd v1.0.36
ownCloud version: 5.0.6

Client Configuration

Browser: Firefox 21.0
Operating system: Windows 7

[j-ed]

Point 4) As far as I can see the stat error is caused because the command is used on an empty $path at the given address. After changing the stat function in files_external/lib/streamwrapper.phpas follows the error is suppressed:

public function stat($path) {
    if(!empty($path) && $path != "/") {
        return stat($this->constructUrl($path));
    } else {
        return true;
    }
}

[j-ed]

I did further investigations and found out that the created FTP link in the files window differs from the links created for LOCAL or SMB shares. This seems to be the reason why the "Cloud not found, i //BilderHome" error message is shown after the file_exists check in apps/files/download.php failed.

LOCAL mount: https://test.privat.lan/owncloud/index.php/apps/files?dir=/Bilder-Ablage
SMB mount:   https://test.privat.lan/owncloud/index.php/apps/files?dir=//MP3-Archiv
Google mount: http://eis1-dev.privatnet.lan/owncloud5/index.php/apps/files?dir=//GoogleDrive
FTP mount:   https://test.privat.lan/owncloud/index.php/apps/files/download//BilderHome

Unfortunately I don't have a clue how and where these URLs are created :-(

@icewind1991 It seems that you know this part of OC very well. Could you give me a hint
where I have to look for the relevant piece of code?

[j-ed]

The problem still exists in OC5.0.10

[DeepDiver1975]

@icewind1991 THX

[j-ed]

The problem still exists in OC5.0.12

[bgarlock]

I am seeing this in 5.0.12 as well. Would love to get this fixed, as this is the last piece I need for a production OC instance.

[karlitschek]

@PVince81 What do you think?

[PVince81]

Just tried in OC 5.0.13 and I'm able to mount an FTP volume.
Creating and navigating into folders works.
BUT creating/uploading files doesn't seem to work.

Works in OC 6 beta 4 except for deleting folders.

[j-ed]

@PVince81 I've just tried to access the FTP mount in OC6 beta4 on my server and the displayed problem is still the same. I'm able to access the ftp share it when I use the ftp command from the command line. What kind of information do you need to identify the root cause of the problem?

[j-ed]

@PVince81 I've just tried to access the FTP mount in OC 5.013 on my server and the displayed problem is still the same.

[PVince81]

I was using vsftpd 2.3.5 on Ubuntu 12.04.3 LTS.
Also, I've configured the FTP server to allow local users to login and allowed write access.

I'll have a try with Pure-FTPd

[PVince81]

There seems to be a bug when specifying the root folder. Make sure to have a trailing slash at the end of it. (but I don't think it's related)

[j-ed]

@PVince81 I've already tested this but unfortunately without success.

[PVince81]

I installed pureftpd 1.0.36 on Ubuntu 12.10 in a VM using the following: http://wiki.ggis.biz/index.php/Pure-FTPd_on_Ubuntu

It works fine with OC6 beta 5 (master) and OC 5.0.13.
Maybe you can compare your config with that page ?

The only issue is if I have big media files on the FTP it takes a long time to index, but that's a known issue that is fixed in OC 6 (mime type detection)

[PVince81]

Btw, on that info page I stopped before the "TLS/SSL support".

[karlitschek]

Should work. Please retest with ownCloud 6

[j-ed]

Ok, I tested the following things:
1.) Upgrade to OC6 RC1 as requested by @karlitschek
2.) Reconfigured Pure-FTPd as requested by @PVince81 and created a FTP account.
3.) Restarted the Apache2 web server and cleared my browser cache.
4.) Tried to connect to the server by using the ftp command from the command line (ftp 127.0.0.1)
Result: FTP Login successful.
5.) Deleted mount.json and added the FTP mount again.
Result: FTP Login failed.
As soon as I re-enter the admin settings dialog the following warning is shown:

    Warning: stat(): stat failed for ftp://ftpuser:ftptest@127.0.0.1/ in /var/www/htdocs/owncloud/apps/files_external/lib/streamwrapper.php on line 88 

6.) Created the following php script and ran it.
Result: FTP Login successful

   <?php

   $ftp_server = "127.0.0.1";
   $ftp_user = "ftpuser";
   $ftp_pass = "ftptest";

   echo "- start -----------------------<br>\n";
   // set up a connection or die
   $conn_id = ftp_connect($ftp_server) or die("Couldn't connect to $ftp_server");

   // login
   $login_result = ftp_login ($conn_id, $ftp_user, $ftp_pass);

   // check open
   if ((!$conn_id) || (!$login_result)) {
       echo "FTP connect failed!<br>\n"; die;
   } else {
       echo "connected to FTP server.<br>\n";
       echo "- file list -------------------<br>\n";

       // get contents of the current directory
       $contents = ftp_nlist($conn_id, ".");

       // output $contents
       foreach($contents as $item) {
           echo "+ $item<br>\n";
       }

   // close this connection
   ftp_close($conn_id);
   }

   echo "- end -------------------------<br>\n";
   ?>

Due to the fact that a basic FTP connect and authentication was possible using the given script, the FTP server access should be ok.

[j-ed]

To investigate further I added a debug_print_backtrace(); right after the mention stat function. Maybe that helps to understand what is going on here.

Warning: stat(): stat failed for ftp://ftpuser:ftptest@127.0.0.1/ in /var/www/htdocs/owncloud/apps/files_external/lib/streamwrapper.php on line 88
#0 OC\Files\Storage\StreamWrapper->stat() called at [/var/www/htdocs/owncloud/lib/private/files/storage/common.php:358]
#1 OC\Files\Storage\Common->test() called at [/var/www/htdocs/owncloud/apps/files_external/lib/config.php:245]
#2 OC_Mount_Config::getBackendStatus(\OC\Files\Storage\FTP, Array ([host] => 127.0.0.1,[user] => ftpuser,[password] => ftptest,[root] => /,[secure] => false)) called at [/var/www/htdocs/owncloud/apps/files_external/lib/config.php:201]
#3 OC_Mount_Config::getSystemMountPoints() called at [/var/www/htdocs/owncloud/apps/files_external/settings.php:31]
#4 include(/var/www/htdocs/owncloud/apps/files_external/settings.php) called at [/var/www/htdocs/owncloud/lib/private/app.php:621]
#5 OC_App::getForms(admin) called at [/var/www/htdocs/owncloud/settings/admin.php:17]
#6 require_once(/var/www/htdocs/owncloud/settings/admin.php) called at [/var/www/htdocs/owncloud/lib/private/route.php(121) : runtime-created function:1]
#7 __lambda_func(Array ([_route] => settings_admin))
#8 call_user_func(lambda_712, Array ([_route] => settings_admin)) called at [/var/www/htdocs/owncloud/lib/private/router.php:127]
#9 OC_Router->match(/settings/admin) called at [/var/www/htdocs/owncloud/lib/base.php:692]
#10 OC::handleRequest() called at [/var/www/htdocs/owncloud/index.php:30] 

BTW: I wonder if it is correct that only the ftp.php and irods.php files don't contain a dedicated stats() function. But I'm not a professional programmer.

[PVince81]

@j-ed thanks for the info.

For FTP ownCloud uses stat($url). Could you try this in your test script ?
stat("ftp://ftpuser:ftptest@127.0.0.1/")
For some unknown reason this seems to fail on your side.

ownCloud also reads files using fopen($url), could you try that as well ? (in case it's just stat() that is broken)

ftp.php relies on the base class' stat implementation (streamwrapper.php) and doesn't need its own specific one.

[j-ed]

@PVince81 I did further investigations on this issue and found several comments on the Internet describing problems with the stat() or filemtime()function when they are used on FTP links. Unfortunately I wasn't able to find a single root cause for that problem.

1. As requested I tried the stat($url) function on the FTP link and it definitely failed. (Although it is working on local files)
2. I also tried to use the fopen($url)function and it worked without any problems.
3. Finally I tried to use the ftp_raw($conn_id, "SIZE $fname")[0] and ftp_raw($conn_id, "MDTM $fname")[0] functions to request FTP file information and both worked too.

[PVince81]

If PHP fails on stat() there isn't much we can do about it except for rewriting it...
Could it be a bug in the PHP FTP module itself ?

Your PHP version (5.3.13) should be recent enough.
If possible you could try with a more recent PHP version ?

[j-ed]

@PVince81 I updated PHP to v5.4.21 but the problem still persists. In addition I checked the PHP changelog for FTP related issues but I wasn't able to find any issue of relevance.

Wouldn't it be sufficient to add a dedicated stat() function to ftp.php?

[PVince81]

@j-ed I'm not sure how to implement such function. It would still need to go through FTP to fetch the relevant information. One idea would be to rewrite the ftp storage to use the approach you used in your test scripts above.

The only thing I can think of now is trying to Google the issue (I know you did already) to find out under what conditions stat() doesn't work for FTP in PHP. 😦

[PVince81]

You don't happen to have something like apparmor or selinux running ?

[j-ed]

Sorry, no apparmor nor selinux running.

[bantu]

@j-ed In your test script consider calling ftp_pasv() as that is what the FTP stream wrapper expects to work as per http://www.php.net/manual/en/wrappers.ftp.php

Also, you may just want to give an external host a try. E.g.
Hostname ftp.pureftpd.org, Username: anonymous, Password: whatever, Root: /pub/pure-ftpd/releases/obsolete/

[bantu]

Additionally, you may want to look into something like wireshark or tcpdump to see what is going on.

[j-ed]

@bantu I ran various FTP tests and also included ftp_pasv() but without any change on the results.
I also tested stat() on ftp.pureftpd.org but with the same result "Warning: stat(): stat failed for ...".
I will try to investigate farther using wireshark etc

[j-ed]

@bantu First I executed the stat() command with the URL "ftp://anonymous:whatever@91.121.49.42/pub/pure-ftpd/releases/obsolete/pure-ftpd-1.0.17.tar.bz2" and traced the network using.tcpdump. Surprisingly no traffic to the destination host was logged at all. I repeated that test with other FTP servers but the result was the same.

Then I accessed the URL using ftp_connect()... ftp_close() functions and the network communication was logged properly.

From my point of view PHP seems to have a problem interpreting the given URL properly. Unfortunately I don't have a clue how to debug PHP to find the root cause of the problem.

[PVince81]

Can you check your php.ini and make sure that "allow_url_fopen" is allowed ? Could be related...

[j-ed]

@PVince81 That was the first thing I checked, although fopen and all ftp_... commands are working properly:

[PVince81]

If even PHP 5.4 isn't able to stat FTP URLs, I'm out of ideas what could be wrong here 😦

[PVince81]

Looks like I've got it as well now... I have two VMs, one with pureftpd and one with vsftpd.
The pureftpd one connects fine but the vsftpd one fails on stat().

[vgezer]

Any update on this? Is this still reproducible?

[j-ed]

I will set-up a new test environment with the latest OC 6.0.4 and check if this problem is reproducible.

[anroots]

This is causing me headaches also. I'm able to connect to my FTP server using a small PHP test script with ftp_login(), but stat() fails, both in owncloud and as a separate test script. Am I correct that

stat('ftp://user:pass@sqroot.eu/data01/sqroot/domeenid/www.sqroot.eu/owncloud/');

should succeed in theory?

[PVince81]

In theory it should succeed, at least it does on most installs.
I did a long research already but wasn't able to find out why it would fail on some setups 😦

[RobinMcCorkell]

Tested FTP shares on oC 7.0.3 with PHP 5.6.3, against vsftpd 3.0.2. No stat() errors in the log, and the mounted share works just fine. @j-ed is this still a problem for you?

[karlitschek]

Can´t be produced. I assume this is fixed. If not please reopen

[j-ed]

I've installed Pure-FTPd on my server and ran some tests. Filezilla works like a charm but as soon as I'm trying to access the server via a FTP share from ownCloud v7.0.4 it still doesn't work. I'm not able to connect to this server using ftp: or ftps:.

Environment

Server Configuration

OS: Linux 3.2.54
Web server: Apache2 v2.4.10
Database: MySQL v5.1.73
PHP version: v5.4.35
Pure-FTPd v1.0.36
ownCloud version: v7.0.4

Client Configuration

Browser: Firefox v34.0
Operating system: Windows 7

[PVince81]

I remember that you were using a custom-built Linux distro. Did you check whether PHP was setup/installed properly there (maybe compare the config/libs with other distros) ?

[j-ed]

@PVince81 As a first step I verified all setting etc. but couldn't find a reason for that problem. I also did some research on the Internet concerning the stat failed for error and found several occurrences where users had reported similar problems with some of the standard functions accessing ftp files.

[PVince81]

@icewind1991 has been working on a new implementation here: #14551

[PVince81]

@j-ed still having this problem with newer PHP versions ?

[j-ed]

@PVince81 I've justed tested it in the following environment, but the problem still persists.

Server Configuration

OS: Linux 3.2.67
Web server: Apache/ 2.4.12
Database version: MySQL 5.1.73
PHP version: 5.4.42
Pure-FTPd v1.0.36
ownCloud version: 8.0.6

Client Configuration

Browser: Firefox 40.0.2
Operating system: Windows 7

[PVince81]

@j-ed can you try with #23101 ?

[hodyroff]

No information. Please comment again if the issue continues on new ownCloud versions.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.