FTP listing support

[Emdek]

QtWebKit by default can only download files through FTP protocol, it is unable to list its contents.

This could be used as reference, but without using deprecated QFtp module:
http://doc.qt.digia.com/qq/32/qq32-webkit-protocols.html

Preferred is parsing replies directly (at least Unix style, DOS can wait) or make use of some simple library which could be shipped (it is important that license is compatible with GPLv3), but it is not rally needed since it is pretty simple (I can provide example raw outputs if needed).

It should be based on LocalListingNetworkReply class as example, use custom reply only for listings (if possible, auth may require to handle them all).
For icons it should only use method that accepts file name to guess content type (no contents downloading), show directories first and for now should simply use the same HTML template (without creating copy).

[Chocimier]

I take this job.

[Chocimier]

I am not longer working on this, sorry. It is harder, than I think it is.

[beastie1]

Would you consider implementing ftp: support as an module, just like irc:, magnet:, mailto: would have their own?

Furthermore, this module would be loadable on first use, so that you don't need to have it running all the time: once the user opens a URL starting with ftp:// for the first time, the module gets loaded.

This is just a small suggestion as I don't know

1. how important FTP support is to everyone for it to be natively supported (i.e. not as an optional module)
2. how elaborate is the support for the protocol in the Qt libraries; in other words would the code be relatively small that it would make more sense to include it directly inside Otter's main executable instead of an external .so/.dll file

[Emdek]

@beastie1, yes, it's doable.
I guess that core part should be built-in, but GUI and write support should be additional module.
It was also proposed on forums, also to include support for SFTP.

Although this feature should have own ticket, separate from this one.
Probably it should be (if we would go for full fledged tool) something like basic, two pane file manager, with pluggable backends to support other protocols.

[beastie1]

Some times it works and loads a page, but most of the time it freezes with high CPU usage and I have to kill the process.

[Emdek]

@beastie1 , do you have such issues with all or only long listings (more than 100 entries)?

[beastie1]

Everything. Could you try these?

http://www.openbsd.org/ftp.html#ftp
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/mirrors-ftp.html

[beastie1]

Actually listing seem to work randomly. Try opening ftp://ftp.freebsd.org

Also there are random but rare segmentation faults.

[beastie1]

Okay I think I've found the problem: a missing / at the end.

Listing any directory, including the root directory, works fine, but remove the / after the gTLD and it freezes.

[Emdek]

@beastie1, good catch, infinite loop while generating navigation, easy to fix, but hard to find, thanks. ;-)

[beastie1]

Actually anything without the trailing slash freezes. Try this: ftp://ftp5.uk.freebsd.org/pub/FreeBSD/README.TXT

Otter doesn't display the text file, it doesn't propose to open/save the file, it just freezes.

[Emdek]

@beastie1, that one works fine for me, I've tested it also on big files (several megabytes) earlier.
Could you please try to generate some backtrace for that freeze?

[beastie1]

It throws a zillion

Warning: QIODevice::read: device not open ((null):0, (null))

into the logs. The log reached 200+MB in 3 seconds.

How do I generate the backtrace? gdb?

[beastie1]

Debug: QNetworkReplyImpl::_q_startOperation was called more than once ((null):0, (null))
Debug: QNetworkReplyImpl::_q_startOperation was called more than once ((null):0, (null))
Killed
Debug: QNetworkReplyImpl::_q_startOperation was called more than once ((null):0, (null))
Warning: QIODevice::read: device not open ((null):0, (null))
[...]

The last line is repeated thousands of times per second until the process is killed.

What do I have to change to include debug information in the Otter binary?

[Emdek]

@beastie1, if you are using CMake then you need to set build type:
http://stackoverflow.com/questions/7724569/debug-vs-release-in-cmake

These warnings from console won't help much, you can safely ignore them.

[beastie1]

Starting program: /tmp/otter-master/build/otter-browser 
[New LWP 100111]
[New Thread 811406400 (LWP 100111/otter-browser)]
[New Thread 811407800 (LWP 100426/otter-browser)]
[New Thread 857d01c00 (LWP 100434/otter-browser)]

Program received signal SIGINT, Interrupt.
[Switching to Thread 857d01c00 (LWP 100434/otter-browser)]
0x00000008069fe8cc in __error () from /lib/libthr.so.3
#0  0x00000008069fe8cc in __error () from /lib/libthr.so.3
#1  0x00000008069fcdac in _pthread_cond_wait () from /lib/libthr.so.3
#2  0x000000080575d954 in QWaitCondition::wait () from /usr/local/lib/libQt5Core.so.5
#3  0x000000080575d729 in QWaitCondition::wait () from /usr/local/lib/libQt5Core.so.5
#4  0x00000008057592f9 in QThread::isInterruptionRequested () from /usr/local/lib/libQt5Core.so.5
#5  0x000000080575c506 in QThreadStorageData::finish () from /usr/local/lib/libQt5Core.so.5
#6  0x00000008069f34f5 in pthread_create () from /lib/libthr.so.3
#7  0x0000000000000000 in ?? ()
The program is running.  Exit anyway? (y or n)

[Emdek]

@beastie1, uhm, sadly this backtrace is not helpful.
You get this for any file or just one some specific server?
It would be best if I could reproduce it locally.

[beastie1]

I was trying to open this file: ftp://ftp5.uk.freebsd.org/pub/FreeBSD/README.TXT

But any server does the same. Both favicon.ico and index.html in the main server (ftp://ftp.freebsd.org/) also freeze.

And this crashes with a segmentation fault.
ftp://ftp5.uk.freebsd.org/pub/HardenedBSD/current//pub/HardenedBSD/current

[Chocimier]

@beastie1 Does "Warning: QIODevice::read: device not open" bug still exists in current master?

[beastie1]

I always build from master before testing. So yes, my December 1st post above is still 100% valid today. Both the freeze and the segmentation fault.

I don't know if it's related but just by opening ftp://ftp.freebsd.org I get this:

Warning: No file found for ".xml" , even though the file appeared in a directory listing. ((null):0, (null))
Warning: Either it was just removed, or the directory doesn't have executable permission... ((null):0, (null))
Warning: ("/usr/local/share/mime") ((null):0, (null))

After clicking index.html and waiting 1 second I get 240,000 lines of

Warning: QIODevice::read: device not open ((null):0, (null))

before I pkill -9 otter.

[beastie1]

BTW, saving these files works perfectly well.