Use page title for filename with :download #2753

iordanisg · 2017-06-26T10:27:24Z

This change is

iordanisg · 2017-06-26T11:28:39Z

The-Compiler

also cc @Kingdread because I still think he knows the download code better than I do 😉

The-Compiler · 2017-06-26T13:18:20Z

qutebrowser/browser/commands.py

@@ -1444,7 +1444,8 @@ def download(self, url=None, dest_old=None, *, mhtml_=False, dest=None):
        else:
            qnam = tab.networkaccessmanager()
            download_manager.get(self._current_url(), user_agent=user_agent,
-                                 qnam=qnam, target=target)
+                                 qnam=qnam, target=target,
+                                 title=self._current_title())


I think passing a title here makes get_request needlessly complicated for only this single use case.

Instead, you should be able to construct a downloads.FileDownloadTarget here (similar to what's done above when --dest is given), and pass that to .get() instead. Then there shouldn't be any changes in qtnetworkdownloads.py needed at all.

The-Compiler · 2017-06-26T13:18:20Z

qutebrowser/browser/qtnetworkdownloads.py

@@ -429,7 +429,8 @@ def get_request(self, request, *, target=None, **kwargs):
                             QNetworkRequest.AlwaysNetwork)

        if request.url().scheme().lower() != 'data':
-            suggested_fn = urlutils.filename_from_url(request.url())
+            suggested_fn = (utils.sanitize_filename(title) + ".html" if title


Oh, good job at finding utils.sanitize_filename! I compltely forgot about it 😆

Kingdread · 2017-06-28T10:30:58Z

I think this method is a bit too simplistic, as it always takes the page title, even for non-HTML pages:

[daniel@persephone]  ~ % ls /tmp/qutebrowser-basedir-4mql1yi0/download/
'Vanamo_Logo.png (1200x1200 pixels).html'
[daniel@persephone]  ~ % file /tmp/qutebrowser-basedir-4mql1yi0/download/Vanamo_Logo.png\ \(1200x1200\ pixels\).html
/tmp/qutebrowser-basedir-4mql1yi0/download/Vanamo_Logo.png (1200x1200 pixels).html: PNG image data, 1200 x 1200, 4-bit colormap, non-interlaced

(the old method produced a Vanamo_Logo.png file, which is fine)

I think the question is: How do we determine if we got a sensible filename out of the URL, or when should we use the page title?

The-Compiler · 2017-06-28T18:47:42Z

Hmm, damn. This is indeed something I didn't consider, and a good point...

I have two ideas:

Use the title when there's no path in the URL, and have a whitelist of file extensions where it makes sense to do so too (as a first approximation, .html, .htm, .php, but probably more)
Don't do this at all. 😟

Kingdread · 2017-06-29T18:49:08Z

I'd probably add "if there's no extension", as many modern pages don't even use extensions anymore (for example, GitHub: https://github.com/qutebrowser/qutebrowser/pull/2753).

Alternatively, we could put this behind a flag (:download --title?), but then you would need to remember to use the flag.

The-Compiler · 2017-07-05T11:10:31Z

Any update, @iordanisg?

The-Compiler · 2017-07-05T11:10:46Z

Whoops, didn't mean to close that one, sorry.

iordanisg · 2017-07-05T11:48:21Z

@The-Compiler: your whitelist idea sounds good to me, I hope I'll be able to work something out soon (busy week so far)

@Kingdread: thanks for the feedback!

iordanisg · 2017-07-06T10:24:09Z

About the failing download test: a page title is created on the fly (even if the page doesn't have one) only when running the tests.

The-Compiler

This is getting a bit more complex than I thought originally (sorry!)

I suggest moving it into a def suggested_fn_from_title(url, title): (or so) function in qutebrowser/browser/downloads.py.
Could you please also add an unittest in tests/unit/browser/webkit/test_downloads.py for all those special cases? This can probably benefit from pytest's parametrization, see e.g. test_format_seconds in tests/unit/utils/test_utils.py for a simple example.

The-Compiler · 2017-07-06T11:30:46Z

qutebrowser/browser/commands.py

+            ext_whitelist = [".html", ".htm", ".php", ""]
+            _, ext = os.path.splitext(self._current_url().path())
+            if ext.lower() in ext_whitelist and tab.title():
+                suggested_fn = utils.sanitize_filename(tab.title()) + ext


When QtWebEngine auto-generates a title, this ends up as no title.html.html. I'd suggest checking for if not tab.title().endswith(ext): before adding it.

The-Compiler · 2017-07-06T11:30:46Z

tests/end2end/features/downloads.feature

+        And I wait until the download is finished
+        Then the downloaded file qutebrowser.png should exist
+
+    Scenario: Using :download with no URL and no page title


It looks like QtWebEngine autogenerates a filename while QtWebKit doesn't, which also explains that you don't see it locally. After adding some unittests (see my review comment), I'd suggest just removing this end2end test entirely (but keep the other two).

The-Compiler · 2017-07-06T15:48:41Z

qutebrowser/browser/downloads.py

@@ -182,6 +182,26 @@ def transform_path(path):
    return path


+def suggested_fn_from_title(url, title=None):


url should be called url_path to make it clear that it isn't a full URL.

The-Compiler · 2017-07-06T15:48:41Z

qutebrowser/browser/downloads.py

@@ -182,6 +182,26 @@ def transform_path(path):
    return path


+def suggested_fn_from_title(url, title=None):
+    """Suggest a filename depending on the URL extension and page title.


nitpick: Add an empty line here

The-Compiler · 2017-07-06T15:48:41Z

qutebrowser/browser/downloads.py

+        url: a string with the URL path
+        title: the page title string
+
+    Returns None if the extension is not in the whitelist


nitpick: Put this under a Return: similar like with Args:

The-Compiler · 2017-07-06T15:48:41Z

tests/unit/browser/webkit/test_downloads.py

+     'Installing qutebrowser _ qutebrowser.html'),
+    ('http://qutebrowser.org/INSTALL.html.html',
+     'Installing qutebrowser | qutebrowser',
+     'Installing qutebrowser _ qutebrowser.html'),


I don't understand this test - I think to resemble the behavior with the double extension, this should have a INSTALL.html filename (not .html.html) and a title which ends with .html, no?

The-Compiler · 2017-07-06T15:48:41Z

qutebrowser/browser/downloads.py

+    _, ext = os.path.splitext(url)
+    if ext.lower() in ext_whitelist and title:
+        suggested_fn = utils.sanitize_filename(title)
+        if not suggested_fn.endswith(ext):


Maybe if not suggested_fn.lower().endswith(ext.lower())?

The-Compiler · 2017-07-06T15:48:41Z

tests/unit/browser/webkit/test_downloads.py

+    ('http://qutebrowser.org/page-with-no-title.html',
+     '',
+     None),
+])


Please also add a test with an upper-case extension (to make sure the title is used), and with an upper-case .HTML in the title (see comment above).

The-Compiler · 2017-07-06T20:56:24Z

Looks good to me now - @Kingdread what do you think?

Kingdread · 2017-07-08T09:00:29Z

qutebrowser/browser/downloads.py

+    _, ext = os.path.splitext(url_path)
+    if ext.lower() in ext_whitelist and title:
+        suggested_fn = utils.sanitize_filename(title)
+        if not suggested_fn.lower().endswith(ext.lower()):


From the test cases and the code I assume that the .html is not appended if it's not in the original URL? Is that intended? I'd assume a title like "qutebrowser home page" on "http://qutebrowser.org" would result in a file qutebrowser home page.html.

Also, won't this add .php even for HTML files? Like you get a index.php locally, even if it's just the produced HTML content?

Hmm, good points - maybe it's be best to check if it ends in either .htm or .html and if not, just append .html?

Very good points, indeed. What about when there is an uppercase .HTML or .HTM in the title? Would it make sense to keep it that way in the filename or convert it to lower case?

I think it'd be fine to keep it, but I don't have a strong opinion either way. I'd go for "whatever is easier to implement" for this one.

The-Compiler · 2017-07-08T15:07:34Z

@Kingdread anything else? Otherwise, this looks ready to merge to me.

Kingdread · 2017-07-09T07:24:16Z

No, no more comments 😛

The-Compiler · 2017-07-09T11:01:39Z

And merged! Thanks @Kingdread for the reviews, and @iordanisg for the contribution and your patience! 😄

iordanisg · 2017-07-09T13:44:48Z

Thank you both!

Use page title for filename with :download

a24d7f6

The-Compiler requested changes Jun 26, 2017

View changes

The-Compiler added the PR: work in progress label Jun 26, 2017

iordanisg added 3 commits June 26, 2017 23:21

Add suggested_fn argument to get_request

8a5b48d

Fix PEP-8 issue

5e2be8a

Add file path to download

0a09758

The-Compiler closed this Jul 5, 2017

The-Compiler reopened this Jul 5, 2017

Use page title only for whitelisted extensions

57e4d49

The-Compiler requested changes Jul 6, 2017

View changes

Refactor suggested_fn_from_title, add unit tests

3bfafb5

The-Compiler requested changes Jul 6, 2017

View changes

Improve function docstring, add more tests

82d194c

The-Compiler approved these changes Jul 6, 2017

View changes

The-Compiler added PR: needs final review/merge and removed PR: work in progress labels Jul 6, 2017

Kingdread reviewed Jul 8, 2017

View changes

Adjust suggested_fn_from_title, add tests

c9fd182

The-Compiler merged commit c9fd182 into qutebrowser:master Jul 9, 2017

qutebrowser-bot added the status: stale label Dec 20, 2017

Use page title for filename with :download #2753

Use page title for filename with :download #2753

iordanisg commented Jun 26, 2017 •

edited by The-Compiler

iordanisg commented Jun 26, 2017

The-Compiler left a comment

The-Compiler Jun 26, 2017

The-Compiler Jun 26, 2017

Kingdread commented Jun 28, 2017

The-Compiler commented Jun 28, 2017

Kingdread commented Jun 29, 2017 •

edited

The-Compiler commented Jul 5, 2017

The-Compiler commented Jul 5, 2017

iordanisg commented Jul 5, 2017

iordanisg commented Jul 6, 2017 •

edited

The-Compiler left a comment

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler Jul 6, 2017

The-Compiler commented Jul 6, 2017

Kingdread Jul 8, 2017

Kingdread Jul 8, 2017

The-Compiler Jul 8, 2017

iordanisg Jul 8, 2017

The-Compiler Jul 8, 2017

The-Compiler commented Jul 8, 2017

Kingdread commented Jul 9, 2017

The-Compiler commented Jul 9, 2017

iordanisg commented Jul 9, 2017

		@@ -182,6 +182,26 @@ def transform_path(path):
		return path


		def suggested_fn_from_title(url, title=None):

Use page title for filename with :download #2753

Use page title for filename with :download #2753

Conversation

iordanisg commented Jun 26, 2017 • edited by The-Compiler

iordanisg commented Jun 26, 2017

The-Compiler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kingdread commented Jun 28, 2017

The-Compiler commented Jun 28, 2017

Kingdread commented Jun 29, 2017 • edited

The-Compiler commented Jul 5, 2017

The-Compiler commented Jul 5, 2017

iordanisg commented Jul 5, 2017

iordanisg commented Jul 6, 2017 • edited

The-Compiler left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

The-Compiler commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

The-Compiler commented Jul 8, 2017

Kingdread commented Jul 9, 2017

The-Compiler commented Jul 9, 2017

iordanisg commented Jul 9, 2017

iordanisg commented Jun 26, 2017 •

edited by The-Compiler

Kingdread commented Jun 29, 2017 •

edited

iordanisg commented Jul 6, 2017 •

edited