Use page title for filename with :download #2753

iordanisg · Jun 26, 2017

This change is

iordanisg · Jun 26, 2017

Per #2652

The-Compiler · Jun 26, 2017

The-Compiler requested changes Jun 26, 2017

View changes

also cc @Kingdread because I still think he knows the download code better than I do 😉

The-Compiler · Jun 26, 2017

qutebrowser/browser/commands.py

@@ -1444,7 +1444,8 @@ def download(self, url=None, dest_old=None, *, mhtml_=False, dest=None):
        else:
            qnam = tab.networkaccessmanager()
            download_manager.get(self._current_url(), user_agent=user_agent,
-                                 qnam=qnam, target=target)
+                                 qnam=qnam, target=target,
+                                 title=self._current_title())


I think passing a title here makes get_request needlessly complicated for only this single use case.

Instead, you should be able to construct a downloads.FileDownloadTarget here (similar to what's done above when --dest is given), and pass that to .get() instead. Then there shouldn't be any changes in qtnetworkdownloads.py needed at all.

The-Compiler · Jun 26, 2017

qutebrowser/browser/qtnetworkdownloads.py

@@ -429,7 +429,8 @@ def get_request(self, request, *, target=None, **kwargs):
                             QNetworkRequest.AlwaysNetwork)

        if request.url().scheme().lower() != 'data':
-            suggested_fn = urlutils.filename_from_url(request.url())
+            suggested_fn = (utils.sanitize_filename(title) + ".html" if title


Oh, good job at finding utils.sanitize_filename! I compltely forgot about it 😆

Kingdread · Jun 28, 2017

I think this method is a bit too simplistic, as it always takes the page title, even for non-HTML pages:

[daniel@persephone]  ~ % ls /tmp/qutebrowser-basedir-4mql1yi0/download/
'Vanamo_Logo.png (1200x1200 pixels).html'
[daniel@persephone]  ~ % file /tmp/qutebrowser-basedir-4mql1yi0/download/Vanamo_Logo.png\ \(1200x1200\ pixels\).html
/tmp/qutebrowser-basedir-4mql1yi0/download/Vanamo_Logo.png (1200x1200 pixels).html: PNG image data, 1200 x 1200, 4-bit colormap, non-interlaced

(the old method produced a Vanamo_Logo.png file, which is fine)

I think the question is: How do we determine if we got a sensible filename out of the URL, or when should we use the page title?

The-Compiler · Jun 28, 2017

Hmm, damn. This is indeed something I didn't consider, and a good point...

I have two ideas:

Use the title when there's no path in the URL, and have a whitelist of file extensions where it makes sense to do so too (as a first approximation, .html, .htm, .php, but probably more)
Don't do this at all. 😟

Kingdread · Jun 29, 2017

I'd probably add "if there's no extension", as many modern pages don't even use extensions anymore (for example, GitHub: https://github.com/qutebrowser/qutebrowser/pull/2753).

Alternatively, we could put this behind a flag (:download --title?), but then you would need to remember to use the flag.

The-Compiler · Jul 5, 2017

Any update, @iordanisg?

The-Compiler · Jul 5, 2017

Whoops, didn't mean to close that one, sorry.

iordanisg · Jul 5, 2017

@The-Compiler: your whitelist idea sounds good to me, I hope I'll be able to work something out soon (busy week so far)

@Kingdread: thanks for the feedback!

iordanisg · Jul 6, 2017

About the failing download test: a page title is created on the fly (even if the page doesn't have one) only when running the tests.

The-Compiler · Jul 6, 2017

The-Compiler requested changes Jul 6, 2017

View changes

This is getting a bit more complex than I thought originally (sorry!)

I suggest moving it into a def suggested_fn_from_title(url, title): (or so) function in qutebrowser/browser/downloads.py.
Could you please also add an unittest in tests/unit/browser/webkit/test_downloads.py for all those special cases? This can probably benefit from pytest's parametrization, see e.g. test_format_seconds in tests/unit/utils/test_utils.py for a simple example.

The-Compiler · Jul 6, 2017

qutebrowser/browser/commands.py

+            ext_whitelist = [".html", ".htm", ".php", ""]
+            _, ext = os.path.splitext(self._current_url().path())
+            if ext.lower() in ext_whitelist and tab.title():
+                suggested_fn = utils.sanitize_filename(tab.title()) + ext


When QtWebEngine auto-generates a title, this ends up as no title.html.html. I'd suggest checking for if not tab.title().endswith(ext): before adding it.

The-Compiler · Jul 6, 2017

tests/end2end/features/downloads.feature

+        And I wait until the download is finished
+        Then the downloaded file qutebrowser.png should exist
+
+    Scenario: Using :download with no URL and no page title


It looks like QtWebEngine autogenerates a filename while QtWebKit doesn't, which also explains that you don't see it locally. After adding some unittests (see my review comment), I'd suggest just removing this end2end test entirely (but keep the other two).

The-Compiler · Jul 6, 2017

The-Compiler requested changes Jul 6, 2017

View changes

The-Compiler · Jul 6, 2017

qutebrowser/browser/downloads.py

@@ -182,6 +182,26 @@ def transform_path(path):
    return path


+def suggested_fn_from_title(url, title=None):


url should be called url_path to make it clear that it isn't a full URL.

The-Compiler · Jul 6, 2017

qutebrowser/browser/downloads.py

@@ -182,6 +182,26 @@ def transform_path(path):
    return path


+def suggested_fn_from_title(url, title=None):
+    """Suggest a filename depending on the URL extension and page title.


nitpick: Add an empty line here

The-Compiler · Jul 6, 2017

qutebrowser/browser/downloads.py

+        url: a string with the URL path
+        title: the page title string
+
+    Returns None if the extension is not in the whitelist


nitpick: Put this under a Return: similar like with Args:

The-Compiler · Jul 6, 2017

tests/unit/browser/webkit/test_downloads.py

+     'Installing qutebrowser _ qutebrowser.html'),
+    ('http://qutebrowser.org/INSTALL.html.html',
+     'Installing qutebrowser | qutebrowser',
+     'Installing qutebrowser _ qutebrowser.html'),


I don't understand this test - I think to resemble the behavior with the double extension, this should have a INSTALL.html filename (not .html.html) and a title which ends with .html, no?

The-Compiler · Jul 6, 2017

qutebrowser/browser/downloads.py

+    _, ext = os.path.splitext(url)
+    if ext.lower() in ext_whitelist and title:
+        suggested_fn = utils.sanitize_filename(title)
+        if not suggested_fn.endswith(ext):


Maybe if not suggested_fn.lower().endswith(ext.lower())?

The-Compiler · Jul 6, 2017

tests/unit/browser/webkit/test_downloads.py

+    ('http://qutebrowser.org/page-with-no-title.html',
+     '',
+     None),
+])


Please also add a test with an upper-case extension (to make sure the title is used), and with an upper-case .HTML in the title (see comment above).

The-Compiler · Jul 6, 2017

Looks good to me now - @Kingdread what do you think?

The-Compiler · Jul 6, 2017

The-Compiler approved these changes Jul 6, 2017

View changes

Kingdread · Jul 8, 2017

Kingdread reviewed Jul 8, 2017

View changes

Kingdread · Jul 8, 2017

qutebrowser/browser/downloads.py

+    _, ext = os.path.splitext(url_path)
+    if ext.lower() in ext_whitelist and title:
+        suggested_fn = utils.sanitize_filename(title)
+        if not suggested_fn.lower().endswith(ext.lower()):


From the test cases and the code I assume that the .html is not appended if it's not in the original URL? Is that intended? I'd assume a title like "qutebrowser home page" on "http://qutebrowser.org" would result in a file qutebrowser home page.html.

Also, won't this add .php even for HTML files? Like you get a index.php locally, even if it's just the produced HTML content?

Hmm, good points - maybe it's be best to check if it ends in either .htm or .html and if not, just append .html?

Very good points, indeed. What about when there is an uppercase .HTML or .HTM in the title? Would it make sense to keep it that way in the filename or convert it to lower case?

I think it'd be fine to keep it, but I don't have a strong opinion either way. I'd go for "whatever is easier to implement" for this one.

The-Compiler · Jul 8, 2017

@Kingdread anything else? Otherwise, this looks ready to merge to me.

Kingdread · Jul 9, 2017

No, no more comments 😛

The-Compiler · Jul 9, 2017

And merged! Thanks @Kingdread for the reviews, and @iordanisg for the contribution and your patience! 😄

iordanisg · Jul 9, 2017

Thank you both!

Use page title for filename with :download

Loading status checks…

a24d7f6

The-Compiler added the PR: work in progress label Jun 26, 2017

iordanisg added some commits Jun 26, 2017

Add suggested_fn argument to get_request

Loading status checks…

8a5b48d

Fix PEP-8 issue

Loading status checks…

5e2be8a

Add file path to download

Loading status checks…

0a09758

The-Compiler closed this Jul 5, 2017

The-Compiler reopened this Jul 5, 2017

Use page title only for whitelisted extensions

Loading status checks…

57e4d49

Refactor suggested_fn_from_title, add unit tests

Loading status checks…

3bfafb5

Improve function docstring, add more tests

Loading status checks…

82d194c

The-Compiler added PR: needs final review/merge and removed PR: work in progress labels Jul 6, 2017

Adjust suggested_fn_from_title, add tests

c9fd182

The-Compiler merged commit c9fd182 into qutebrowser:master Jul 9, 2017

qutebrowser-bot added the status: stale label Dec 20, 2017

qutebrowser/qutebrowser

Join GitHub today

Use page title for filename with :download #2753

Conversation

iordanisg commented Jun 26, 2017 • edited by The-Compiler

This comment has been minimized.

iordanisg commented Jun 26, 2017

The-Compiler requested changes Jun 26, 2017 View changes

This comment has been minimized.

This comment has been minimized.

The-Compiler added the PR: work in progress label Jun 26, 2017

iordanisg added some commits Jun 26, 2017

This comment has been minimized.

Kingdread commented Jun 28, 2017

This comment has been minimized.

The-Compiler commented Jun 28, 2017

This comment has been minimized.

Kingdread commented Jun 29, 2017 • edited

This comment has been minimized.

The-Compiler commented Jul 5, 2017

The-Compiler closed this Jul 5, 2017

This comment has been minimized.

The-Compiler commented Jul 5, 2017

The-Compiler reopened this Jul 5, 2017

This comment has been minimized.

iordanisg commented Jul 5, 2017

This comment has been minimized.

iordanisg commented Jul 6, 2017 • edited

The-Compiler requested changes Jul 6, 2017 View changes

This comment has been minimized.

This comment has been minimized.

The-Compiler requested changes Jul 6, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

The-Compiler commented Jul 6, 2017

The-Compiler approved these changes Jul 6, 2017 View changes

The-Compiler added PR: needs final review/merge and removed PR: work in progress labels Jul 6, 2017

Kingdread reviewed Jul 8, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

The-Compiler commented Jul 8, 2017

This comment has been minimized.

Kingdread commented Jul 9, 2017

The-Compiler merged commit c9fd182 into qutebrowser:master Jul 9, 2017

This comment has been minimized.

The-Compiler commented Jul 9, 2017

This comment has been minimized.

iordanisg commented Jul 9, 2017

qutebrowser-bot added the status: stale label Dec 20, 2017

iordanisg commented Jun 26, 2017 •

edited by The-Compiler

The-Compiler requested changes Jun 26, 2017

View changes

Kingdread commented Jun 29, 2017 •

edited

iordanisg commented Jul 6, 2017 •

edited

The-Compiler requested changes Jul 6, 2017

View changes

The-Compiler requested changes Jul 6, 2017

View changes

The-Compiler approved these changes Jul 6, 2017

View changes

Kingdread reviewed Jul 8, 2017

View changes

The-Compiler merged commit `c9fd182` into qutebrowser:master Jul 9, 2017