Merged upstream, made requested modifications to the search engine co…

…de and added tests

.appveyor.yml

@@ -5,15 +5,15 @@ cache:
 build: off
 environment:
     PYTHONUNBUFFERED: 1
-    PYTHON: C:\Python36\python.exe
+    PYTHON: C:\Python36-x64\python.exe
     matrix:
-    - TESTENV: py36-pyqt510
+    - TESTENV: py36-pyqt511
     - TESTENV: pylint
 
 install:
   - '%PYTHON% -m pip install -U pip'
   - '%PYTHON% -m pip install -r misc\requirements\requirements-tox.txt'
-  - 'set PATH=%PATH%;C:\Python36'
+  - 'set PATH=C:\Python36-x64;%PATH'
 
 test_script:
   - '%PYTHON% -m tox -e %TESTENV%'

.github/CODEOWNERS

@@ -1,5 +1,5 @@
 qutebrowser/browser/history.py  @rcorre
-qutebrowser/completion/*  @rcorre
+qutebrowser/completion/**  @rcorre
 qutebrowser/misc/sql.py  @rcorre
 tests/end2end/features/completion.feature  @rcorre
 tests/end2end/features/test_completion_bdd.py  @rcorre

.github/CONTRIBUTING.asciidoc

@@ -1,3 +1,8 @@
+IMPORTANT: I'm currently (July 2018) more busy than usual until September,
+because of exams coming up. Review of non-trivial pull requests will thus be
+delayed until then. If you're reading this note after mid-September, please
+open an issue.
+
 - Before you start to work on something, please leave a comment on the relevant
   issue (or open one). This makes sure there is no duplicate work done.
 

.travis.yml

@@ -20,18 +20,21 @@ matrix:
     - os: linux
       env: TESTENV=py36-pyqt59
     - os: linux
-      env: TESTENV=py36-pyqt510-cov
-      # We need a newer Xvfb as a WORKAROUND for:
-      # https://bugreports.qt.io/browse/QTBUG-64928
-      sudo: required
+      env: TESTENV=py36-pyqt510
       addons:
-         apt:
-           sources:
-           - sourceline: "deb http://us.archive.ubuntu.com/ubuntu/ xenial main universe"
-           packages:
-           - xvfb
+        apt:
+          packages:
+          - xfonts-base
+    - os: linux
+      env: TESTENV=py36-pyqt511-cov
+    # https://github.com/travis-ci/travis-ci/issues/9069
+    - os: linux
+      python: 3.7
+      sudo: required
+      dist: xenial
+      env: TESTENV=py37-pyqt511
     - os: osx
-      env: TESTENV=py36 OSX=sierra
+      env: TESTENV=py37 OSX=sierra
       osx_image: xcode9.2
       language: generic
     # https://github.com/qutebrowser/qutebrowser/issues/2013
@@ -66,6 +69,10 @@ matrix:
       env: TESTENV=shellcheck
       services: docker
   fast_finish: true
+  allow_failures:
+    # https://github.com/qutebrowser/qutebrowser/issues/4055
+    - os: linux
+      env: TESTENV=py36-pyqt510
 
 cache:
   directories:

README.asciidoc

@@ -59,7 +59,7 @@ Getting help
 
 You can get help in the IRC channel
 irc://irc.freenode.org/#qutebrowser[`#qutebrowser`] on
-http://freenode.net/[Freenode]
+https://freenode.net/[Freenode]
 (https://webchat.freenode.net/?channels=#qutebrowser[webchat]), or by writing a
 message to the
 https://lists.schokokeks.org/mailman/listinfo.cgi/qutebrowser[mailinglist] at
@@ -96,8 +96,8 @@ Requirements
 
 The following software and libraries are required to run qutebrowser:
 
-* http://www.python.org/[Python] 3.5 or newer (3.6 recommended)
-* http://qt.io/[Qt] 5.7.1 or newer (5.10 recommended) with the following modules:
+* https://www.python.org/[Python] 3.5 or newer (3.6 recommended)
+* https://www.qt.io/[Qt] 5.7.1 or newer (5.11.1 recommended) with the following modules:
   - QtCore / qtbase
   - QtQuick (part of qtbase in some distributions)
   - QtSQL (part of qtbase in some distributions)
@@ -106,14 +106,14 @@ The following software and libraries are required to run qutebrowser:
   - QtWebKit - only the
     link:https://github.com/annulen/webkit/wiki[updated fork] (5.212) is
     supported
-* http://www.riverbankcomputing.com/software/pyqt/intro[PyQt] 5.7.0 or newer
-  (5.10 recommended) for Python 3
+* https://www.riverbankcomputing.com/software/pyqt/intro[PyQt] 5.7.0 or newer
+  (5.11.2 recommended) for Python 3
 * https://pypi.python.org/pypi/setuptools/[pkg_resources/setuptools]
-* http://fdik.org/pyPEG/[pyPEG2]
+* https://fdik.org/pyPEG/[pyPEG2]
 * http://jinja.pocoo.org/[jinja2]
 * http://pygments.org/[pygments]
 * https://github.com/yaml/pyyaml[PyYAML]
-* http://www.attrs.org/[attrs]
+* https://www.attrs.org/[attrs]
 
 The following libraries are optional:
 
@@ -152,7 +152,7 @@ https://github.com/qutebrowser/qutebrowser/graphs/contributors[hundreds of contr
 
 Additionally, the following people have contributed graphics:
 
-* Jad/link:http://yelostudio.com[yelo] (new icon)
+* Jad/link:https://yelostudio.com[yelo] (new icon)
 * WOFall (original icon)
 * regines (key binding cheatsheet)
 
@@ -170,18 +170,15 @@ Active
 
 * https://fanglingsu.github.io/vimb/[vimb] (C, GTK+ with WebKit2)
 * https://luakit.github.io/luakit/[luakit] (C/Lua, GTK+ with WebKit2)
-* http://surf.suckless.org/[surf] (C, GTK+ with WebKit1/WebKit2)
-* http://www.uzbl.org/[uzbl] (C, GTK+ with WebKit1/WebKit2)
+* https://surf.suckless.org/[surf] (C, GTK+ with WebKit1/WebKit2)
+* https://github.com/next-browser/next/[next] (Lisp, Emacs-like, GTK+ with WebKit)
+* https://github.com/parkouss/webmacs/[webmacs] (Python, Emacs-like with QtWebEngine)
 * Chrome/Chromium addons:
-  https://github.com/1995eaton/chromium-vim[cVim],
-  http://vimium.github.io/[Vimium],
+  https://vimium.github.io/[Vimium],
   https://github.com/brookhong/Surfingkeys[Surfingkeys],
-  https://key.saka.io/[Saka Key]
 * Firefox addons (based on WebExtensions):
   https://addons.mozilla.org/en-GB/firefox/addon/vimium-ff/[Vimium-FF] (experimental),
-  https://key.saka.io[Saka Key],
   https://github.com/ueokande/vim-vixen[Vim Vixen],
-  https://github.com/shinglyu/QuantumVim[QuantumVim],
   https://github.com/cmcaine/tridactyl[Tridactyl] (working
   on a https://bugzilla.mozilla.org/show_bug.cgi?id=1215061[better API] for
   keyboard integration in Firefox).
@@ -192,17 +189,23 @@ Inactive
 * https://bitbucket.org/portix/dwb[dwb] (C, GTK+ with WebKit1,
 https://bitbucket.org/portix/dwb/pull-requests/22/several-cleanups-to-increase-portability/diff[unmaintained] -
 main inspiration for qutebrowser)
-* http://sourceforge.net/p/vimprobable/wiki/Home/[vimprobable] (C, GTK+ with
+* https://sourceforge.net/p/vimprobable/wiki/Home/[vimprobable] (C, GTK+ with
   WebKit1)
-* http://pwmt.org/projects/jumanji/[jumanji] (C, GTK+ with WebKit1)
+* https://wiki.archlinux.org/index.php?title=Jumanji[jumanji] (C, GTK+ with WebKit1,
+original site is gone but Arch Linux has some data)
 * http://conkeror.org/[conkeror] (Javascript, Emacs-like, XULRunner/Gecko)
+* https://www.uzbl.org/[uzbl] (C, GTK+ with WebKit1/WebKit2)
 * Firefox addons (not based on WebExtensions or no recent activity):
   http://www.vimperator.org/[Vimperator],
-  http://5digits.org/pentadactyl/[Pentadactyl],
+  http://bug.5digits.org/pentadactyl/index[Pentadactyl],
   https://github.com/akhodakivskiy/VimFx[VimFx],
+  https://key.saka.io[Saka Key],
+  https://github.com/shinglyu/QuantumVim[QuantumVim],
 * Chrome/Chromium addons:
   https://chrome.google.com/webstore/detail/vichrome/gghkfhpblkcmlkmpcpgaajbbiikbhpdi?hl=en[ViChrome],
   https://github.com/jinzhu/vrome[Vrome]
+  https://key.saka.io[Saka Key],
+  https://github.com/1995eaton/chromium-vim[cVim],
 
 License
 -------
@@ -229,4 +232,4 @@ display PDF files in the browser. Windows releases come with a bundled pdf.js.
 pdf.js is distributed under the terms of the Apache License. You can
 find a copy of the license in `qutebrowser/3rdparty/pdfjs/LICENSE` (in the
 Windows release or after running `scripts/dev/update_3rdparty.py`), or online
-http://www.apache.org/licenses/LICENSE-2.0.html[here].
+https://www.apache.org/licenses/LICENSE-2.0.html[here].

doc/changelog.asciidoc

@@ -15,12 +15,90 @@ breaking changes (such as renamed commands) can happen in minor releases.
 // `Fixed` for any bug fixes.
 // `Security` to invite users to upgrade in case of vulnerabilities.
 
-v1.4.0 (unreleased)
+v1.5.0 (unreleased)
 -------------------
 
 Added
 ~~~~~
 
+- The qute-pass userscript now has optional OTP support.
+- When `:spawn --userscript` is called with a count, that count is now
+  passed to userscripts as `$QUTE_COUNT`.
+- New `content.mouse_lock` setting to handle HTML5 pointer locking.
+
+Changed
+~~~~~~~
+
+- The `:repeat` command now takes a count which is multiplied with the given
+  "times" argument.
+- The default keybinding to leave passthrough mode was changed from `<Ctrl-V>`
+  to `<Shift-Escape>`, which makes pasting from the clipboard easier in
+  passthrough mode and is also unlikely to conflict with webpage bindings.
+- The `app_id` is now set to `qutebrowser` for Wayland.
+- `Command` or `Cmd` can now be used (instead of `Meta`) to map the Command key
+  on macOS.
+
+v1.4.2 (unreleased)
+-------------------
+
+Changed
+~~~~~~~
+
+- The `content.xss_auditing` setting is now enabled by default, to mirror
+  Chromium's rather than Qt's default behavior.
+- Long URLs in the statusbar are now elided at the end rather than in the
+  middle, to make sure the hostname is completely visible whenever possible.
+
+Fixed
+~~~~~
+
+- Crash in Qt 5.7.1 when a website uses `window.print()`.
+- The workaround for Nouveau graphic drivers now works properly again.
+- Crash when using `:follow-selected` with a link which is outside of the view.
+- Workaround for windows not showing as urgent with some window managers
+  (like i3).
+- Crash when opening URLs with some unicode characters (IDNA 2008). Those URLs
+  still won't open though, due to missing support in Qt.
+- Crash when a download directory which can't be created is configured.
+- Crash in the `importer.py` script when importing Chrome bookmarks from newer Chrome versions.
+- The `content.webrtc_public_interfaces_only` option didn't work on Qt 5.11 previously (it now does).
+  Note it still does not work on Qt 5.10 (due to a Qt bug) and Qt < 5.9.2.
+- Repeated escaping of entries in `qute://log` when refreshing page.
+- The host blocker doesn't block 0.0.0.0 anymore.
+
+v1.4.1
+------
+
+Security
+~~~~~~~~
+
+- CVE-2018-10895: Fix CSRF issue on the qute://settings page, leading to
+  possible arbitrary code execution. See the related GitHub issue for details:
+  https://github.com/qutebrowser/qutebrowser/issues/4060
+
+Fixed
+~~~~~
+
+- Rare crash when an error occurs in downloads.
+- Newlines are now stripped from the :version pastebin URL.
+- There's a new `mkvenv-pypi-old` environment in `tox.ini` which installs an
+  older Qt, which is needed on Ubuntu 16.04.
+- Worked around a Qt issue which redirects to a `chrome-error://` page when
+  trying to use U2F.
+- The `link_pyqt.py` script now works correctly with PyQt 5.11.
+- The Windows installer now uninstalls the old version before installing the
+  new one, fixing issues with qutebrowser not starting after installing v1.4.0
+  over v1.3.3.
+- The `:buffer` completion now sorts tabs with indices >= 10 correctly again.
+
+v1.4.0
+------
+
+Added
+~~~~~
+
+- Support for the bundled `sip` module in PyQt 5.11 and other changes in
+  Qt/PyQt 5.11.x.
 - New `--debug-flag log-requests` to log requests to the debug log for
   debugging.
 - New `--first` flag for `:hint` (bound to `gi` for inputs) which automatically
@@ -40,11 +118,14 @@ Added
   * Support for requesting persistent storage via
     `navigator.webkitPersistentStorage.requestQuota` with a new
     `content.persistent_storage` setting (requires Qt 5.11).
+    This setting also supports URL patterns.
   * Support for registering custom protocol handlers via
     `navigator.registerProtocolHandler` with a new
     `content.register_protocol_handler` setting (requires Qt 5.11).
+    This setting also supports URL patterns.
   * Support for WebRTC screen sharing with a new `content.desktop_capture`
     setting (requires Qt 5.10).
+    This setting also supports URL patterns.
   * New `content.autoplay` setting to enable/disable automatic video playback
     (requires Qt 5.10).
   * New `content.webrtc_public_interfaces_only` setting to only expose public
@@ -55,8 +136,17 @@ Added
 Changed
 ~~~~~~~
 
+- The following settings now support URL patterns:
+  * `content.headers.do_not_track`
+  * `content.headers.custom`
+  * `content.headers.accept_language`
+  * `content.headers.user_agent`
+  * `content.ssl_strict`
+  * `content.geolocation`
+  * `content.notifications`
+  * `content.media_capture`
 - The Windows/macOS releases now bundle Qt 5.11.1 which is based on
-  Chromium 65.0.3325.151 with security fixes up to Chromium 67.0.3396.79.
+  Chromium 65.0.3325.151 with security fixes up to Chromium 67.0.3396.87.
 - New short flags for commandline arguments: `-B` and `-T` for `--basedir` and
   `--temp-basedir`; `-d` and `-D` for `--debug` and `--debug-flag`.
 - Deleting history items via `:history-clear` or `:completion-item-del` now
@@ -87,26 +177,58 @@ Changed
   browsing.
 - When a prompt is opened in insert/passthrough mode, the mode is restored
   after closing the prompt.
+- On Qt 5.10 or newer, dictionaries are now read from the qutebrowser data
+  directory (e.g. `~/.local/share/qutebrowser`) instead of `/usr/share/qt`.
+  Existing dictionaries are copied over.
+- If an error while parsing `~/.netrc` occurs, the cause of the error is now
+  logged.
+- On Qt 5.9 or newer, certificate errors now show Chromium's detailed error
+  page.
+- Greasemonkey scripts now support a "@qute-js-world" tag to run them in a
+  different JavaScript context.
 
 Fixed
 ~~~~~
 
 - Various subtle keyboard focus issues.
+- The security fix in v1.3.3 caused URLs with ampersands
+  (`www.example.com?one=1&two=2`) to send the wrong arguments when clicked on
+  the `qute://history` page.
+- Crash when opening a PDF page with PDF.js enabled (on QtWebKit), but no
+  PDF.js installed.
+- Crash when closing a tab shortly after opening it.
 
 Removed
 ~~~~~~~
 
+- No prebuilt binaries for 32-bit Windows are supplied anymore. This is due to
+  Qt removing QtWebEngine support for those upstream. It might be possible to
+  distribute 32-bit binaries again with Qt 5.12 in December, but that will only
+  happen if it turns out enough people actually need 32-bit support.
 - `:tab-detach` which has been deprecated in v1.1.0 has been removed.
 - The `content.developer_extras` setting got removed. On QtWebKit, developer
   extras are now automatically enabled when opening the inspector.
 
-v1.3.3 (unreleased)
--------------------
+v1.3.3
+------
+
+Security
+~~~~~~~~
+
+- An XSS vulnerability on the `qute://history` page allowed websites to inject
+  HTML into the page via a crafted title tag. This could allow them to steal
+  your browsing history. If you're currently unable to upgrade, avoid using
+  `:history`. A CVE request for this issue is pending, see
+  https://github.com/qutebrowser/qutebrowser/issues/4011[#4011] for updates.
 
 Fixed
 ~~~~~
 
 - Crash in a workaround for a Qt 5.11 bug in rare circumstances.
+- Workaround for a Qt bug which preserves searches between page loads.
+- In v1.3.2 a dependency on the `PyQt5.QtQuickWidgets` module was accidentally
+  introduced. Since that module isn't packaged everywhere, it's been removed
+  again.
 
 v1.3.2
 ------