New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Completion refactor V3 #2295

Merged
merged 163 commits into from Jul 21, 2017

Conversation

Projects
None yet
8 participants
@rcorre
Collaborator

rcorre commented Feb 8, 2017

This is a combination of my previous efforts for #74 and #1765.
It refactors completion models into functions that instantiate models on-demand. In order to avoid taking a performace hit, some models are backed by SQL tables.

This approach is a little different than last time in that the SQL does not
exist solely in the completion module. Instead, the bookmark-manager,
quickmark-manager, and web-history are now backed by an in-memory SQL
database rather than a python dict (see e130063 and f095dbe). On startup, they
read from a file on disk and write entries into the in-memory database. This is
the main part I'm looking for feedback on right now. How does it feel? The
advantage of this approach is that SQL-based completion models can be
returned on-demand just as the simple completion models are.

It would likely be even more performant and simpler to replace the old bookmark/quickmark/history text files with an on-disk database, but that could
be a usability hit for those that like to mess with these files manually.

P.S. don't expect this to work just yet, but it will be there soon.


This change is Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 8, 2017

Collaborator

Can we close #2144 and/or #2178 with this open?

It would likely be even more performant and simpler to replace the old bookmark/quickmark/history text files with an on-disk database, but that could
be a usability hit for those that like to mess with these files manually.

I've been thinking about that too. I originally wanted everything to be plain-text, but at least for cookies I can't do that anymore with QtWebEngine either.

I wonder if we could just ship a script (say, qutebrowser-dumpsql or whatever) which dumps those SQL databases to the earlier plain-text formats, so at least thinks like https://github.com/qutebrowser/qutebrowser/blob/master/misc/userscripts/qutedmenu could be ported easily.

We need the conversion code in qutebrowser anyways (to migrate existing data), so the only question is whether the performance hit on start is too big. If it is, I'm open to doing that.

Collaborator

The-Compiler commented Feb 8, 2017

Can we close #2144 and/or #2178 with this open?

It would likely be even more performant and simpler to replace the old bookmark/quickmark/history text files with an on-disk database, but that could
be a usability hit for those that like to mess with these files manually.

I've been thinking about that too. I originally wanted everything to be plain-text, but at least for cookies I can't do that anymore with QtWebEngine either.

I wonder if we could just ship a script (say, qutebrowser-dumpsql or whatever) which dumps those SQL databases to the earlier plain-text formats, so at least thinks like https://github.com/qutebrowser/qutebrowser/blob/master/misc/userscripts/qutedmenu could be ported easily.

We need the conversion code in qutebrowser anyways (to migrate existing data), so the only question is whether the performance hit on start is too big. If it is, I'm open to doing that.

@The-Compiler

I've only taken a relatively quick look, and I already love how this makes everything simpler, despite involving SQL 😆 Great work so far!

The only thing I'm really unsure about is that it's a QAbstractItemModel actually used in the view, and not the SQL model. Doesn't this hinder performance a lot again? Can we use the Qt model directly?

Show outdated Hide outdated qutebrowser/browser/history.py
Signals:
add_completion_item: Emitted before a new Entry is added.
Used to sync with the completion.
arg: The new Entry.
item_added: Emitted after a new Entry is added.
Used to tell the savemanager that the history is dirty.
arg: The new Entry.

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Probably can be removed as the signal is gone.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Probably can be removed as the signal is gone.

Show outdated Hide outdated qutebrowser/browser/urlmarks.py
@@ -53,43 +53,25 @@ class InvalidUrlError(Error):
pass
class DoesNotExistError(Error):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Why get rid of this? Abusing a KeyError (which I view more of a Python built-in exception and can also happen by accident) for this seems wrong to me.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Why get rid of this? Abusing a KeyError (which I view more of a Python built-in exception and can also happen by accident) for this seems wrong to me.

This comment has been minimized.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

To expand on this: I think it's fine to use KeyError on a lower level, i.e. with the SqlTable object with a dict-like interface. However, on this higher level, I think it makes sense to re-raise them as more specific exceptions.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

To expand on this: I think it's fine to use KeyError on a lower level, i.e. with the SqlTable object with a dict-like interface. However, on this higher level, I think it makes sense to re-raise them as more specific exceptions.

This comment has been minimized.

@rcorre

rcorre Feb 12, 2017

Collaborator

done

@rcorre

rcorre Feb 12, 2017

Collaborator

done

Show outdated Hide outdated qutebrowser/completion/models/sortfilter.py
@@ -48,21 +47,12 @@ def __init__(self, source, parent=None):
self.srcmodel = source
self.pattern = ''
self.pattern_re = None
dumb_sort = self.srcmodel.DUMB_SORT

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

I'm guessing all the dumb_sort stuff can go because it was only used for the URL model, which doesn't use QSortFilterProxyModel anymore?

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

I'm guessing all the dumb_sort stuff can go because it was only used for the URL model, which doesn't use QSortFilterProxyModel anymore?

This comment has been minimized.

@rcorre

rcorre Feb 12, 2017

Collaborator

done

@rcorre

rcorre Feb 12, 2017

Collaborator

done

Show outdated Hide outdated qutebrowser/completion/models/sortfilter.py
else:
self.setSortRole(completion.Role.sort)
self._sort_order = dumb_sort
self.lessThan = self.intelligentLessThan

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

You can just rename the method now 😉

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

You can just rename the method now 😉

This comment has been minimized.

@rcorre

rcorre Feb 10, 2017

Collaborator

oh, right 😀

@rcorre

rcorre Feb 10, 2017

Collaborator

oh, right 😀

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
from qutebrowser.utils import usertypes, log
class SqlCompletionModel(QAbstractItemModel):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

The original reason we wanted to have sqlite models was performance, as Qt probably implements lazy loading and all that there very efficiently. Don't we lose all that by having this be a QAbstractItemModel?

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

The original reason we wanted to have sqlite models was performance, as Qt probably implements lazy loading and all that there very efficiently. Don't we lose all that by having this be a QAbstractItemModel?

This comment has been minimized.

@rcorre

rcorre Feb 10, 2017

Collaborator

There's no 'natural' way to make a SqlTableModel/SqlQueryModel look like a tree. This is a wrapper around several SqlTableModels, each of which populates a single category. I hope I'm not losing out on lazy loading here! At the very least I'm still relying on SqlTableModel for filtering!

@rcorre

rcorre Feb 10, 2017

Collaborator

There's no 'natural' way to make a SqlTableModel/SqlQueryModel look like a tree. This is a wrapper around several SqlTableModels, each of which populates a single category. I hope I'm not losing out on lazy loading here! At the very least I'm still relying on SqlTableModel for filtering!

This comment has been minimized.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

Makes sense - I originally thought about ditching the tree model entirely and using a QVBoxLayout of QTableViews, but that'd probably make things like sizing a lot more difficult.

Makes me wonder whether it somehow helps to implement canFetchMore and fetchMore - though we don't really have a slow "data generation" step, so I guess it won't help.

I think a few benchmark tests (after #2281 is in) would probably help there too, as we can test what's helping and what's not much easier.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

Makes sense - I originally thought about ditching the tree model entirely and using a QVBoxLayout of QTableViews, but that'd probably make things like sizing a lot more difficult.

Makes me wonder whether it somehow helps to implement canFetchMore and fetchMore - though we don't really have a slow "data generation" step, so I guess it won't help.

I think a few benchmark tests (after #2281 is in) would probably help there too, as we can test what's helping and what's not much easier.

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
fields = (t.record().fieldName(i) for i in self.columns_to_filter)
query = ' or '.join("{} like '%{}%' escape '\\'"
.format(field, pattern)
for field in fields)

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

What if there's a ' in the pattern? Can't we use QSqlQuery here too?

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

What if there's a ' in the pattern? Can't we use QSqlQuery here too?

Show outdated Hide outdated qutebrowser/misc/sql.py
return query
class SqlTable(QObject):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Love the pythonic interface over SQL stuff! 👍

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Love the pythonic interface over SQL stuff! 👍

This comment has been minimized.

@rcorre

rcorre Feb 10, 2017

Collaborator

Thanks! That actually answers your question about KeyError though. I saw this as a dict-like interface over a sql table. When you try to access or delete a key that doesn't exist, you get a KeyError just as you would if it were a python dict. I didn't see a reason to invent a new exception just for that (I do have SqlException, but that's more for things like 'your SQL connection died')

@rcorre

rcorre Feb 10, 2017

Collaborator

Thanks! That actually answers your question about KeyError though. I saw this as a dict-like interface over a sql table. When you try to access or delete a key that doesn't exist, you get a KeyError just as you would if it were a python dict. I didn't see a reason to invent a new exception just for that (I do have SqlException, but that's more for things like 'your SQL connection died')

Show outdated Hide outdated tests/unit/completion/test_sqlmodel.py
# You should have received a copy of the GNU General Public License
# along with qutebrowser. If not, see <http://www.gnu.org/licenses/>.
"""Tests for the base sql completion model."""

This comment has been minimized.

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Those should probably use qtmodeltester where appropriate, and maybe pytest-benchmark for some interesting operations (which I'm introducing in #2281).

@The-Compiler

The-Compiler Feb 9, 2017

Collaborator

Those should probably use qtmodeltester where appropriate, and maybe pytest-benchmark for some interesting operations (which I'm introducing in #2281).

This comment has been minimized.

@rcorre

rcorre Feb 10, 2017

Collaborator

I must've lost that while cherry-picking over from my previous branch. Working on this model actually made me really appreciate qmodeltester, it caught like 10 dumb things I was doing!

@rcorre

rcorre Feb 10, 2017

Collaborator

I must've lost that while cherry-picking over from my previous branch. Working on this model actually made me really appreciate qmodeltester, it caught like 10 dumb things I was doing!

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Looks like it's still not back?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Looks like it's still not back?

Show outdated Hide outdated qutebrowser/completion/models/base.py
completion view.
DUMB_SORT: the dumb sorting used by the model
dumb_sort: the dumb sorting used by the model

This comment has been minimized.

@rcorre

rcorre Feb 10, 2017

Collaborator

this can go too

@rcorre

rcorre Feb 10, 2017

Collaborator

this can go too

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 10, 2017

Collaborator

To answer your main question more fully, no, we can't use a QSqlTableModel/QSqlQueryModel directly as long as we want to group by category using a tree view. We need to wrap the table models in either an item model or a proxy model, and I found the item model to be much easier.

Furthermore, we may actually want a model that combines both SQL and item-based sources. Otherwise, to support #2191, we'd need to create a sql table to include searchengines. I'm imagining something like:

model = CompletionModel()
model.add_sql_category('History')
model.add_sql_category('Bookmarks')
model.add_sql_category('Quickmarks')
model.add_list_category([('ddg', 'duckduckgo'), ...])
Collaborator

rcorre commented Feb 10, 2017

To answer your main question more fully, no, we can't use a QSqlTableModel/QSqlQueryModel directly as long as we want to group by category using a tree view. We need to wrap the table models in either an item model or a proxy model, and I found the item model to be much easier.

Furthermore, we may actually want a model that combines both SQL and item-based sources. Otherwise, to support #2191, we'd need to create a sql table to include searchengines. I'm imagining something like:

model = CompletionModel()
model.add_sql_category('History')
model.add_sql_category('Bookmarks')
model.add_sql_category('Quickmarks')
model.add_list_category([('ddg', 'duckduckgo'), ...])
@The-Compiler

A quick look at Travis shows that you have a circular import Python 3.4 doesn't like, some actual failures, and of course some linting errors, but I'm guessing that's all WIP. The Ubuntu Xenial one fails to init the sqlite backend, I'm guessing some additional package is needed there?

edit: Looks like it's this one. We should also check if QtSql and sqlite is available in earlyinit.py to show an error if it isn't.

Show outdated Hide outdated qutebrowser/browser/urlmarks.py
@@ -53,43 +53,25 @@ class InvalidUrlError(Error):
pass
class DoesNotExistError(Error):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

To expand on this: I think it's fine to use KeyError on a lower level, i.e. with the SqlTable object with a dict-like interface. However, on this higher level, I think it makes sense to re-raise them as more specific exceptions.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

To expand on this: I think it's fine to use KeyError on a lower level, i.e. with the SqlTable object with a dict-like interface. However, on this higher level, I think it makes sense to re-raise them as more specific exceptions.

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
from qutebrowser.utils import usertypes, log
class SqlCompletionModel(QAbstractItemModel):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

Makes sense - I originally thought about ditching the tree model entirely and using a QVBoxLayout of QTableViews, but that'd probably make things like sizing a lot more difficult.

Makes me wonder whether it somehow helps to implement canFetchMore and fetchMore - though we don't really have a slow "data generation" step, so I guess it won't help.

I think a few benchmark tests (after #2281 is in) would probably help there too, as we can test what's helping and what's not much easier.

@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

Makes sense - I originally thought about ditching the tree model entirely and using a QVBoxLayout of QTableViews, but that'd probably make things like sizing a lot more difficult.

Makes me wonder whether it somehow helps to implement canFetchMore and fetchMore - though we don't really have a slow "data generation" step, so I guess it won't help.

I think a few benchmark tests (after #2281 is in) would probably help there too, as we can test what's helping and what's not much easier.

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 10, 2017

Collaborator

I also just noticed this from the Qt docs:

The driver is locked for updates while a select is executed. This may cause problems when using QSqlTableModel because Qt's item views fetch data as needed (with QSqlQuery::fetchMore() in the case of QSqlTableModel).

Not sure if that applies to us, and if/how we need to handle it.

Collaborator

The-Compiler commented Feb 10, 2017

I also just noticed this from the Qt docs:

The driver is locked for updates while a select is executed. This may cause problems when using QSqlTableModel because Qt's item views fetch data as needed (with QSqlQuery::fetchMore() in the case of QSqlTableModel).

Not sure if that applies to us, and if/how we need to handle it.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 12, 2017

Collaborator

Hmm, if I'm understanding that correctly, it could be a problem if you try to enter a history item while you have completion open and it is select()ing. Maybe if you hit o while a page is loading, and then it finishes loading while the completion menu is still filling out.

Collaborator

rcorre commented Feb 12, 2017

Hmm, if I'm understanding that correctly, it could be a problem if you try to enter a history item while you have completion open and it is select()ing. Maybe if you hit o while a page is loading, and then it finishes loading while the completion menu is still filling out.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 16, 2017

Collaborator

Hmm ... I can't figure out what's up with this failure. For some reason it only has a problem initializing SQL for those tests. Other tests, like the whole completion test suite, run fine, so Sqlite is clearly installed ...
Those tests worked fine locally even without init_sql

Also, for Xenial, do I need to install libqt5sql5-sqlite in travis_install.sh, or in https://github.com/rcorre/docker-travis, or both? What's the difference?

Collaborator

rcorre commented Feb 16, 2017

Hmm ... I can't figure out what's up with this failure. For some reason it only has a problem initializing SQL for those tests. Other tests, like the whole completion test suite, run fine, so Sqlite is clearly installed ...
Those tests worked fine locally even without init_sql

Also, for Xenial, do I need to install libqt5sql5-sqlite in travis_install.sh, or in https://github.com/rcorre/docker-travis, or both? What's the difference?

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 16, 2017

Collaborator

Bleh, never mind. Its one of those ones that only reproes when you run the full test suite (or at least everything under tests/unit/browser). Maybe calling sql.init()/sql.close() a bunch of times is problematic?

Collaborator

rcorre commented Feb 16, 2017

Bleh, never mind. Its one of those ones that only reproes when you run the full test suite (or at least everything under tests/unit/browser). Maybe calling sql.init()/sql.close() a bunch of times is problematic?

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 16, 2017

Collaborator

I can reproduce it with tox -e py36 -- tests/unit/browser tests/unit/utils. Right now I don't know what's causing this or what to do about it... 😟

Collaborator

The-Compiler commented Feb 16, 2017

I can reproduce it with tox -e py36 -- tests/unit/browser tests/unit/utils. Right now I don't know what's causing this or what to do about it... 😟

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 17, 2017

Collaborator

Ok, looks like all that's left is one failure on master and that tricky circular import. I thought I solved that early on but for some reason py3.4 still isn't happy with it.

Collaborator

rcorre commented Feb 17, 2017

Ok, looks like all that's left is one failure on master and that tricky circular import. I thought I solved that early on but for some reason py3.4 still isn't happy with it.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 17, 2017

Collaborator

Once that's fixed up, I think this is feature-complete and ready for review. There are a few pieces of work that I think are non-critical and could come in a later review:

  • Implement the searchengine completion from #2191 with the new SQL backend. This is non-trivial as it requires a model that mixes SQL and non-SQL sources. Such a model might actually lead to a cleaner API though, I'll have to play with it.
  • Replace history.Entry with SqlTable.Entry, a namedtuple type that is automatically generated from the SQL record.
  • benchmark performance (maybe this should even be a PR that goes in before this one, to get a baseline).
  • de-complicate delete_cur_item (just something I noticed while working on this)
Collaborator

rcorre commented Feb 17, 2017

Once that's fixed up, I think this is feature-complete and ready for review. There are a few pieces of work that I think are non-critical and could come in a later review:

  • Implement the searchengine completion from #2191 with the new SQL backend. This is non-trivial as it requires a model that mixes SQL and non-SQL sources. Such a model might actually lead to a cleaner API though, I'll have to play with it.
  • Replace history.Entry with SqlTable.Entry, a namedtuple type that is automatically generated from the SQL record.
  • benchmark performance (maybe this should even be a PR that goes in before this one, to get a baseline).
  • de-complicate delete_cur_item (just something I noticed while working on this)
@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 17, 2017

Collaborator

Oh, and those sql failures were leaked state. ☹️

Collaborator

rcorre commented Feb 17, 2017

Oh, and those sql failures were leaked state. ☹️

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 17, 2017

Collaborator

oh, that's unfortunate. configmodels imports config and config imports configmodels to reference the completion functions. Looks like our options are:

  1. Go back to using usertypes.Completion as a level of indirection between
    commands and completion models.

  2. Move set_command from qutebrowser.config to qutebrowser.browser.commands

  3. Drop python3.4 support

None are great options. Well, ok, I like 3, but I'm guessing a fair number of
people won't 😁

Collaborator

rcorre commented Feb 17, 2017

oh, that's unfortunate. configmodels imports config and config imports configmodels to reference the completion functions. Looks like our options are:

  1. Go back to using usertypes.Completion as a level of indirection between
    commands and completion models.

  2. Move set_command from qutebrowser.config to qutebrowser.browser.commands

  3. Drop python3.4 support

None are great options. Well, ok, I like 3, but I'm guessing a fair number of
people won't 😁

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Also, for Xenial, do I need to install libqt5sql5-sqlite in travis_install.sh, or in https://github.com/rcorre/docker-travis, or both? What's the difference?

Sorry, must've missed that one earlier!

Travis runs various test environments - some of them in their native (Ubuntu Trusty) environment, others (with DOCKER=... set) in Docker images built in the docker-travis repo.

For the native environments, what is installed in travis_install.sh matters. For Docker environments, what's installed in their Dockerfile matters.

Implement the searchengine completion from #2191 with the new SQL backend. This is non-trivial as it requires a model that mixes SQL and non-SQL sources. Such a model might actually lead to a cleaner API though, I'll have to play with it.

Hmm, I guess it'd make sense to close #2191 and reimplement that with the new completion API at some point then? Kind of a bad situation for @rsteube though...

benchmark performance (maybe this should even be a PR that goes in before this one, to get a baseline).

I really like that idea. We badly need some benchmarks for the completion (mainly filtering with many items), and it'd make sense to have them in before this PR, as that performance really shouldn't get worse. Do you want to work on this?

Oh, and those sql failures were leaked state. ☹

Dang. I noticed the first one, but I didn't get how it'd cause SQL errors and fixing it didn't seem to help, so I dropped it again. 😆

oh, that's unfortunate. configmodels imports config and config imports configmodels to reference the completion functions.

Looks like you found a good solution to this, but it makes me wonder if we're going to run into similar issues with this structure in the feature... I vaguely remember something similar was the reason for the usertypes.Completion indirection originally, though I don't really like it.

Drop python3.4 support

I've actually been thinking about that, mostly because of #1456 and because most people on an old Debian/Ubuntu would probably be happy with #1912. Not so sure yet... 😉


Some other observations:

(everthing after this was added as an edit)

  • qutebrowser/completion/models/urlmodel.py has 100% coverage but is not in perfect_files!
  • earlyinit still should check that sqlite is available and abort if not
  • qutebrowser/mainwindow/statusbar/bar.py:147: Unused attribute '_option' (I can look at that one - it seems like vulture now likes to complain about that as well, but with _option we already tell pylint to shut up... So I'll make vulture just shut up about those.)
  • qutebrowser/utils/utils.py:765: Unused function 'newest_slice' (If we can be reasonably sure it's not needed anywhere else, let's drop it.)
  • You'll need to adjust INSTALL to point out those dependencies too
  • Is there some relevant version info (I can't find anything in QtSQL, but it seems like the sqlite version might be relevant?) we should add to version.py? We can probably either use sqlite3.sqlite_version from Python (and hope it's the same version), or use SELECT sqlite_version(); from Qt
  • The Windows build also seems to be missing sqlite, I'm not sure why yet. It currently uses Qt 5.6 installed via the PyQt .exe installer. I want to update it to Qt 5.8 with QtWebEngine only at some point, but I wonder why it's missing?
  • I did rerun the docker builds with qutebrowser/docker-travis#3 in and they pass.
Collaborator

The-Compiler commented Feb 20, 2017

Also, for Xenial, do I need to install libqt5sql5-sqlite in travis_install.sh, or in https://github.com/rcorre/docker-travis, or both? What's the difference?

Sorry, must've missed that one earlier!

Travis runs various test environments - some of them in their native (Ubuntu Trusty) environment, others (with DOCKER=... set) in Docker images built in the docker-travis repo.

For the native environments, what is installed in travis_install.sh matters. For Docker environments, what's installed in their Dockerfile matters.

Implement the searchengine completion from #2191 with the new SQL backend. This is non-trivial as it requires a model that mixes SQL and non-SQL sources. Such a model might actually lead to a cleaner API though, I'll have to play with it.

Hmm, I guess it'd make sense to close #2191 and reimplement that with the new completion API at some point then? Kind of a bad situation for @rsteube though...

benchmark performance (maybe this should even be a PR that goes in before this one, to get a baseline).

I really like that idea. We badly need some benchmarks for the completion (mainly filtering with many items), and it'd make sense to have them in before this PR, as that performance really shouldn't get worse. Do you want to work on this?

Oh, and those sql failures were leaked state. ☹

Dang. I noticed the first one, but I didn't get how it'd cause SQL errors and fixing it didn't seem to help, so I dropped it again. 😆

oh, that's unfortunate. configmodels imports config and config imports configmodels to reference the completion functions.

Looks like you found a good solution to this, but it makes me wonder if we're going to run into similar issues with this structure in the feature... I vaguely remember something similar was the reason for the usertypes.Completion indirection originally, though I don't really like it.

Drop python3.4 support

I've actually been thinking about that, mostly because of #1456 and because most people on an old Debian/Ubuntu would probably be happy with #1912. Not so sure yet... 😉


Some other observations:

(everthing after this was added as an edit)

  • qutebrowser/completion/models/urlmodel.py has 100% coverage but is not in perfect_files!
  • earlyinit still should check that sqlite is available and abort if not
  • qutebrowser/mainwindow/statusbar/bar.py:147: Unused attribute '_option' (I can look at that one - it seems like vulture now likes to complain about that as well, but with _option we already tell pylint to shut up... So I'll make vulture just shut up about those.)
  • qutebrowser/utils/utils.py:765: Unused function 'newest_slice' (If we can be reasonably sure it's not needed anywhere else, let's drop it.)
  • You'll need to adjust INSTALL to point out those dependencies too
  • Is there some relevant version info (I can't find anything in QtSQL, but it seems like the sqlite version might be relevant?) we should add to version.py? We can probably either use sqlite3.sqlite_version from Python (and hope it's the same version), or use SELECT sqlite_version(); from Qt
  • The Windows build also seems to be missing sqlite, I'm not sure why yet. It currently uses Qt 5.6 installed via the PyQt .exe installer. I want to update it to Qt 5.8 with QtWebEngine only at some point, but I wonder why it's missing?
  • I did rerun the docker builds with qutebrowser/docker-travis#3 in and they pass.
@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 20, 2017

Collaborator

Hmm, I guess it'd make sense to close #2191 and reimplement that with the new completion API at some point then? Kind of a bad situation for @rsteube though...

Yeah, that's unfortunate, but I can probably at least re-use the tests.

Do you want to work on [completion benchmarks]?

Yes.

Looks like you found a good solution

Yeah, I listed 3 options and picked the fourth 😁. Its a step backward for #640 but the lesser of four evils...

with _option we already tell pylint to shut up...

Is it not an unused variable? I was about to remove it, along with
newest_slice (which was rendered unnecessary).

Collaborator

rcorre commented Feb 20, 2017

Hmm, I guess it'd make sense to close #2191 and reimplement that with the new completion API at some point then? Kind of a bad situation for @rsteube though...

Yeah, that's unfortunate, but I can probably at least re-use the tests.

Do you want to work on [completion benchmarks]?

Yes.

Looks like you found a good solution

Yeah, I listed 3 options and picked the fourth 😁. Its a step backward for #640 but the lesser of four evils...

with _option we already tell pylint to shut up...

Is it not an unused variable? I was about to remove it, along with
newest_slice (which was rendered unnecessary).

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Yeah, I listed 3 options and picked the fourth 😁. Its a step backward for #640 but the lesser of four evils...

I agree - with #640 we'd probably just have a qutebrowser.misc.objects.config there.

Is it not an unused variable? I was about to remove it

Oh, right. I assumed it was an argument for some reason, not an attribute.

Collaborator

The-Compiler commented Feb 20, 2017

Yeah, I listed 3 options and picked the fourth 😁. Its a step backward for #640 but the lesser of four evils...

I agree - with #640 we'd probably just have a qutebrowser.misc.objects.config there.

Is it not an unused variable? I was about to remove it

Oh, right. I assumed it was an argument for some reason, not an attribute.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 20, 2017

Collaborator

I think I fixed all the things, except earlyinit. Is there any advantage to
adding a check earlier? init will throw a SqlException if sqlite is not
available. The only way I know to check is actually trying to open a database
(QSqlDatabase.isDriverAvailable('QSQLITE') will return True even if I
uninstall sqlite). Should sql.init happen in earlyinit instead of init?

Collaborator

rcorre commented Feb 20, 2017

I think I fixed all the things, except earlyinit. Is there any advantage to
adding a check earlier? init will throw a SqlException if sqlite is not
available. The only way I know to check is actually trying to open a database
(QSqlDatabase.isDriverAvailable('QSQLITE') will return True even if I
uninstall sqlite). Should sql.init happen in earlyinit instead of init?

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

QSqlDatabase.isDriverAvailable('QSQLITE') will return True even if I
uninstall sqlite

And sql.init() actually fails then?

Should sql.init happen in earlyinit instead of init?

You're right - my main point is that there should be an error message showing pointing out what's wrong. This is usually done in earlyinit so far, but if it's more practical, it's fine to do it elsewhere during init as well.

Collaborator

The-Compiler commented Feb 20, 2017

QSqlDatabase.isDriverAvailable('QSQLITE') will return True even if I
uninstall sqlite

And sql.init() actually fails then?

Should sql.init happen in earlyinit instead of init?

You're right - my main point is that there should be an error message showing pointing out what's wrong. This is usually done in earlyinit so far, but if it's more practical, it's fine to do it elsewhere during init as well.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 20, 2017

Collaborator

yup, sql.init() is called from the main app init and will fail if sqlite
is not installed.

earlyinit didn't seem like the place:

# NOTE: No qutebrowser or PyQt import should be done here, as some early
# initialization needs to take place before that!
Collaborator

rcorre commented Feb 20, 2017

yup, sql.init() is called from the main app init and will fail if sqlite
is not installed.

earlyinit didn't seem like the place:

# NOTE: No qutebrowser or PyQt import should be done here, as some early
# initialization needs to take place before that!
@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Feb 20, 2017

Codecov Report

Merging #2295 into master will decrease coverage by -0.05%.
The diff coverage is 93.64%.

@@            Coverage Diff             @@
##           master    #2295      +/-   ##
==========================================
- Coverage   81.18%   81.14%   -0.05%     
==========================================
  Files         132      133       +1     
  Lines       18139    18008     -131     
  Branches     2728     2708      -20     
==========================================
- Hits        14727    14612     -115     
+ Misses       2939     2932       -7     
+ Partials      473      464       -9
Impacted Files Coverage Δ
qutebrowser/mainwindow/statusbar/bar.py 96.92% <ø> (-0.03%)
qutebrowser/utils/utils.py 100% <ø> (ø)
qutebrowser/utils/usertypes.py 100% <ø> (ø)
qutebrowser/completion/completionwidget.py 91.76% <100%> (+2.22%)
qutebrowser/utils/log.py 84.42% <100%> (+0.05%)
qutebrowser/completion/models/urlmodel.py 100% <100%> (+18.81%)
qutebrowser/utils/version.py 100% <100%> (ø)
qutebrowser/completion/models/sortfilter.py 100% <100%> (ø)
qutebrowser/browser/webkit/webkithistory.py 100% <100%> (ø)
qutebrowser/completion/models/base.py 100% <100%> (ø)
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c03ad3...dee9a75. Read the comment docs.

codecov bot commented Feb 20, 2017

Codecov Report

Merging #2295 into master will decrease coverage by -0.05%.
The diff coverage is 93.64%.

@@            Coverage Diff             @@
##           master    #2295      +/-   ##
==========================================
- Coverage   81.18%   81.14%   -0.05%     
==========================================
  Files         132      133       +1     
  Lines       18139    18008     -131     
  Branches     2728     2708      -20     
==========================================
- Hits        14727    14612     -115     
+ Misses       2939     2932       -7     
+ Partials      473      464       -9
Impacted Files Coverage Δ
qutebrowser/mainwindow/statusbar/bar.py 96.92% <ø> (-0.03%)
qutebrowser/utils/utils.py 100% <ø> (ø)
qutebrowser/utils/usertypes.py 100% <ø> (ø)
qutebrowser/completion/completionwidget.py 91.76% <100%> (+2.22%)
qutebrowser/utils/log.py 84.42% <100%> (+0.05%)
qutebrowser/completion/models/urlmodel.py 100% <100%> (+18.81%)
qutebrowser/utils/version.py 100% <100%> (ø)
qutebrowser/completion/models/sortfilter.py 100% <100%> (ø)
qutebrowser/browser/webkit/webkithistory.py 100% <100%> (ø)
qutebrowser/completion/models/base.py 100% <100%> (ø)
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c03ad3...dee9a75. Read the comment docs.

@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Feb 20, 2017

Codecov Report

Merging #2295 into master will decrease coverage by 0.08%.
The diff coverage is 94.02%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2295      +/-   ##
==========================================
- Coverage   81.01%   80.92%   -0.09%     
==========================================
  Files         132      134       +2     
  Lines       18504    18344     -160     
  Branches     2829     2793      -36     
==========================================
- Hits        14991    14845     -146     
+ Misses       3031     3026       -5     
+ Partials      482      473       -9
Impacted Files Coverage Δ
qutebrowser/misc/earlyinit.py 65.26% <ø> (-0.52%) ⬇️
qutebrowser/config/configdata.py 100% <ø> (ø) ⬆️
qutebrowser/utils/utils.py 100% <ø> (ø) ⬆️
qutebrowser/utils/usertypes.py 100% <ø> (ø) ⬆️
qutebrowser/browser/commands.py 84.71% <100%> (-0.55%) ⬇️
qutebrowser/utils/log.py 85.91% <100%> (+0.04%) ⬆️
qutebrowser/completion/completiondelegate.py 76.25% <100%> (+0.17%) ⬆️
qutebrowser/misc/lineparser.py 89.1% <100%> (-0.83%) ⬇️
qutebrowser/completion/models/histcategory.py 100% <100%> (ø)
qutebrowser/completion/models/urlmodel.py 100% <100%> (+18.81%) ⬆️
... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a572b0f...33a9c8c. Read the comment docs.

codecov bot commented Feb 20, 2017

Codecov Report

Merging #2295 into master will decrease coverage by 0.08%.
The diff coverage is 94.02%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2295      +/-   ##
==========================================
- Coverage   81.01%   80.92%   -0.09%     
==========================================
  Files         132      134       +2     
  Lines       18504    18344     -160     
  Branches     2829     2793      -36     
==========================================
- Hits        14991    14845     -146     
+ Misses       3031     3026       -5     
+ Partials      482      473       -9
Impacted Files Coverage Δ
qutebrowser/misc/earlyinit.py 65.26% <ø> (-0.52%) ⬇️
qutebrowser/config/configdata.py 100% <ø> (ø) ⬆️
qutebrowser/utils/utils.py 100% <ø> (ø) ⬆️
qutebrowser/utils/usertypes.py 100% <ø> (ø) ⬆️
qutebrowser/browser/commands.py 84.71% <100%> (-0.55%) ⬇️
qutebrowser/utils/log.py 85.91% <100%> (+0.04%) ⬆️
qutebrowser/completion/completiondelegate.py 76.25% <100%> (+0.17%) ⬆️
qutebrowser/misc/lineparser.py 89.1% <100%> (-0.83%) ⬇️
qutebrowser/completion/models/histcategory.py 100% <100%> (ø)
qutebrowser/completion/models/urlmodel.py 100% <100%> (+18.81%) ⬆️
... and 37 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a572b0f...33a9c8c. Read the comment docs.

@The-Compiler

First full review of this. A few bigger design things and some smaller stuff, but generally it's visible a lot of thought and work went into this already! 👍

Show outdated Hide outdated qutebrowser/app.py
@@ -404,6 +405,13 @@ def _init_modules(args, crash_handler):
config.init(qApp)
save_manager.init_autosave()
log.init.debug("Initializing keys...")
keyconf.init(qApp)
save_manager.init_autosave()

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Did you intend to call this again?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Did you intend to call this again?

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

nope, good catch. Actually, this came from 5734c46 which isn't necessary as I solved the circular import another way, but it might not be bad to eliminate configs dependence on keyconf anyways (it only depended on the module to initialize it)

@rcorre

rcorre Feb 20, 2017

Collaborator

nope, good catch. Actually, this came from 5734c46 which isn't necessary as I solved the circular import another way, but it might not be bad to eliminate configs dependence on keyconf anyways (it only depended on the module to initialize it)

Show outdated Hide outdated qutebrowser/misc/sql.py
raise KeyError('No row with {} = "{}"'.format(field, value))
self.changed.emit()
def insert(self, values, replace=False):

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I'm kinda wondering if there'd be a nicer API here, using keyword arguments. It looks like QSqlQuery::bindValue has an overload taking a string as placeholder.

Then instead of:

self.insert([entry.url_str(), entry.title, entry.atime, entry.redirect], replace=True)

you could do:

self.insert(url=entry.url_str(), title=entry.title, atime=entry.atime, redirect=entry.redirect)

which seems way less errorprone. Not sure what to do about redirect as it could conflict with a column with that name - I guess it could just be a separate method (with e.g. a private common method with a redirect_ parameter or so behind it).

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I'm kinda wondering if there'd be a nicer API here, using keyword arguments. It looks like QSqlQuery::bindValue has an overload taking a string as placeholder.

Then instead of:

self.insert([entry.url_str(), entry.title, entry.atime, entry.redirect], replace=True)

you could do:

self.insert(url=entry.url_str(), title=entry.title, atime=entry.atime, redirect=entry.redirect)

which seems way less errorprone. Not sure what to do about redirect as it could conflict with a column with that name - I guess it could just be a separate method (with e.g. a private common method with a redirect_ parameter or so behind it).

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

yes! I meant to do that after 572aaf1. It could be self.insert(self.Entry(field, field, ...), replace=True)

@rcorre

rcorre Feb 20, 2017

Collaborator

yes! I meant to do that after 572aaf1. It could be self.insert(self.Entry(field, field, ...), replace=True)

Show outdated Hide outdated qutebrowser/browser/qutescheme.py
quickmarks = sorted(objreg.get('quickmark-manager').marks.items(),
key=lambda x: x[0]) # Sort by name
bookmarks = sorted(objreg.get('bookmark-manager'), key=lambda x: x.title)
quickmarks = sorted(objreg.get('quickmark-manager'), key=lambda x: x.name)

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

operator.attrgetter('name') (and the equivalent for above) would probably be a bit nicer, but I'll let you decide 😉

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

operator.attrgetter('name') (and the equivalent for above) would probably be a bit nicer, but I'll let you decide 😉

Show outdated Hide outdated qutebrowser/browser/qutescheme.py
@@ -183,7 +181,7 @@ def qute_history(url):
def history_iter(reverse):
"""Iterate through the history and get items we're interested in."""
curr_timestamp = time.mktime(curr_date.timetuple())
history = objreg.get('web-history').history_dict.values()
history = list(objreg.get('web-history'))

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

We tried to avoid calling list() here in #2281 because it constructs a copy of the entire history, which took almost a second on my machine with some entries. And indeed, test_qute_history_benchmark goes from 80us to >1s with your branch on my machine.

I guess you could implement history_iter much more efficiently using SQL though, by doing all the filtering there?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

We tried to avoid calling list() here in #2281 because it constructs a copy of the entire history, which took almost a second on my machine with some entries. And indeed, test_qute_history_benchmark goes from 80us to >1s with your branch on my machine.

I guess you could implement history_iter much more efficiently using SQL though, by doing all the filtering there?

Show outdated Hide outdated qutebrowser/browser/urlmarks.py
- changed gets emitted with the name as first argument and the URL as
second argument.
- removed gets emitted with the name as argument.
The primary key for quickmarks is their *name*.

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I don't think this comment is needed anymore, the code now makes this pretty clear (primary_key='name')

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I don't think this comment is needed anymore, the code now makes this pretty clear (primary_key='name')

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

done

@rcorre

rcorre Feb 20, 2017

Collaborator

done

Show outdated Hide outdated qutebrowser/misc/sql.py
@@ -0,0 +1,178 @@
# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
# Copyright 2016 Ryan Roden-Corrent (rcorre) <ryan@rcorre.net>

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

2016-2017 😉

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

2016-2017 😉

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

Whoa, its 2017? When did that happen? Two months ago?

@rcorre

rcorre Feb 20, 2017

Collaborator

Whoa, its 2017? When did that happen? Two months ago?

Show outdated Hide outdated tests/unit/completion/test_sqlmodel.py
table = sql.SqlTable(name, ['a'], primary_key='a')
for rownum in range(rowcount):
table.insert([rownum])
model.new_category(name)

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I don't really understand this yet - how does model know about table?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

I don't really understand this yet - how does model know about table?

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

Described above, but basically everthing is referencing the same in-memory sql database. If something (e.g. the BookmarkManager creates a table named "Bookmarks", then something else (e.g. a completion model) can access that table given nothing more than it's name.

@rcorre

rcorre Feb 20, 2017

Collaborator

Described above, but basically everthing is referencing the same in-memory sql database. If something (e.g. the BookmarkManager creates a table named "Bookmarks", then something else (e.g. a completion model) can access that table given nothing more than it's name.

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
if sort_by:
sortstr = 'asc' if sort_order == Qt.AscendingOrder else 'desc'
querystr += ' order by {} {}'.format(sort_by, sortstr)

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Maybe assert that sort_order is None if sort_by is None to avoid mistakes?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Maybe assert that sort_order is None if sort_by is None to avoid mistakes?

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
querystr += ' and ' + where
if sort_by:
sortstr = 'asc' if sort_order == Qt.AscendingOrder else 'desc'

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Maybe assert sort_order is not None to avoid mistakes?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Maybe assert sort_order is not None to avoid mistakes?

Show outdated Hide outdated tests/unit/misc/test_sql.py
assert 'thirteen' in table
def test_index():

This comment has been minimized.

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Why test_index? Seems more like test_getitem or so to me?

@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Why test_index? Seems more like test_getitem or so to me?

This comment has been minimized.

@rcorre

rcorre Feb 20, 2017

Collaborator

oh, yeah. I keep thinking of it as the 'indexing operator'

@rcorre

rcorre Feb 20, 2017

Collaborator

oh, yeah. I keep thinking of it as the 'indexing operator'

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 20, 2017

Collaborator

Some more thoughts after some playing with it:

  • Reading my history with 240k items takes about a minute with full CPU load, on an i7 with NVMe SSD. We definitely need to switch to a sqlite database on-disk I'm afraid.
  • The newest items I can see in the :open completion is from 2015-12-02, so something seems to be going wrong there. Some newer items from 2015-12-03 follow. It looks like they're not ordered? Running SQL query: "select url, title, strftime('%Y-%m-%d', atime, 'unixepoch') from History where (url like ? escape '\' or title like ? escape '\') and not redirect limit 999999999999999999"
  • After those items, suddenly I only see blank items.
  • When pressing shift-tab I get to the last actual item, but when pressing tab again I iterate over those empty lines.
  • When increasing web-history-max-items to something very high (so I get all my history) and restarting, it seems very fluent, filtering my entire history. Which is awesome!
  • (edit) the flake8 failure is now fixed in master
Collaborator

The-Compiler commented Feb 20, 2017

Some more thoughts after some playing with it:

  • Reading my history with 240k items takes about a minute with full CPU load, on an i7 with NVMe SSD. We definitely need to switch to a sqlite database on-disk I'm afraid.
  • The newest items I can see in the :open completion is from 2015-12-02, so something seems to be going wrong there. Some newer items from 2015-12-03 follow. It looks like they're not ordered? Running SQL query: "select url, title, strftime('%Y-%m-%d', atime, 'unixepoch') from History where (url like ? escape '\' or title like ? escape '\') and not redirect limit 999999999999999999"
  • After those items, suddenly I only see blank items.
  • When pressing shift-tab I get to the last actual item, but when pressing tab again I iterate over those empty lines.
  • When increasing web-history-max-items to something very high (so I get all my history) and restarting, it seems very fluent, filtering my entire history. Which is awesome!
  • (edit) the flake8 failure is now fixed in master
@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 20, 2017

Collaborator

Whew! Thanks for all the great comments and giving it a test drive! Those empty items sound like I might've done something wrong with canFetchMore, and it does sound like something is off with sorting.

Collaborator

rcorre commented Feb 20, 2017

Whew! Thanks for all the great comments and giving it a test drive! Those empty items sound like I might've done something wrong with canFetchMore, and it does sound like something is off with sorting.

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Feb 21, 2017

Collaborator

it does sound like something is off with sorting.

doh

Nonetheless, loading takes too long, and since it reads the history file from top to bottom, older items are read first. If you press o several times during startup, you will see newer items at the top each time. on-disk database it is. It will be much simpler code-wise to have history, bookmarks, and quickmarks all stored in one database (.local/share/qutebrowser/db.sqlite). Getting rid of plaintext quick/bookmarks might be more of a usability hit than history though. I'm just not sure how many people actually rely on that.

Collaborator

rcorre commented Feb 21, 2017

it does sound like something is off with sorting.

doh

Nonetheless, loading takes too long, and since it reads the history file from top to bottom, older items are read first. If you press o several times during startup, you will see newer items at the top each time. on-disk database it is. It will be much simpler code-wise to have history, bookmarks, and quickmarks all stored in one database (.local/share/qutebrowser/db.sqlite). Getting rid of plaintext quick/bookmarks might be more of a usability hit than history though. I'm just not sure how many people actually rely on that.

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
querystr += ' and ' + where
if sort_by:
assert sort_order == 'asc' or sort_order == 'desc'

This comment has been minimized.

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

assert sort_order in ['asc', 'desc'], sort_order (the , sort_order is to print that value if the assert fails)

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

assert sort_order in ['asc', 'desc'], sort_order (the , sort_order is to print that value if the assert fails)

Show outdated Hide outdated qutebrowser/completion/models/sqlmodel.py
class _SqlCompletionCategory(QSqlQueryModel):
"""Wraps a SqlQuery for use as a completion category."""

This comment has been minimized.

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

Add a blank line before the docstring for consistency with other classes.

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

Add a blank line before the docstring for consistency with other classes.

Show outdated Hide outdated tests/unit/browser/webkit/test_history.py
try:
from PyQt5.QtWebKit import QWebHistoryInterface
QWebHistoryInterface.setDefaultInterface(None)
except:

This comment has been minimized.

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

except ImportError:?

@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

except ImportError:?

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 21, 2017

Collaborator

Some more random things in addition to the small things I commented above:

  • test_url_completion seems to fail now
  • pylint: test_sqlmodel: W: 23, 0: Unused Qt imported from PyQt5.QtCore (unused-import)
  • How does the error look to the user when sqlite was not found? Kinda hard to tell from the CI output with --no-err-windows.
  • I opened #2340 and mentioned it on the ML to try and find out what impact using sqlite on-disk for everything would have. I like the idea though.
Collaborator

The-Compiler commented Feb 21, 2017

Some more random things in addition to the small things I commented above:

  • test_url_completion seems to fail now
  • pylint: test_sqlmodel: W: 23, 0: Unused Qt imported from PyQt5.QtCore (unused-import)
  • How does the error look to the user when sqlite was not found? Kinda hard to tell from the CI output with --no-err-windows.
  • I opened #2340 and mentioned it on the ML to try and find out what impact using sqlite on-disk for everything would have. I like the idea though.
@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 22, 2017

Collaborator

I did a proof-of-concept hack to batch history insert queries:

diff --git a/qutebrowser/browser/history.py b/qutebrowser/browser/history.py
index 7cf0ee9b0..1d94a1967 100644
--- a/qutebrowser/browser/history.py
+++ b/qutebrowser/browser/history.py
@@ -144,6 +144,7 @@ class WebHistory(sql.SqlTable):
         self._temp_history = []
         self._new_history = []
         self._saved_count = 0
+        self._sql_data = [[], [], [], []]
 
     def __repr__(self):
         return utils.get_repr(self, length=len(self))
@@ -175,8 +176,9 @@ class WebHistory(sql.SqlTable):
                 # information about previous hits change the items in
                 # old_urls to be lists or change Entry to have a
                 # list of atimes.
-                self._add_entry(entry)
+                self._add_lazy_entry(entry)
 
+        self._write_lazy()
         self._initial_read_done = True
         self.async_read_done.emit()
         objreg.get('save-manager').add_saveable(
@@ -187,6 +189,15 @@ class WebHistory(sql.SqlTable):
             self._new_history.append(entry)
         self._temp_history.clear()
 
+    def _add_lazy_entry(self, entry):
+        self._sql_data[0].append(entry.url_str())
+        self._sql_data[1].append(entry.title)
+        self._sql_data[2].append(entry.atime)
+        self._sql_data[3].append(entry.redirect)
+
+    def _write_lazy(self):
+        self.insert(self._sql_data, replace=True, batch=True)
+
     def _add_entry(self, entry):
         """Add an entry to the in-memory database."""
         self.insert([entry.url_str(), entry.title, entry.atime,
diff --git a/qutebrowser/misc/sql.py b/qutebrowser/misc/sql.py
index c9249d362..7dfc887eb 100644
--- a/qutebrowser/misc/sql.py
+++ b/qutebrowser/misc/sql.py
@@ -56,7 +56,7 @@ def version():
     return result.record().value(0)
 
 
-def run_query(querystr, values=None):
+def run_query(querystr, values=None, batch=False):
     """Run the given SQL query string on the database.
 
     Args:
@@ -68,8 +68,10 @@ def run_query(querystr, values=None):
     query.prepare(querystr)
     for val in values or []:
         query.addBindValue(val)
-    log.sql.debug('Query bindings: {}'.format(query.boundValues()))
-    if not query.exec_():
+    if not batch:
+        log.sql.debug('Query bindings: {}'.format(query.boundValues()))
+    exec_ = query.execBatch if batch else query.exec_
+    if not exec_():
         raise SqlException('Failed to exec query "{}": "{}"'.format(
                            querystr, query.lastError().text()))
     return query
@@ -160,7 +162,7 @@ class SqlTable(QObject):
             raise KeyError('No row with {} = "{}"'.format(field, value))
         self.changed.emit()
 
-    def insert(self, values, replace=False):
+    def insert(self, values, replace=False, batch=False):
         """Append a row to the table.
 
         Args:
@@ -170,7 +172,7 @@ class SqlTable(QObject):
         cmd = "REPLACE" if replace else "INSERT"
         paramstr = ','.join(['?'] * len(values))
         run_query("{} INTO {} values({})".format(cmd, self._name, paramstr),
-            values)
+                  values, batch=batch)
         self.changed.emit()
 
     def delete_all(self):

This brings us down from 55s to 6.7s to process my history (4.6s on master).

I still think we should have the history as SQL on disk, after some points raised in #2340 I'm not so sure about the rest anymore. Either way, we should definitely do this (even if it's only for the initial import for conversion, for the history), and have a benchmark for this 😉


(feel free to continue work on benchmarks and stuff first - I'm just dumping stuff here, and whenever you feel like things are ready I'm happy to go over older comments again and check nothing slipped through the cracks - though we might surpass #1891 and #962 with the number of comments here eventually 😆)

Collaborator

The-Compiler commented Feb 22, 2017

I did a proof-of-concept hack to batch history insert queries:

diff --git a/qutebrowser/browser/history.py b/qutebrowser/browser/history.py
index 7cf0ee9b0..1d94a1967 100644
--- a/qutebrowser/browser/history.py
+++ b/qutebrowser/browser/history.py
@@ -144,6 +144,7 @@ class WebHistory(sql.SqlTable):
         self._temp_history = []
         self._new_history = []
         self._saved_count = 0
+        self._sql_data = [[], [], [], []]
 
     def __repr__(self):
         return utils.get_repr(self, length=len(self))
@@ -175,8 +176,9 @@ class WebHistory(sql.SqlTable):
                 # information about previous hits change the items in
                 # old_urls to be lists or change Entry to have a
                 # list of atimes.
-                self._add_entry(entry)
+                self._add_lazy_entry(entry)
 
+        self._write_lazy()
         self._initial_read_done = True
         self.async_read_done.emit()
         objreg.get('save-manager').add_saveable(
@@ -187,6 +189,15 @@ class WebHistory(sql.SqlTable):
             self._new_history.append(entry)
         self._temp_history.clear()
 
+    def _add_lazy_entry(self, entry):
+        self._sql_data[0].append(entry.url_str())
+        self._sql_data[1].append(entry.title)
+        self._sql_data[2].append(entry.atime)
+        self._sql_data[3].append(entry.redirect)
+
+    def _write_lazy(self):
+        self.insert(self._sql_data, replace=True, batch=True)
+
     def _add_entry(self, entry):
         """Add an entry to the in-memory database."""
         self.insert([entry.url_str(), entry.title, entry.atime,
diff --git a/qutebrowser/misc/sql.py b/qutebrowser/misc/sql.py
index c9249d362..7dfc887eb 100644
--- a/qutebrowser/misc/sql.py
+++ b/qutebrowser/misc/sql.py
@@ -56,7 +56,7 @@ def version():
     return result.record().value(0)
 
 
-def run_query(querystr, values=None):
+def run_query(querystr, values=None, batch=False):
     """Run the given SQL query string on the database.
 
     Args:
@@ -68,8 +68,10 @@ def run_query(querystr, values=None):
     query.prepare(querystr)
     for val in values or []:
         query.addBindValue(val)
-    log.sql.debug('Query bindings: {}'.format(query.boundValues()))
-    if not query.exec_():
+    if not batch:
+        log.sql.debug('Query bindings: {}'.format(query.boundValues()))
+    exec_ = query.execBatch if batch else query.exec_
+    if not exec_():
         raise SqlException('Failed to exec query "{}": "{}"'.format(
                            querystr, query.lastError().text()))
     return query
@@ -160,7 +162,7 @@ class SqlTable(QObject):
             raise KeyError('No row with {} = "{}"'.format(field, value))
         self.changed.emit()
 
-    def insert(self, values, replace=False):
+    def insert(self, values, replace=False, batch=False):
         """Append a row to the table.
 
         Args:
@@ -170,7 +172,7 @@ class SqlTable(QObject):
         cmd = "REPLACE" if replace else "INSERT"
         paramstr = ','.join(['?'] * len(values))
         run_query("{} INTO {} values({})".format(cmd, self._name, paramstr),
-            values)
+                  values, batch=batch)
         self.changed.emit()
 
     def delete_all(self):

This brings us down from 55s to 6.7s to process my history (4.6s on master).

I still think we should have the history as SQL on disk, after some points raised in #2340 I'm not so sure about the rest anymore. Either way, we should definitely do this (even if it's only for the initial import for conversion, for the history), and have a benchmark for this 😉


(feel free to continue work on benchmarks and stuff first - I'm just dumping stuff here, and whenever you feel like things are ready I'm happy to go over older comments again and check nothing slipped through the cracks - though we might surpass #1891 and #962 with the number of comments here eventually 😆)

Show outdated Hide outdated qutebrowser/completion/models/completionmodel.py
"""Override to forward the call to the categories."""
cat = self._cat_from_idx(parent)
if cat:
return cat.canFetchMore(parent)

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

In 4be3eeb you added parent here - that seems odd. Wouldn't the underlying category model now get a parent which belongs to another model (the CompletionModel)? Shouldn't it never get a parent, as from the category model's point of view, all items are toplevel items (i.e. it's a table)?

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

In 4be3eeb you added parent here - that seems odd. Wouldn't the underlying category model now get a parent which belongs to another model (the CompletionModel)? Shouldn't it never get a parent, as from the category model's point of view, all items are toplevel items (i.e. it's a table)?

This comment has been minimized.

@rcorre

rcorre Feb 23, 2017

Collaborator

I shoulda stuck a WIP on this commit message, that's why there are so many TODO's 😁

Good catch on this though, I should be passing an invalid index. That's what QSqlQueryModel expects, and the list models should always return false.

@rcorre

rcorre Feb 23, 2017

Collaborator

I shoulda stuck a WIP on this commit message, that's why there are so many TODO's 😁

Good catch on this though, I should be passing an invalid index. That's what QSqlQueryModel expects, and the list models should always return false.

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

No worries - I primarily look through the diffs to keep myself updated, and then just comment on stuff going through my mind to make sure nothing gets lost/forgotten. If you know about those already, just ignore me 😉

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

No worries - I primarily look through the diffs to keep myself updated, and then just comment on stuff going through my mind to make sure nothing gets lost/forgotten. If you know about those already, just ignore me 😉

Show outdated Hide outdated qutebrowser/completion/models/listcategory.py
super().__init__(parent)
self.name = name
# self.setColumnCount(3) TODO needed?
# TODO: batch insert?

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Doesn't seem possible - but probably doesn't matter much here.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Doesn't seem possible - but probably doesn't matter much here.

This comment has been minimized.

@rcorre

rcorre Feb 24, 2017

Collaborator

for some reason I was thinking insertRows was more efficient (so it only calls beginInsertRows once, but looking at it again I think that's just to prevent firing a bunch of changed signals, which aren't hooked up at this point.

@rcorre

rcorre Feb 24, 2017

Collaborator

for some reason I was thinking insertRows was more efficient (so it only calls beginInsertRows once, but looking at it again I think that's just to prevent firing a bunch of changed signals, which aren't hooked up at this point.

This comment has been minimized.

@The-Compiler

The-Compiler Feb 24, 2017

Collaborator

I think I missed insertRows before! However, it looks like it's only there for creating empty rows, i.e. without any existing QStandardItem.

@The-Compiler

The-Compiler Feb 24, 2017

Collaborator

I think I missed insertRows before! However, it looks like it's only there for creating empty rows, i.e. without any existing QStandardItem.

Show outdated Hide outdated qutebrowser/completion/models/listcategory.py
def __init__(self, name, items, parent=None):
super().__init__(parent)
self.name = name
# self.setColumnCount(3) TODO needed?

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

I don't think so, looks like QStandardItemModel is clever enough to increase it by itself when doing appendRow.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

I don't think so, looks like QStandardItemModel is clever enough to increase it by itself when doing appendRow.

Show outdated Hide outdated qutebrowser/completion/models/listcategory.py
self.name = name
# self.setColumnCount(3) TODO needed?
# TODO: batch insert?
# TODO: can I just insert a tuple instead of a list?

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Since the Qt API wants a list, I'd just keep it as a list.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Since the Qt API wants a list, I'd just keep it as a list.

Show outdated Hide outdated qutebrowser/completion/models/listcategory.py
Module attributes:
Role: An enum of user defined model roles.
"""

This comment has been minimized.

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Outdated docstring (though I kinda need to think about making module-docstrings optional, they're kinda pointless with single-class modules)

@The-Compiler

The-Compiler Feb 23, 2017

Collaborator

Outdated docstring (though I kinda need to think about making module-docstrings optional, they're kinda pointless with single-class modules)

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 17, 2017

Collaborator

Reviewed 6 of 8 files at r17.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from Reviewable

Collaborator

The-Compiler commented Jul 17, 2017

Reviewed 6 of 8 files at r17.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 17, 2017

Collaborator

Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


scripts/dev/check_coverage.py, line 160 at r17 (raw file):

        'completion/models/urlmodel.py'),
    ('tests/unit/completion/test_sqlcategory.py',
        'completion/models/sqlcategory.py'),

This should be histcategory now.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):
Hmmm... this (and test_url_completion in test_models.py) seems to pass on Travis, but fail on my machine with `tox -e py36 -- tests/unit/completion:

_______________________________________________________________ test_sorting[-1-before0-after0] _______________________________________________________________

max_items = -1, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a288>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a1f8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7d68) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7d68) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000195 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[3-before1-after1] ________________________________________________________________

max_items = 3, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a0d8>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a048>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b646ef28) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b646ef28) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 3}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000186 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[2-before2-after2] ________________________________________________________________

max_items = 2, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')], model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>
hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b6756f78>, config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b6756ee8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7ba8) with depth 0 (2 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7ba8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 2}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000184 seconds.

tests/unit/completion/test_histcategory.py ..
tests/unit/completion/test_listcategory.py ......
tests/unit/completion/test_models.py ....F

_____________________________________________________________________ test_url_completion _____________________________________________________________________

qtmodeltester = <pytestqt.modeltest.ModelTester object at 0x7fa4b68edf60>, web_history_populated = <qutebrowser.browser.history.WebHistory length=3>
quickmarks = <helpers.stubs.QuickmarkManagerStub object at 0x7fa4b6756948>, bookmarks = <helpers.stubs.BookmarkManagerStub object at 0x7fa4b67569d8>

    def test_url_completion(qtmodeltester, web_history_populated,
                            quickmarks, bookmarks):
        """Test the results of url completion.
    
        Verify that:
            - quickmarks, bookmarks, and urls are included
            - entries are sorted by access time
            - only the most recent entry is included for each url
        """
        model = urlmodel.url()
        model.set_pattern('')
        qtmodeltester.data_display_may_return_none = True
        qtmodeltester.check(model)
    
        _check_completions(model, {
            "Quickmarks": [
                ('https://duckduckgo.com', 'ddg', None),
                ('https://wiki.archlinux.org', 'aw', None),
                ('https://wikipedia.org', 'wiki', None),
            ],
            "Bookmarks": [
                ('http://qutebrowser.org', 'qutebrowser | qutebrowser', None),
                ('https://github.com', 'GitHub', None),
                ('https://python.org', 'Welcome to Python.org', None),
            ],
            "History": [
                ('https://github.com', 'https://github.com', '2016-05-01'),
                ('https://python.org', 'Welcome to Python.org', '2016-03-08'),
>               ('http://qutebrowser.org', 'qutebrowser', '2015-09-05'),
            ],
        })

tests/unit/completion/test_models.py:299: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = <qutebrowser.completion.models.completionmodel.CompletionModel object at 0x7fa4b6756b88>
expected = {'Bookmarks': [('http://qutebrowser.org', 'qutebrowser | qutebrowser', None), ('https://github.com', 'GitHub', None), ...://duckduckgo.com', 'ddg', None), ('https://wiki.archlinux.org', 'aw', None), ('https://wikipedia.org', 'wiki', None)]}

    def _check_completions(model, expected):
        """Check that a model contains the expected items in any order.
    
        Args:
            expected: A dict of form
                {
                    CategoryName: [(name, desc, misc), ...],
                    CategoryName: [(name, desc, misc), ...],
                    ...
                }
        """
        actual = {}
        assert model.rowCount() == len(expected)
        for i in range(0, model.rowCount()):
            catidx = model.index(i, 0)
            catname = model.data(catidx)
            actual[catname] = []
            for j in range(model.rowCount(catidx)):
                name = model.data(model.index(j, 0, parent=catidx))
                desc = model.data(model.index(j, 1, parent=catidx))
                misc = model.data(model.index(j, 2, parent=catidx))
                actual[catname].append((name, desc, misc))
>       assert actual == expected
E       AssertionError: assert {'Bookmarks':...wiki', None)]} == {'Bookmarks': ...wiki', None)]}
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'History': [('https://github.com', 'https://github.com', '2016-04-30'), ('https://python.org', 'Welcome to Python.org', '2016-03-07'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-04')]} != {'History': [('https://github.com', 'https://github.com', '2016-05-01'), ('https://python.org', 'Welcome to Python.org', '2016-03-08'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-05')]}
E         Use -v to get the full diff

tests/unit/completion/test_models.py:56: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime > :earliest and atime <= :latest ORDER BY atime desc"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime <= :latest ORDER BY atime desc limit :limit offset :offset"
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1441404000, ':redirect': False, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1441404000, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1457391600, ':redirect': False, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1457391600, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1462053600, ':redirect': False, ':title': 'https://github.com', ':url': 'https://github.com'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1462053600, ':title': 'https://github.com', ':url': 'https://github.com'}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b642c0b8) with depth 0 (3 rows, 3 columns)
modeltest: 0/0 'Quickmarks' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 0/0 'Quickmarks' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 0/0 'Quickmarks' (0x7fa4b642c908) done
modeltest: 1/0 'Bookmarks' (0x7fa4b642c668) has 3 children
modeltest: Checking children of 1/0 'Bookmarks' (0x7fa4b642c668) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 1/0 'Bookmarks' (0x7fa4b642c668) done
modeltest: 2/0 'History' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 2/0 'History' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 2/0 'History' (0x7fa4b642c908) done
modeltest: Children check for <invalid> (0x7fa4b642c0b8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
completionmodel.py         180 DEBUG    Setting completion pattern ''
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000194 seconds.

Comments from Reviewable

Collaborator

The-Compiler commented Jul 17, 2017

Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


scripts/dev/check_coverage.py, line 160 at r17 (raw file):

        'completion/models/urlmodel.py'),
    ('tests/unit/completion/test_sqlcategory.py',
        'completion/models/sqlcategory.py'),

This should be histcategory now.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):
Hmmm... this (and test_url_completion in test_models.py) seems to pass on Travis, but fail on my machine with `tox -e py36 -- tests/unit/completion:

_______________________________________________________________ test_sorting[-1-before0-after0] _______________________________________________________________

max_items = -1, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a288>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a1f8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7d68) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7d68) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000195 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[3-before1-after1] ________________________________________________________________

max_items = 3, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a0d8>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a048>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b646ef28) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b646ef28) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 3}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000186 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[2-before2-after2] ________________________________________________________________

max_items = 2, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')], model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>
hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b6756f78>, config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b6756ee8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7ba8) with depth 0 (2 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7ba8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 2}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000184 seconds.

tests/unit/completion/test_histcategory.py ..
tests/unit/completion/test_listcategory.py ......
tests/unit/completion/test_models.py ....F

_____________________________________________________________________ test_url_completion _____________________________________________________________________

qtmodeltester = <pytestqt.modeltest.ModelTester object at 0x7fa4b68edf60>, web_history_populated = <qutebrowser.browser.history.WebHistory length=3>
quickmarks = <helpers.stubs.QuickmarkManagerStub object at 0x7fa4b6756948>, bookmarks = <helpers.stubs.BookmarkManagerStub object at 0x7fa4b67569d8>

    def test_url_completion(qtmodeltester, web_history_populated,
                            quickmarks, bookmarks):
        """Test the results of url completion.
    
        Verify that:
            - quickmarks, bookmarks, and urls are included
            - entries are sorted by access time
            - only the most recent entry is included for each url
        """
        model = urlmodel.url()
        model.set_pattern('')
        qtmodeltester.data_display_may_return_none = True
        qtmodeltester.check(model)
    
        _check_completions(model, {
            "Quickmarks": [
                ('https://duckduckgo.com', 'ddg', None),
                ('https://wiki.archlinux.org', 'aw', None),
                ('https://wikipedia.org', 'wiki', None),
            ],
            "Bookmarks": [
                ('http://qutebrowser.org', 'qutebrowser | qutebrowser', None),
                ('https://github.com', 'GitHub', None),
                ('https://python.org', 'Welcome to Python.org', None),
            ],
            "History": [
                ('https://github.com', 'https://github.com', '2016-05-01'),
                ('https://python.org', 'Welcome to Python.org', '2016-03-08'),
>               ('http://qutebrowser.org', 'qutebrowser', '2015-09-05'),
            ],
        })

tests/unit/completion/test_models.py:299: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = <qutebrowser.completion.models.completionmodel.CompletionModel object at 0x7fa4b6756b88>
expected = {'Bookmarks': [('http://qutebrowser.org', 'qutebrowser | qutebrowser', None), ('https://github.com', 'GitHub', None), ...://duckduckgo.com', 'ddg', None), ('https://wiki.archlinux.org', 'aw', None), ('https://wikipedia.org', 'wiki', None)]}

    def _check_completions(model, expected):
        """Check that a model contains the expected items in any order.
    
        Args:
            expected: A dict of form
                {
                    CategoryName: [(name, desc, misc), ...],
                    CategoryName: [(name, desc, misc), ...],
                    ...
                }
        """
        actual = {}
        assert model.rowCount() == len(expected)
        for i in range(0, model.rowCount()):
            catidx = model.index(i, 0)
            catname = model.data(catidx)
            actual[catname] = []
            for j in range(model.rowCount(catidx)):
                name = model.data(model.index(j, 0, parent=catidx))
                desc = model.data(model.index(j, 1, parent=catidx))
                misc = model.data(model.index(j, 2, parent=catidx))
                actual[catname].append((name, desc, misc))
>       assert actual == expected
E       AssertionError: assert {'Bookmarks':...wiki', None)]} == {'Bookmarks': ...wiki', None)]}
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'History': [('https://github.com', 'https://github.com', '2016-04-30'), ('https://python.org', 'Welcome to Python.org', '2016-03-07'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-04')]} != {'History': [('https://github.com', 'https://github.com', '2016-05-01'), ('https://python.org', 'Welcome to Python.org', '2016-03-08'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-05')]}
E         Use -v to get the full diff

tests/unit/completion/test_models.py:56: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime > :earliest and atime <= :latest ORDER BY atime desc"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime <= :latest ORDER BY atime desc limit :limit offset :offset"
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1441404000, ':redirect': False, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1441404000, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1457391600, ':redirect': False, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1457391600, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1462053600, ':redirect': False, ':title': 'https://github.com', ':url': 'https://github.com'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1462053600, ':title': 'https://github.com', ':url': 'https://github.com'}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b642c0b8) with depth 0 (3 rows, 3 columns)
modeltest: 0/0 'Quickmarks' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 0/0 'Quickmarks' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 0/0 'Quickmarks' (0x7fa4b642c908) done
modeltest: 1/0 'Bookmarks' (0x7fa4b642c668) has 3 children
modeltest: Checking children of 1/0 'Bookmarks' (0x7fa4b642c668) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 1/0 'Bookmarks' (0x7fa4b642c668) done
modeltest: 2/0 'History' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 2/0 'History' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 2/0 'History' (0x7fa4b642c908) done
modeltest: Children check for <invalid> (0x7fa4b642c0b8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
completionmodel.py         180 DEBUG    Setting completion pattern ''
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000194 seconds.

Comments from Reviewable

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 17, 2017

Collaborator

Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


scripts/dev/check_coverage.py, line 160 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This should be histcategory now.

Done.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Hmmm... this (and test_url_completion in test_models.py) seems to pass on Travis, but fail on my machine with `tox -e py36 -- tests/unit/completion:

_______________________________________________________________ test_sorting[-1-before0-after0] _______________________________________________________________

max_items = -1, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a288>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a1f8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7d68) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7d68) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000195 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[3-before1-after1] ________________________________________________________________

max_items = 3, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a0d8>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a048>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b646ef28) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b646ef28) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 3}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000186 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[2-before2-after2] ________________________________________________________________

max_items = 2, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')], model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>
hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b6756f78>, config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b6756ee8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7ba8) with depth 0 (2 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7ba8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 2}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000184 seconds.

tests/unit/completion/test_histcategory.py ..
tests/unit/completion/test_listcategory.py ......
tests/unit/completion/test_models.py ....F

_____________________________________________________________________ test_url_completion _____________________________________________________________________

qtmodeltester = <pytestqt.modeltest.ModelTester object at 0x7fa4b68edf60>, web_history_populated = <qutebrowser.browser.history.WebHistory length=3>
quickmarks = <helpers.stubs.QuickmarkManagerStub object at 0x7fa4b6756948>, bookmarks = <helpers.stubs.BookmarkManagerStub object at 0x7fa4b67569d8>

    def test_url_completion(qtmodeltester, web_history_populated,
                            quickmarks, bookmarks):
        """Test the results of url completion.
    
        Verify that:
            - quickmarks, bookmarks, and urls are included
            - entries are sorted by access time
            - only the most recent entry is included for each url
        """
        model = urlmodel.url()
        model.set_pattern('')
        qtmodeltester.data_display_may_return_none = True
        qtmodeltester.check(model)
    
        _check_completions(model, {
            "Quickmarks": [
                ('https://duckduckgo.com', 'ddg', None),
                ('https://wiki.archlinux.org', 'aw', None),
                ('https://wikipedia.org', 'wiki', None),
            ],
            "Bookmarks": [
                ('http://qutebrowser.org', 'qutebrowser | qutebrowser', None),
                ('https://github.com', 'GitHub', None),
                ('https://python.org', 'Welcome to Python.org', None),
            ],
            "History": [
                ('https://github.com', 'https://github.com', '2016-05-01'),
                ('https://python.org', 'Welcome to Python.org', '2016-03-08'),
>               ('http://qutebrowser.org', 'qutebrowser', '2015-09-05'),
            ],
        })

tests/unit/completion/test_models.py:299: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = <qutebrowser.completion.models.completionmodel.CompletionModel object at 0x7fa4b6756b88>
expected = {'Bookmarks': [('http://qutebrowser.org', 'qutebrowser | qutebrowser', None), ('https://github.com', 'GitHub', None), ...://duckduckgo.com', 'ddg', None), ('https://wiki.archlinux.org', 'aw', None), ('https://wikipedia.org', 'wiki', None)]}

    def _check_completions(model, expected):
        """Check that a model contains the expected items in any order.
    
        Args:
            expected: A dict of form
                {
                    CategoryName: [(name, desc, misc), ...],
                    CategoryName: [(name, desc, misc), ...],
                    ...
                }
        """
        actual = {}
        assert model.rowCount() == len(expected)
        for i in range(0, model.rowCount()):
            catidx = model.index(i, 0)
            catname = model.data(catidx)
            actual[catname] = []
            for j in range(model.rowCount(catidx)):
                name = model.data(model.index(j, 0, parent=catidx))
                desc = model.data(model.index(j, 1, parent=catidx))
                misc = model.data(model.index(j, 2, parent=catidx))
                actual[catname].append((name, desc, misc))
>       assert actual == expected
E       AssertionError: assert {'Bookmarks':...wiki', None)]} == {'Bookmarks': ...wiki', None)]}
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'History': [('https://github.com', 'https://github.com', '2016-04-30'), ('https://python.org', 'Welcome to Python.org', '2016-03-07'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-04')]} != {'History': [('https://github.com', 'https://github.com', '2016-05-01'), ('https://python.org', 'Welcome to Python.org', '2016-03-08'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-05')]}
E         Use -v to get the full diff

tests/unit/completion/test_models.py:56: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime > :earliest and atime <= :latest ORDER BY atime desc"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime <= :latest ORDER BY atime desc limit :limit offset :offset"
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1441404000, ':redirect': False, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1441404000, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1457391600, ':redirect': False, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1457391600, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1462053600, ':redirect': False, ':title': 'https://github.com', ':url': 'https://github.com'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1462053600, ':title': 'https://github.com', ':url': 'https://github.com'}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b642c0b8) with depth 0 (3 rows, 3 columns)
modeltest: 0/0 'Quickmarks' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 0/0 'Quickmarks' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 0/0 'Quickmarks' (0x7fa4b642c908) done
modeltest: 1/0 'Bookmarks' (0x7fa4b642c668) has 3 children
modeltest: Checking children of 1/0 'Bookmarks' (0x7fa4b642c668) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 1/0 'Bookmarks' (0x7fa4b642c668) done
modeltest: 2/0 'History' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 2/0 'History' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 2/0 'History' (0x7fa4b642c908) done
modeltest: Children check for <invalid> (0x7fa4b642c0b8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
completionmodel.py         180 DEBUG    Setting completion pattern ''
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000194 seconds.

hmm ... I wonder if it is a timezone issue


Comments from Reviewable

Collaborator

rcorre commented Jul 17, 2017

Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


scripts/dev/check_coverage.py, line 160 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This should be histcategory now.

Done.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Hmmm... this (and test_url_completion in test_models.py) seems to pass on Travis, but fail on my machine with `tox -e py36 -- tests/unit/completion:

_______________________________________________________________ test_sorting[-1-before0-after0] _______________________________________________________________

max_items = -1, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a288>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a1f8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b666fa90>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7d68) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7d68) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000195 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[3-before1-after1] ________________________________________________________________

max_items = 3, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]
model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b644a0d8>
config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b644a048>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4b688aa58>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16'), ('a', 'a', '2017-04-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b646ef28) with depth 0 (3 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b646ef28) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 3}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1492293600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000186 seconds.

tests/unit/completion/test_histcategory.py F

_______________________________________________________________ test_sorting[2-before2-after2] ________________________________________________________________

max_items = 2, before = [('a', 'a', '2017-04-16'), ('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]
after = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')], model_validator = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>
hist = <qutebrowser.misc.sql.SqlTable object at 0x7fa4b6756f78>, config_stub = <helpers.stubs.ConfigStub object at 0x7fa4b6756ee8>

    @pytest.mark.parametrize('max_items, before, after', [
        (-1, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (3, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
            ('a', 'a', '2017-04-16'),
        ]),
        (2, [
            ('a', 'a', '2017-04-16'),
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ], [
            ('b', 'b', '2017-06-16'),
            ('c', 'c', '2017-05-16'),
        ])
    ])
    def test_sorting(max_items, before, after, model_validator, hist, config_stub):
        """Validate the filtering and sorting results of set_pattern."""
        config_stub.data['completion']['web-history-max-items'] = max_items
        for url, title, atime in before:
            timestamp = time.mktime(time.strptime(atime, '%Y-%m-%d'))
            hist.insert({'url': url, 'title': title, 'last_atime': timestamp})
        cat = histcategory.HistoryCategory()
        model_validator.set_model(cat)
        cat.set_pattern('')
>       model_validator.validate(after)

tests/unit/completion/test_histcategory.py:131: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <helpers.fixtures.ModelValidator object at 0x7fa4c65fd1d0>, expected = [('b', 'b', '2017-06-16'), ('c', 'c', '2017-05-16')]

    def validate(self, expected):
        assert self._model.rowCount() == len(expected)
        for row, items in enumerate(expected):
            for col, item in enumerate(items):
>               assert self._model.data(self._model.index(row, col)) == item
E               AssertionError: assert '2017-06-15' == '2017-06-16'
E                 - 2017-06-15
E                 ?          ^
E                 + 2017-06-16
E                 ?          ^

tests/helpers/fixtures.py:504: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url , title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b65e7ba8) with depth 0 (2 rows, 3 columns)
modeltest: Children check for <invalid> (0x7fa4b65e7ba8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1492293600.0, ':title': 'a', ':url': 'a'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1497564000.0, ':title': 'b', ':url': 'b'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1494885600.0, ':title': 'c', ':url': 'c'}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                      99 DEBUG    Running SQL query: "SELECT min(last_atime) FROM (SELECT last_atime FROM CompletionHistory ORDER BY last_atime DESC LIMIT :limit)"
sql.py                     102 DEBUG    query bindings: {':limit': 2}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\') AND last_atime >= 1494885600.0 ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000184 seconds.

tests/unit/completion/test_histcategory.py ..
tests/unit/completion/test_listcategory.py ......
tests/unit/completion/test_models.py ....F

_____________________________________________________________________ test_url_completion _____________________________________________________________________

qtmodeltester = <pytestqt.modeltest.ModelTester object at 0x7fa4b68edf60>, web_history_populated = <qutebrowser.browser.history.WebHistory length=3>
quickmarks = <helpers.stubs.QuickmarkManagerStub object at 0x7fa4b6756948>, bookmarks = <helpers.stubs.BookmarkManagerStub object at 0x7fa4b67569d8>

    def test_url_completion(qtmodeltester, web_history_populated,
                            quickmarks, bookmarks):
        """Test the results of url completion.
    
        Verify that:
            - quickmarks, bookmarks, and urls are included
            - entries are sorted by access time
            - only the most recent entry is included for each url
        """
        model = urlmodel.url()
        model.set_pattern('')
        qtmodeltester.data_display_may_return_none = True
        qtmodeltester.check(model)
    
        _check_completions(model, {
            "Quickmarks": [
                ('https://duckduckgo.com', 'ddg', None),
                ('https://wiki.archlinux.org', 'aw', None),
                ('https://wikipedia.org', 'wiki', None),
            ],
            "Bookmarks": [
                ('http://qutebrowser.org', 'qutebrowser | qutebrowser', None),
                ('https://github.com', 'GitHub', None),
                ('https://python.org', 'Welcome to Python.org', None),
            ],
            "History": [
                ('https://github.com', 'https://github.com', '2016-05-01'),
                ('https://python.org', 'Welcome to Python.org', '2016-03-08'),
>               ('http://qutebrowser.org', 'qutebrowser', '2015-09-05'),
            ],
        })

tests/unit/completion/test_models.py:299: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = <qutebrowser.completion.models.completionmodel.CompletionModel object at 0x7fa4b6756b88>
expected = {'Bookmarks': [('http://qutebrowser.org', 'qutebrowser | qutebrowser', None), ('https://github.com', 'GitHub', None), ...://duckduckgo.com', 'ddg', None), ('https://wiki.archlinux.org', 'aw', None), ('https://wikipedia.org', 'wiki', None)]}

    def _check_completions(model, expected):
        """Check that a model contains the expected items in any order.
    
        Args:
            expected: A dict of form
                {
                    CategoryName: [(name, desc, misc), ...],
                    CategoryName: [(name, desc, misc), ...],
                    ...
                }
        """
        actual = {}
        assert model.rowCount() == len(expected)
        for i in range(0, model.rowCount()):
            catidx = model.index(i, 0)
            catname = model.data(catidx)
            actual[catname] = []
            for j in range(model.rowCount(catidx)):
                name = model.data(model.index(j, 0, parent=catidx))
                desc = model.data(model.index(j, 1, parent=catidx))
                misc = model.data(model.index(j, 2, parent=catidx))
                actual[catname].append((name, desc, misc))
>       assert actual == expected
E       AssertionError: assert {'Bookmarks':...wiki', None)]} == {'Bookmarks': ...wiki', None)]}
E         Omitting 2 identical items, use -vv to show
E         Differing items:
E         {'History': [('https://github.com', 'https://github.com', '2016-04-30'), ('https://python.org', 'Welcome to Python.org', '2016-03-07'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-04')]} != {'History': [('https://github.com', 'https://github.com', '2016-05-01'), ('https://python.org', 'Welcome to Python.org', '2016-03-08'), ('http://qutebrowser.org', 'qutebrowser', '2015-09-05')]}
E         Use -v to get the full diff

tests/unit/completion/test_models.py:56: AssertionError
--------------------------------------------------------------------- Captured log setup ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS History (url , title , atime , redirect )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                      99 DEBUG    Running SQL query: "CREATE TABLE IF NOT EXISTS CompletionHistory (url PRIMARY KEY, title , last_atime )"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS CompletionHistoryAtimeIndex ON CompletionHistory (last_atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryIndex ON History (url)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                      99 DEBUG    Running SQL query: "CREATE INDEX IF NOT EXISTS HistoryAtimeIndex ON History (atime)"
sql.py                     102 DEBUG    query bindings: {}
sql.py                      80 DEBUG    Preparing SQL query: "SELECT EXISTS(SELECT * FROM History WHERE url = :val)"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime > :earliest and atime <= :latest ORDER BY atime desc"
sql.py                      80 DEBUG    Preparing SQL query: "SELECT * FROM History where not redirect and not url like "qute://%" and atime <= :latest ORDER BY atime desc limit :limit offset :offset"
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1441404000, ':redirect': False, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1441404000, ':title': 'qutebrowser', ':url': 'http://qutebrowser.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1457391600, ':redirect': False, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1457391600, ':title': 'Welcome to Python.org', ':url': 'https://python.org'}
sql.py                      80 DEBUG    Preparing SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                      99 DEBUG    Running SQL query: "INSERT INTO History (url, title, atime, redirect) values(:url, :title, :atime, :redirect)"
sql.py                     102 DEBUG    query bindings: {':atime': 1462053600, ':redirect': False, ':title': 'https://github.com', ':url': 'https://github.com'}
sql.py                      80 DEBUG    Preparing SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                      99 DEBUG    Running SQL query: "REPLACE INTO CompletionHistory (url, title, last_atime) values(:url, :title, :last_atime)"
sql.py                     102 DEBUG    query bindings: {':last_atime': 1462053600, ':title': 'https://github.com', ':url': 'https://github.com'}
-------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
modeltest: Checking children of <invalid> (0x7fa4b642c0b8) with depth 0 (3 rows, 3 columns)
modeltest: 0/0 'Quickmarks' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 0/0 'Quickmarks' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 0/0 'Quickmarks' (0x7fa4b642c908) done
modeltest: 1/0 'Bookmarks' (0x7fa4b642c668) has 3 children
modeltest: Checking children of 1/0 'Bookmarks' (0x7fa4b642c668) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 1/0 'Bookmarks' (0x7fa4b642c668) done
modeltest: 2/0 'History' (0x7fa4b642c908) has 3 children
modeltest: Checking children of 2/0 'History' (0x7fa4b642c908) with depth 1 (3 rows, 3 columns)
modeltest: Children check for 2/0 'History' (0x7fa4b642c908) done
modeltest: Children check for <invalid> (0x7fa4b642c0b8) done
---------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------
sql.py                      80 DEBUG    Preparing SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
completionmodel.py         180 DEBUG    Setting completion pattern ''
sql.py                      99 DEBUG    Running SQL query: "SELECT url, title, strftime('%Y-%m-%d', last_atime, 'unixepoch') FROM CompletionHistory WHERE (url LIKE :pat escape '\' or title LIKE :pat escape '\')  ORDER BY last_atime DESC"
sql.py                     102 DEBUG    query bindings: {':pat': '%%'}
debug.py                   264 DEBUG    Running completion query took 0.000194 seconds.

hmm ... I wonder if it is a timezone issue


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 17, 2017

Collaborator

Reviewed 1 of 1 files at r18.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

hmm ... I wonder if it is a timezone issue

Looks like it! They pass after a timedatectl set-timezone UTC and fail again after timedatectl set-timezone Europe/Zurich. I'll have to admit that I know nothing about timezones in Python though, so no idea what the proper fix would be...


Comments from Reviewable

Collaborator

The-Compiler commented Jul 17, 2017

Reviewed 1 of 1 files at r18.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

hmm ... I wonder if it is a timezone issue

Looks like it! They pass after a timedatectl set-timezone UTC and fail again after timedatectl set-timezone Europe/Zurich. I'll have to admit that I know nothing about timezones in Python though, so no idea what the proper fix would be...


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 17, 2017

Collaborator

Timezones are painful... I took a quick look, and I'd propose the following:

  • Internally, all timestamps are stored and treated as UTC
  • Only when showing the timestamp to the user (in the completion) they get converted to local time

I commented on the parts doing time handling I saw, not sure if it's everything though. Also not 100% sure about this being the best solution, maybe having everything in local time is easier.


Review status: all files reviewed at latest revision, 9 unresolved discussions.


qutebrowser/browser/history.py, line 146 at r18 (raw file):

        self.add_url(url, title)

    def add_url(self, url, title="", *, redirect=False, atime=None):

FYI: the atime argument is only used in tests, so that bit should be fine.


qutebrowser/browser/history.py, line 161 at r18 (raw file):

            return

        atime = int(atime) if (atime is not None) else int(time.time())

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?


qutebrowser/browser/history.py, line 211 at r18 (raw file):

        redirect = 'r' in flags
        return (url, title, int(atime), redirect)

Since the old history used local times (I think? It used time.time() at least...), this should probably convert to UTC.


qutebrowser/browser/qutescheme.py, line 221 at r18 (raw file):

        try:
            start_time = QUrlQuery(url).queryItemValue("start_time")
            start_time = float(start_time) if start_time else time.time()

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

        # replace ' to avoid breaking the query
        timefmt = "strftime('{}', last_atime, 'unixepoch')".format(

You should probably add 'localtime' as modifier (sqlite docs) so here it gets converted to local time.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

        // Find/create table
        var tableId = ["hist", date.getDate(), date.getMonth(),
            date.getYear()].join("-");

Those are all local time - maybe the IDs should be UTC instead?


qutebrowser/javascript/history.js, line 178 at r18 (raw file):

            var item = history[i];
            // python's time.time returns seconds, but js Date expects ms
            var currentItemDate = new Date(item.time * 1000);

FYI: This seems to take UTC.


qutebrowser/javascript/history.js, line 180 at r18 (raw file):

            var currentItemDate = new Date(item.time * 1000);
            getSessionNode(currentItemDate).appendChild(makeHistoryRow(
                item.url, item.title, currentItemDate.toLocaleTimeString()

FYI: This converts from UTC to local time.


Comments from Reviewable

Collaborator

The-Compiler commented Jul 17, 2017

Timezones are painful... I took a quick look, and I'd propose the following:

  • Internally, all timestamps are stored and treated as UTC
  • Only when showing the timestamp to the user (in the completion) they get converted to local time

I commented on the parts doing time handling I saw, not sure if it's everything though. Also not 100% sure about this being the best solution, maybe having everything in local time is easier.


Review status: all files reviewed at latest revision, 9 unresolved discussions.


qutebrowser/browser/history.py, line 146 at r18 (raw file):

        self.add_url(url, title)

    def add_url(self, url, title="", *, redirect=False, atime=None):

FYI: the atime argument is only used in tests, so that bit should be fine.


qutebrowser/browser/history.py, line 161 at r18 (raw file):

            return

        atime = int(atime) if (atime is not None) else int(time.time())

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?


qutebrowser/browser/history.py, line 211 at r18 (raw file):

        redirect = 'r' in flags
        return (url, title, int(atime), redirect)

Since the old history used local times (I think? It used time.time() at least...), this should probably convert to UTC.


qutebrowser/browser/qutescheme.py, line 221 at r18 (raw file):

        try:
            start_time = QUrlQuery(url).queryItemValue("start_time")
            start_time = float(start_time) if start_time else time.time()

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

        # replace ' to avoid breaking the query
        timefmt = "strftime('{}', last_atime, 'unixepoch')".format(

You should probably add 'localtime' as modifier (sqlite docs) so here it gets converted to local time.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

        // Find/create table
        var tableId = ["hist", date.getDate(), date.getMonth(),
            date.getYear()].join("-");

Those are all local time - maybe the IDs should be UTC instead?


qutebrowser/javascript/history.js, line 178 at r18 (raw file):

            var item = history[i];
            // python's time.time returns seconds, but js Date expects ms
            var currentItemDate = new Date(item.time * 1000);

FYI: This seems to take UTC.


qutebrowser/javascript/history.js, line 180 at r18 (raw file):

            var currentItemDate = new Date(item.time * 1000);
            getSessionNode(currentItemDate).appendChild(makeHistoryRow(
                item.url, item.title, currentItemDate.toLocaleTimeString()

FYI: This converts from UTC to local time.


Comments from Reviewable

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 20, 2017

Collaborator

I made all the conversions, but the times aren't coming out right. I don't think we actually have the problem we think we have. timestamps are timezone-independent. So maybe nothing needs to change in the implementation, but just in the test. You're right, timezones are painful.


Review status: all files reviewed at latest revision, 9 unresolved discussions.


qutebrowser/browser/history.py, line 146 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: the atime argument is only used in tests, so that bit should be fine.

Done.


qutebrowser/browser/history.py, line 161 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?

Done.


qutebrowser/browser/history.py, line 211 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Since the old history used local times (I think? It used time.time() at least...), this should probably convert to UTC.

Done.


qutebrowser/browser/qutescheme.py, line 221 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?

Done.


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

You should probably add 'localtime' as modifier (sqlite docs) so here it gets converted to local time.

Done.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Those are all local time - maybe the IDs should be UTC instead?

Done.


qutebrowser/javascript/history.js, line 178 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: This seems to take UTC.

Done. Nothing to do here now that item.time is utc


qutebrowser/javascript/history.js, line 180 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: This converts from UTC to local time.

Done.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Looks like it! They pass after a timedatectl set-timezone UTC and fail again after timedatectl set-timezone Europe/Zurich. I'll have to admit that I know nothing about timezones in Python though, so no idea what the proper fix would be...

Just pushed the UTC conversion. Hope that fixes it!


Comments from Reviewable

Collaborator

rcorre commented Jul 20, 2017

I made all the conversions, but the times aren't coming out right. I don't think we actually have the problem we think we have. timestamps are timezone-independent. So maybe nothing needs to change in the implementation, but just in the test. You're right, timezones are painful.


Review status: all files reviewed at latest revision, 9 unresolved discussions.


qutebrowser/browser/history.py, line 146 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: the atime argument is only used in tests, so that bit should be fine.

Done.


qutebrowser/browser/history.py, line 161 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?

Done.


qutebrowser/browser/history.py, line 211 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Since the old history used local times (I think? It used time.time() at least...), this should probably convert to UTC.

Done.


qutebrowser/browser/qutescheme.py, line 221 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

time.time() is timezone specific - probably datetime.datetime.utcnow().timestamp() would be better?

Done.


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

You should probably add 'localtime' as modifier (sqlite docs) so here it gets converted to local time.

Done.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Those are all local time - maybe the IDs should be UTC instead?

Done.


qutebrowser/javascript/history.js, line 178 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: This seems to take UTC.

Done. Nothing to do here now that item.time is utc


qutebrowser/javascript/history.js, line 180 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

FYI: This converts from UTC to local time.

Done.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Looks like it! They pass after a timedatectl set-timezone UTC and fail again after timedatectl set-timezone Europe/Zurich. I'll have to admit that I know nothing about timezones in Python though, so no idea what the proper fix would be...

Just pushed the UTC conversion. Hope that fixes it!


Comments from Reviewable

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 20, 2017

Collaborator

tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Just pushed the UTC conversion. Hope that fixes it!

oops, shouldn't have submitted my review, just the single comment about timestamps. I actually I didn't push it (except to my WIP branch)


Comments from Reviewable

Collaborator

rcorre commented Jul 20, 2017

tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Just pushed the UTC conversion. Hope that fixes it!

oops, shouldn't have submitted my review, just the single comment about timestamps. I actually I didn't push it (except to my WIP branch)


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 20, 2017

Collaborator

Review status: all files reviewed at latest revision, 3 unresolved discussions.


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Done.

This one still applies, see #2830.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Done.

This one still applies (but no hard feelings either way)


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

oops, shouldn't have submitted my review, just the single comment about timestamps. I actually I didn't push it (except to my WIP branch)

I resolved the discussions which are outdated now with #2830 and things being a bit clearer. I commented on the rest. This is all based on what's pushed right now - i.e. ignoring the WIP branch.


Comments from Reviewable

Collaborator

The-Compiler commented Jul 20, 2017

Review status: all files reviewed at latest revision, 3 unresolved discussions.


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Done.

This one still applies, see #2830.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Done.

This one still applies (but no hard feelings either way)


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

oops, shouldn't have submitted my review, just the single comment about timestamps. I actually I didn't push it (except to my WIP branch)

I resolved the discussions which are outdated now with #2830 and things being a bit clearer. I commented on the rest. This is all based on what's pushed right now - i.e. ignoring the WIP branch.


Comments from Reviewable

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 20, 2017

Collaborator

tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

I resolved the discussions which are outdated now with #2830 and things being a bit clearer. I commented on the rest. This is all based on what's pushed right now - i.e. ignoring the WIP branch.

hmm, I can't actually reproduce your failure. I've tried playing around with a bunch of different TZ settings but can't get test_histcategory to fail. e.g. TZ=UTC24 tox -e py36 -- tests/unit/completion/testhistcategory.py should throw dates off by a whole day (you can verify with TZ=UTC24 python -c "import datetime; print(datetime.datetime.now())"


Comments from Reviewable

Collaborator

rcorre commented Jul 20, 2017

tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

I resolved the discussions which are outdated now with #2830 and things being a bit clearer. I commented on the rest. This is all based on what's pushed right now - i.e. ignoring the WIP branch.

hmm, I can't actually reproduce your failure. I've tried playing around with a bunch of different TZ settings but can't get test_histcategory to fail. e.g. TZ=UTC24 tox -e py36 -- tests/unit/completion/testhistcategory.py should throw dates off by a whole day (you can verify with TZ=UTC24 python -c "import datetime; print(datetime.datetime.now())"


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 20, 2017

Collaborator

Review status: all files reviewed at latest revision, 3 unresolved discussions.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

hmm, I can't actually reproduce your failure. I've tried playing around with a bunch of different TZ settings but can't get test_histcategory to fail. e.g. TZ=UTC24 tox -e py36 -- tests/unit/completion/testhistcategory.py should throw dates off by a whole day (you can verify with TZ=UTC24 python -c "import datetime; print(datetime.datetime.now())"

That's because tox isolates environment variables by default - you'll need to add TZ to passenv in the [testenv] section in tox.ini:

diff --git a/tox.ini b/tox.ini
index 04cf826d1..ce3da0edb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -13,7 +13,7 @@ skipsdist = true
 setenv =
     QT_QPA_PLATFORM_PLUGIN_PATH={envdir}/Lib/site-packages/PyQt5/plugins/platforms
     PYTEST_QT_API=pyqt5
-passenv = PYTHON DISPLAY XAUTHORITY HOME USERNAME USER CI TRAVIS XDG_* QUTE_* DOCKER
+passenv = PYTHON DISPLAY XAUTHORITY HOME USERNAME USER CI TRAVIS XDG_* QUTE_* DOCKER TZ
 deps =
     -r{toxinidir}/requirements.txt
     -r{toxinidir}/misc/requirements/requirements-tests.txt

Comments from Reviewable

Collaborator

The-Compiler commented Jul 20, 2017

Review status: all files reviewed at latest revision, 3 unresolved discussions.


tests/unit/completion/test_histcategory.py, line 122 at r17 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

hmm, I can't actually reproduce your failure. I've tried playing around with a bunch of different TZ settings but can't get test_histcategory to fail. e.g. TZ=UTC24 tox -e py36 -- tests/unit/completion/testhistcategory.py should throw dates off by a whole day (you can verify with TZ=UTC24 python -c "import datetime; print(datetime.datetime.now())"

That's because tox isolates environment variables by default - you'll need to add TZ to passenv in the [testenv] section in tox.ini:

diff --git a/tox.ini b/tox.ini
index 04cf826d1..ce3da0edb 100644
--- a/tox.ini
+++ b/tox.ini
@@ -13,7 +13,7 @@ skipsdist = true
 setenv =
     QT_QPA_PLATFORM_PLUGIN_PATH={envdir}/Lib/site-packages/PyQt5/plugins/platforms
     PYTEST_QT_API=pyqt5
-passenv = PYTHON DISPLAY XAUTHORITY HOME USERNAME USER CI TRAVIS XDG_* QUTE_* DOCKER
+passenv = PYTHON DISPLAY XAUTHORITY HOME USERNAME USER CI TRAVIS XDG_* QUTE_* DOCKER TZ
 deps =
     -r{toxinidir}/requirements.txt
     -r{toxinidir}/misc/requirements/requirements-tests.txt

Comments from Reviewable

Add 'localtime' to sql history query.
We need to tell sqlite to convert the timestamps to localtime during
formatting, otherwise it formats them as though you are in UTC.

Also fix up a few uses of mktime.
@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 20, 2017

Collaborator

Reviewed 3 of 3 files at r19.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


Comments from Reviewable

Collaborator

The-Compiler commented Jul 20, 2017

Reviewed 3 of 3 files at r19.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 20, 2017

Collaborator

Review status: all files reviewed at latest revision, 2 unresolved discussions.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

            # start_time is the last second of curr_date
            start_time = next_date.timestamp() - 1

Mea culpa, this acually breaks because this is a datetime.date and not a datetime.datetime... So either back to using time.mktime(next_date.timetuple()) here, or make this datetime objects, i.e. drop the .date() above and use something like datetime.datetime.combine(datetime.date.today(), datetime.time()).


Comments from Reviewable

Collaborator

The-Compiler commented Jul 20, 2017

Review status: all files reviewed at latest revision, 2 unresolved discussions.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

            # start_time is the last second of curr_date
            start_time = next_date.timestamp() - 1

Mea culpa, this acually breaks because this is a datetime.date and not a datetime.datetime... So either back to using time.mktime(next_date.timetuple()) here, or make this datetime objects, i.e. drop the .date() above and use something like datetime.datetime.combine(datetime.date.today(), datetime.time()).


Comments from Reviewable

Fix qutescheme timestamp error.
A date object doesn't have a timestamp property. Go back to using
mktime.
@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 21, 2017

Collaborator

qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Mea culpa, this acually breaks because this is a datetime.date and not a datetime.datetime... So either back to using time.mktime(next_date.timetuple()) here, or make this datetime objects, i.e. drop the .date() above and use something like datetime.datetime.combine(datetime.date.today(), datetime.time()).

Fixed, but I'm wondering why tests only failed on a few builds ... and why I couldn't get them to fail locally. I need to look into whether this is actually covered. Also I don't think the datetime.combine method would work here since the html history page wants dhe start and end of the dayay boundaries (so we don't want the current time embedded in there). Though I can't help but wonder if we can consolidate some logic between the js and html history pages, as right now they have pretty divergent logic...


Comments from Reviewable

Collaborator

rcorre commented Jul 21, 2017

qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Mea culpa, this acually breaks because this is a datetime.date and not a datetime.datetime... So either back to using time.mktime(next_date.timetuple()) here, or make this datetime objects, i.e. drop the .date() above and use something like datetime.datetime.combine(datetime.date.today(), datetime.time()).

Fixed, but I'm wondering why tests only failed on a few builds ... and why I couldn't get them to fail locally. I need to look into whether this is actually covered. Also I don't think the datetime.combine method would work here since the html history page wants dhe start and end of the dayay boundaries (so we don't want the current time embedded in there). Though I can't help but wonder if we can consolidate some logic between the js and html history pages, as right now they have pretty divergent logic...


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 21, 2017

Collaborator

I think this is ready apart from the check_coverage.py thing. If you want to, feel free to push the merge button after that's pushed, or let me know things are ready (unless you disagree) and I'll do so 😆 🎉


Reviewed 1 of 1 files at r20.
Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Fixed, but I'm wondering why tests only failed on a few builds ... and why I couldn't get them to fail locally. I need to look into whether this is actually covered. Also I don't think the datetime.combine method would work here since the html history page wants dhe start and end of the dayay boundaries (so we don't want the current time embedded in there). Though I can't help but wonder if we can consolidate some logic between the js and html history pages, as right now they have pretty divergent logic...

datetime.time() is midnight, so I think this would've been equivalent - but I'm okay with either really.

This only happens on some builds because if the check above which uses the javascript page with a new QtWebKit or QtWebEngine. So it's only tested on legacy QtWebKit.

I don't think it's worth it to consolidate the two. I'll drop the HTML one with v1.0, which will drop legacy QtWebKit and allow to force JS for qute://* via per-domain settings.


scripts/dev/check_coverage.py, line 160 at r20 (raw file):

        'completion/models/urlmodel.py'),
    ('tests/unit/completion/test_histcategory.py',
        'completion/models/histcategory.py'),

Looks like you can add completion/models/listcategory.py too - not sure why we didn't notice earlier though?


Comments from Reviewable

Collaborator

The-Compiler commented Jul 21, 2017

I think this is ready apart from the check_coverage.py thing. If you want to, feel free to push the merge button after that's pushed, or let me know things are ready (unless you disagree) and I'll do so 😆 🎉


Reviewed 1 of 1 files at r20.
Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

Fixed, but I'm wondering why tests only failed on a few builds ... and why I couldn't get them to fail locally. I need to look into whether this is actually covered. Also I don't think the datetime.combine method would work here since the html history page wants dhe start and end of the dayay boundaries (so we don't want the current time embedded in there). Though I can't help but wonder if we can consolidate some logic between the js and html history pages, as right now they have pretty divergent logic...

datetime.time() is midnight, so I think this would've been equivalent - but I'm okay with either really.

This only happens on some builds because if the check above which uses the javascript page with a new QtWebKit or QtWebEngine. So it's only tested on legacy QtWebKit.

I don't think it's worth it to consolidate the two. I'll drop the HTML one with v1.0, which will drop legacy QtWebKit and allow to force JS for qute://* via per-domain settings.


scripts/dev/check_coverage.py, line 160 at r20 (raw file):

        'completion/models/urlmodel.py'),
    ('tests/unit/completion/test_histcategory.py',
        'completion/models/histcategory.py'),

Looks like you can add completion/models/listcategory.py too - not sure why we didn't notice earlier though?


Comments from Reviewable

@rcorre

This comment has been minimized.

Show comment
Hide comment
@rcorre

rcorre Jul 21, 2017

Collaborator

pushed a fix for the coverage. I'll wait for one more test run to make sure nothing crazy happens, then one of us can merge it 🎊


Review status: 54 of 55 files reviewed at latest revision, 3 unresolved discussions.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

datetime.time() is midnight, so I think this would've been equivalent - but I'm okay with either really.

This only happens on some builds because if the check above which uses the javascript page with a new QtWebKit or QtWebEngine. So it's only tested on legacy QtWebKit.

I don't think it's worth it to consolidate the two. I'll drop the HTML one with v1.0, which will drop legacy QtWebKit and allow to force JS for qute://* via per-domain settings.

oh, good. I'm all for dropping the HTML page


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This one still applies, see #2830.

Done.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This one still applies (but no hard feelings either way)

This is obsolete now, correct?


scripts/dev/check_coverage.py, line 160 at r20 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Looks like you can add completion/models/listcategory.py too - not sure why we didn't notice earlier though?

Done.


Comments from Reviewable

Collaborator

rcorre commented Jul 21, 2017

pushed a fix for the coverage. I'll wait for one more test run to make sure nothing crazy happens, then one of us can merge it 🎊


Review status: 54 of 55 files reviewed at latest revision, 3 unresolved discussions.


qutebrowser/browser/qutescheme.py, line 254 at r19 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

datetime.time() is midnight, so I think this would've been equivalent - but I'm okay with either really.

This only happens on some builds because if the check above which uses the javascript page with a new QtWebKit or QtWebEngine. So it's only tested on legacy QtWebKit.

I don't think it's worth it to consolidate the two. I'll drop the HTML one with v1.0, which will drop legacy QtWebKit and allow to force JS for qute://* via per-domain settings.

oh, good. I'm all for dropping the HTML page


qutebrowser/completion/models/histcategory.py, line 42 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This one still applies, see #2830.

Done.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

This one still applies (but no hard feelings either way)

This is obsolete now, correct?


scripts/dev/check_coverage.py, line 160 at r20 (raw file):

Previously, The-Compiler (Florian Bruhin) wrote…

Looks like you can add completion/models/listcategory.py too - not sure why we didn't notice earlier though?

Done.


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 21, 2017

Collaborator

:lgtm: 🎉


Reviewed 1 of 1 files at r21.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

This is obsolete now, correct?

Yep! They could still be UTC, but it really doesn't matter, and if someone reads the source of the page, localtime is probably less confusing.


Comments from Reviewable

Collaborator

The-Compiler commented Jul 21, 2017

:lgtm: 🎉


Reviewed 1 of 1 files at r21.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


qutebrowser/javascript/history.js, line 51 at r18 (raw file):

Previously, rcorre (Ryan Roden-Corrent) wrote…

This is obsolete now, correct?

Yep! They could still be UTC, but it really doesn't matter, and if someone reads the source of the page, localtime is probably less confusing.


Comments from Reviewable

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 21, 2017

Collaborator

I've never been this excited about clicking a button... let's do this, and thanks so much!

Collaborator

The-Compiler commented Jul 21, 2017

I've never been this excited about clicking a button... let's do this, and thanks so much!

@The-Compiler The-Compiler merged commit fba2533 into qutebrowser:master Jul 21, 2017

3 checks passed

code-review/reviewable 55 files reviewed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@amosbird

This comment has been minimized.

Show comment
Hide comment
@amosbird

amosbird Jul 21, 2017

Contributor

Great!

Contributor

amosbird commented Jul 21, 2017

Great!

@Quantum-cross

This comment has been minimized.

Show comment
Hide comment
@Quantum-cross

Quantum-cross Aug 10, 2017

Wow, I felt this change hard after a git pull, it feels really awesome on my older x230. Honestly the old completion system is the only thing that dragged this computer down... now nothing can stop it! Thank you all for the hard work!

Wow, I felt this change hard after a git pull, it feels really awesome on my older x230. Honestly the old completion system is the only thing that dragged this computer down... now nothing can stop it! Thank you all for the hard work!