Merge quickmarks and bookmarks / making bookmarks more powerful

[The-Compiler]

See #840 - the difference between quickmarks and bookmarks also is confusing for users sometimes, so it might make sense to merge them into one. Basically, simply having bookmarks with an optional name/label, and when that is set, they're "quickmarks".

edit: The new command should also make sure the URL actually is an URL.

Please do not :-) QuickMarks go well with more character like DWB. B + QM = very fast. Bookmarks has the problem that it can not be searched. that's the only issues. The also do not need a shortcut. I think, the division as vimperator or dactyl is a good approach

[lamarpavel]

@sdoubleyou I don't understand the problem, if bookmarks and quickmarks would be merged, then the resulting marks would be searchable and could optionally have a keyword just like a quickmark. You wouldn't lose anything.

[The-Compiler]

Just as @lamarpavel said, this is mainly about the way marks are stored on disk - I intend the user-interface to stay the same more or less, except of :open optionally also opening marks as explained in #840.

What I'm a bit worried about is that the format most likely has to change, as I'm not 100% happy with the directory format I proposed for bookmarks, and the <label> <url> (quickmarks) and <url> <title> formats are far too inflexible to add new information...

What about using yaml, just like the session files? This would mean something like this:

- url: http://www.qutebrowser.org/
  title: qutebrowser
  label: git
- url: http://www.example.org/
  title: Example Page
...

And could also be easily changed to allow subfolders later:

- projects:
  - url: http://www.qutebrowser.org/
    title: qutebrowser
    label: git
  - url: http://www.herbstluftwm.org/
    title: herbstluftwm
- url: http://www.example.org/
  title: Example Page
...

However, it clearly loses the "very easy to write by hand, modify by a shell script, etc." property of the old format...

@Emdek also proposed using xbel which is a standard format for bookmarks, but meh... it's XML 😉

What do you think?

[The-Compiler]

Oh - @antoyo, what's your take on this? I know I was the one who proposed the folder format, but now I'm thinking this would be better in the long run.

[antoyo]

While the YAML format is more flexible, it is not compatible with the format used by dwb for bookmarks.
But, the compatibility will probably be broken anyway when the support for folders will be added.
And, if needed, we could add a way to import bookmarks from the old (dwb) format.
Moreover, we'll need to way to migrate from the old format if we change.

[arp242]

For what it's worth, I often just edit my quickmarks file directly. I like having it in a separate file for the same reason I like having plain text rather than (for example) sqlite (even though sqlite has some things going for it): it enhances what I call the unixability, that means, editing it without frills with normal unix tools (grep, sed, vi, etc.).

[lamarpavel]

@Carpetsmoker I'm with you on this, especially since qute makes it so easy to bind editing one of the text files to a key (eg spawn $TERM -e 'nvim ~/.config/qutebrowser/quickmarks.conf').

I imagine that when merging quickmarks and bookmarks we would have the same syntax for both in a text file and that former quickmarks are bookmarks with an appendix such as --quick <keyword>. Such a format would still be easily editable in a Unix friendly manner, no?

[The-Compiler]

That would work - but what when you want to add something like labels then? Or have subfolders? Or store a title with quickmarks as well? Having all that in a single line sounds like a very inflexible format...

I'm not sure what to do - I can see both sides. I'd like to be able to easily extend the format in the future without worrying about backwards compability (especially for bookmarks), but I'd also like to have a format with much "unixability" (I like that term!)

[lamarpavel]

Quick idea:
Have a folder marks in which there is one text file for each bookmark (with the option for categorizing them in nested subfolders). Each file could have an easily parseable format with one option per line:

~/.config/qutebrowser/marks/qute-repo
url: https://github.com/The-Compiler/qutebrowser
tags: dev,code
key: qb

Here the name of the bookmark would be the file name, url and tags should be clear and key is the quickmark string. Few things are "more Unix" than using the filesystem in clever ways, right? Performance-wise it should be fine since fs-tree is being cached by the OS after first access, searching through cached file system hierarchies is something many Unix tools rely on and support and it allows little room for ambiguity.

Thoughts?

[The-Compiler]

I have to think about this - I had a similar idea in the discussion with @antoyo in #681, but with only one marks file in every directory.

As a bonus, this would even be valid YAML - and if you use tags: [dev, code] it'd even be a real yaml list.

Some points I'll have to think about:

• How does this scale to thousands of bookmarks? I remember @claudehohl probably has something around this number.
• How does this scale on Windows, which has a horribly slow filesystem?
• What to do with invalid filename chars in bookmark names (on Windows, and maybe OS X)?
• What to do with bookmarks which don't have a name?

One possibility would be to have a name: ... line in the file, and then only have some kind of slug of the URL (or name?) as filename (plus numbers if there are conflicts), but ignore the filename entirely when reading.

[claudehohl]

How does this scale to thousands of bookmarks? I remember @claudehohl probably has something around this number.

That obviously doesn't scale. Chromium has one Bookmarks file; a json. Load it into ram for full speed ;)
And I have ~2700 bookmarks (never delete any, it builds up year after year)...
How does Python scale in reading 100k files? Quickly forget about that idea of having one file per bookmark. *Bookmark added* --> minus one inode, 4k storage wasted. Who wants to edit bookmarks via shell script anyway? I prefer the browsing experience, speed and efficiency. For the rest, there is importers and Python scripts.

[bb010g]

Looking at what @claudehohl has, as weird as it can be, SQLite may be our best choice here. It can deal with tons of data and won't kill the file system.

[claudehohl]

(and now everybody): "SQLite! SQLite! SQLite!"

[The-Compiler]

Just had a quick discussion about it in IRC:

20:15 <Emdek> The-Compiler: someone suggested SQLite as storage, I'm currently using it for history and I would like to kill it
20:21 <The-Compiler> Emdek: why so?
20:21 <The-Compiler> (and bb010g did)
20:24 <Emdek> extra dependency, journal file causes issues on some file systems, some users report performance issues (rare)
20:25 <iggy> extra dep?
20:25 <The-Compiler> no extra dependency with python :P
20:26 <Emdek> yes, extra dependency, requires Qt module and driver
20:29 <The-Compiler> https://docs.python.org/3/library/sqlite3.html in my case - but I'd like to keep it plain text anyways
20:42 <bb010g> Emdek: Could you elaborate on the journal file issues?
20:43 <Emdek> sorry, I have beta to release... there was some ticket in issue tracker of Otter where someone was complaining about it
20:44 <bb010g> The-Compiler: I'd like to do YAML too, but scaling flat files is pretty dang hard.
20:45 <The-Compiler> bb010g: I dunno, it works pretty well for 80'000 history entries here
20:47 <Emdek> SQLite needs to load file into memory anyway

[tex]

Please leave bookmarks on disk in format editable in vim :-)