Merge quickmarks and bookmarks / making bookmarks more powerful

[The-Compiler]

See #840 - the difference between quickmarks and bookmarks also is confusing for users sometimes, so it might make sense to merge them into one. Basically, simply having bookmarks with an optional name/label, and when that is set, they're "quickmarks".

edit: The new command should also make sure the URL actually is an URL.

[ghost]

Please do not :-) QuickMarks go well with more character like DWB. B + QM = very fast. Bookmarks has the problem that it can not be searched. that's the only issues. The also do not need a shortcut. I think, the division as vimperator or dactyl is a good approach

[lamarpavel]

@sdoubleyou I don't understand the problem, if bookmarks and quickmarks would be merged, then the resulting marks would be searchable and could optionally have a keyword just like a quickmark. You wouldn't lose anything.

[The-Compiler]

Just as @lamarpavel said, this is mainly about the way marks are stored on disk - I intend the user-interface to stay the same more or less, except of :open optionally also opening marks as explained in #840.

What I'm a bit worried about is that the format most likely has to change, as I'm not 100% happy with the directory format I proposed for bookmarks, and the <label> <url> (quickmarks) and <url> <title> formats are far too inflexible to add new information...

What about using yaml, just like the session files? This would mean something like this:

- url: http://www.qutebrowser.org/
  title: qutebrowser
  label: git
- url: http://www.example.org/
  title: Example Page
...

And could also be easily changed to allow subfolders later:

- projects:
  - url: http://www.qutebrowser.org/
    title: qutebrowser
    label: git
  - url: http://www.herbstluftwm.org/
    title: herbstluftwm
- url: http://www.example.org/
  title: Example Page
...

However, it clearly loses the "very easy to write by hand, modify by a shell script, etc." property of the old format...

@Emdek also proposed using xbel which is a standard format for bookmarks, but meh... it's XML 😉

What do you think?

[The-Compiler]

Oh - @antoyo, what's your take on this? I know I was the one who proposed the folder format, but now I'm thinking this would be better in the long run.

[antoyo]

While the YAML format is more flexible, it is not compatible with the format used by dwb for bookmarks.
But, the compatibility will probably be broken anyway when the support for folders will be added.
And, if needed, we could add a way to import bookmarks from the old (dwb) format.
Moreover, we'll need to way to migrate from the old format if we change.

[arp242]

For what it's worth, I often just edit my quickmarks file directly. I like having it in a separate file for the same reason I like having plain text rather than (for example) sqlite (even though sqlite has some things going for it): it enhances what I call the unixability, that means, editing it without frills with normal unix tools (grep, sed, vi, etc.).

[lamarpavel]

@Carpetsmoker I'm with you on this, especially since qute makes it so easy to bind editing one of the text files to a key (eg spawn $TERM -e 'nvim ~/.config/qutebrowser/quickmarks.conf').

I imagine that when merging quickmarks and bookmarks we would have the same syntax for both in a text file and that former quickmarks are bookmarks with an appendix such as --quick <keyword>. Such a format would still be easily editable in a Unix friendly manner, no?

[The-Compiler]

That would work - but what when you want to add something like labels then? Or have subfolders? Or store a title with quickmarks as well? Having all that in a single line sounds like a very inflexible format...

I'm not sure what to do - I can see both sides. I'd like to be able to easily extend the format in the future without worrying about backwards compability (especially for bookmarks), but I'd also like to have a format with much "unixability" (I like that term!)

[lamarpavel]

Quick idea:
Have a folder marks in which there is one text file for each bookmark (with the option for categorizing them in nested subfolders). Each file could have an easily parseable format with one option per line:

~/.config/qutebrowser/marks/qute-repo
url: https://github.com/The-Compiler/qutebrowser
tags: dev,code
key: qb

Here the name of the bookmark would be the file name, url and tags should be clear and key is the quickmark string. Few things are "more Unix" than using the filesystem in clever ways, right? Performance-wise it should be fine since fs-tree is being cached by the OS after first access, searching through cached file system hierarchies is something many Unix tools rely on and support and it allows little room for ambiguity.

Thoughts?

[The-Compiler]

I have to think about this - I had a similar idea in the discussion with @antoyo in #681, but with only one marks file in every directory.

As a bonus, this would even be valid YAML - and if you use tags: [dev, code] it'd even be a real yaml list.

Some points I'll have to think about:

• How does this scale to thousands of bookmarks? I remember @claudehohl probably has something around this number.
• How does this scale on Windows, which has a horribly slow filesystem?
• What to do with invalid filename chars in bookmark names (on Windows, and maybe OS X)?
• What to do with bookmarks which don't have a name?

One possibility would be to have a name: ... line in the file, and then only have some kind of slug of the URL (or name?) as filename (plus numbers if there are conflicts), but ignore the filename entirely when reading.

[claudehohl]

How does this scale to thousands of bookmarks? I remember @claudehohl probably has something around this number.

That obviously doesn't scale. Chromium has one Bookmarks file; a json. Load it into ram for full speed ;)
And I have ~2700 bookmarks (never delete any, it builds up year after year)...
How does Python scale in reading 100k files? Quickly forget about that idea of having one file per bookmark. *Bookmark added* --> minus one inode, 4k storage wasted. Who wants to edit bookmarks via shell script anyway? I prefer the browsing experience, speed and efficiency. For the rest, there is importers and Python scripts.

[bb010g]

Looking at what @claudehohl has, as weird as it can be, SQLite may be our best choice here. It can deal with tons of data and won't kill the file system.

[claudehohl]

(and now everybody): "SQLite! SQLite! SQLite!"

[The-Compiler]

Just had a quick discussion about it in IRC:

20:15 <Emdek> The-Compiler: someone suggested SQLite as storage, I'm currently using it for history and I would like to kill it
20:21 <The-Compiler> Emdek: why so?
20:21 <The-Compiler> (and bb010g did)
20:24 <Emdek> extra dependency, journal file causes issues on some file systems, some users report performance issues (rare)
20:25 <iggy> extra dep?
20:25 <The-Compiler> no extra dependency with python :P
20:26 <Emdek> yes, extra dependency, requires Qt module and driver
20:29 <The-Compiler> https://docs.python.org/3/library/sqlite3.html in my case - but I'd like to keep it plain text anyways
20:42 <bb010g> Emdek: Could you elaborate on the journal file issues?
20:43 <Emdek> sorry, I have beta to release... there was some ticket in issue tracker of Otter where someone was complaining about it
20:44 <bb010g> The-Compiler: I'd like to do YAML too, but scaling flat files is pretty dang hard.
20:45 <The-Compiler> bb010g: I dunno, it works pretty well for 80'000 history entries here
20:47 <Emdek> SQLite needs to load file into memory anyway

[tex]

Please leave bookmarks on disk in format editable in vim :-)

[Ambrevar]

• Scalability: we are talking a few thousand URI max. Searching that should be instantaneous, even in Python. No need for a database here.
• Multiple files: unnecessarily complicated and slow.
• SQLite (or other binary formats): Anti-Unix as it locks you down to using a specific tool for manipulating the file. Most importantly, the bookmarks would not be versionable anymore, which is a key feature of browsers like Qutebrowser.
• YAML: It makes it harder to process with common tools like AWK, sed, etc. Important example: you cannot sort them with 'sort -k[N]' anymore.
• As for possible additional fields, do you have an example in mind?

I believe we should stick to a one-liner format. Look at dwb quickmarks: it has a URI, a keyword and a name.

[vyp]

I don't mind YAML, but I'm a -1 on SQLite. plain text bookmarks have the huge advantage (imo) of being able to be tracked via git. And as @Ambrevar says even 10'000 items should not be a problem to search, right?

[lamarpavel]

Having had some time to think about it I agree that one file per mark is not a good way to go about it. For the reasons mentioned several times here I am also against any form of locked in format such as SQL.

YAML or JSON is kind of okay, but I would much prefer formats that are consice and that can be easily parsed with grep, awk, ag etc.

I suggest taking a look at the format of ctags/exuberant tags for inspiration.

[The-Compiler]

I also think either YAML or one-liners are the way to go.

As for the information possibly required:

• A "folder" for hierarchical bookmarks
• A name for quickmarks
• The URL of course
• The page title
• Firefox also has tags and a description. I don't think those are needed.

[tex]

@The-Compiler tags are very important for me, description is handy

[bb010g]

Don't forget shyaml (for YAML) and the excellent jq (for JSON, but pair it with yaml2json or something similar) for command line work.

[fiete201]

I think yaml looks good. Just to argue about the format @The-Compiler showed where every bookmark got 4 or 5 lines, I think there is no problem to tell awk to start with the second line and show me every 5th line afterwards. e.g the url and sort it or so. Or just print me every fourth and fifth line just to have "url" "tag" again and sort that.
The case of needing one liners for unix tools is perhaps not so often than using it inside the browser so this one step more to construct the one liner format should be ok.
At least for me.

[sudo-nice]

I need tags too, please keep it.

[cwmke]

A third for keeping tags. When you start to get into hundereds or thousands of bookmarks, it makes they make it very simple to find related items quickly. Folders, I guess, can serve a similar purpose although I find tags tend to be quicker.

[antoyo]

I'm starting to see the needs for tags too, as I get more bookmarks.
Also, it would be nice be to able to filter bookmarks by tags in :open with something like:

:open [my-tag1][my-tag2]

to see only bookmarks with some tags.

[lucc]

One question: What are you refering to as tags? I know them from Firefox but don't see something similar beeing mentioned in the docs of QB. As far as I understand it there is only an URL and some text (the title) available for each bookmark.

An idea for the file format: Keep it in a line oriented format. You can also define a strict syntax for it, for example tab delimited fields with tab chars not allowd inside fields to remove the need of quoteing. This would also be extendable for future fields. And it is very unix tools friendly and very similar to system files on unix: /etc/{fs,cron,crypt}tab /etc/{passwd,group,shadow}.

An idea for tags folders and completion: If we make the completion "fuzzy" in the sense that it does not care for the order of input words (so completion on "foo bar" will give the same results as completion on "bar foo") we actually should already have folders and tags implicitly. The idea is that user can edit the "title" in the bookmarks/urls file to include any text they like and think of these titles as paths or tags or titles. I could for example have this urls file:

http://example.com some website
http://homepage.net /private/homepage My homepage
https://github.com/The-Compiler/qutebrowser coding python github webbrowser

For the browser these are all just urls with titles. But if the completion is word-order-insensitive I can think of these as titles, paths/folders or tags however I like.

I would also suggest to allow an argument to bookmark-add in order to specify the title for a bookmark.

Oh, and "Yes" I think we can merge bookmarks and quickmarks,

[jcguu95]

Following up after 1-year hanging

This very important issue has hung for more than a year. What's the current state?

I understand that for certain issues using sql would be more suitable. However, the discussion here which last for ~3 years shows me that maybe we do not need to store more information to a bookmark entry.

It seems to me that essentially we only need url, title, tags, and search (see @miseran 's comment on Feb 13, 2018). If this is really the case, I see no problem with using a csv or even a tsv for the format. For example

<url>              <title>                 <tags>
qutebrowser.org        qutebrowser: a keyboard driven browser        favorite best-browser

For users who want hierarchical structures to be set, we can also do that by reading every file withing the bookmark directory. By this I mean:

$QB_DIR/bookmark/
    bookmark.tsv                      # the default bookmark file
    my_bookmark_folder_1/    # let the user decides if creating this folder or not
        other_bookmark_1.tsv
        other_bookmark_2.tsv
...

---

changes made by @rcorre

I can see lots of changes above made by @rcorre, such as bookmark-tag. However, those added commands are not available for me now. I am using qutebrowser v1.7.0 now. Is it because I have to switch to a less stable version to enjoy the features?

[The-Compiler]

@jcguu95 "very important" is quite subjective 😉 @rcorre started a lot of work in #3855 but I haven't been able to take a look yet, because I've been very busy with my studies and other important changes (mainly changes needed to work with Qt updates) in the past year or so. This will change later this year as my studies are almost done now.

[jcguu95]

It's time for qb to fly! Thanks and good luck for it!

[rcorre]

I got pretty overwhelmed by other things as well, but I'm happy to come back to it! I'll probably wait until @The-Compiler has more time though 😁

[jcguu95]

It might not be political correct here. But I happen to come across org-mode.. seems that a very good way to handle plain text file.

[The-Compiler]

@jcguu95 A decision on the storage format has already been made (and implemented in #3855), any more bikeshedding won't be productive.

[Hund]

I would love to see support for tags and even categories. :)

[adigitoleo]

I see no mention of this here, forgive me if it was already talked about:

Whichever system is chosen, can the searching be made case-insensitive (already is)? Or even fuzzy (order of terms does not need to match in returned result)? Update: refer to #6114

Just a small suggestion, as after probably a year of using QB, I've noticed that I'm often retyping my quickmark search term because of incorrect case or word-order.

[The-Compiler]

@adigitoleo that seems like a separate thing - mind opening a new issue?