ToDo list

[jarun]

Continued from #268.

Notes

The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!

Some of the fresh-baked features may not have been released yet. Grab the master branch for those.

Identified tasks

• support archive.is
• bukuserver: disable favicon
• bukuserver: bookmarklet documentation (see Bookmarklet #385)
• bukuserver: thumbnail
• bukuserver: add login feature (see Security features in bukuserver #309)
• bukuserver: generate separate package (see Package bukuserver #307)
• Import keywords from browser html (see Support for "real" keyword searches #311)
• Android app (with the same schema) (probably a separate project)
• Add option --preserve to ignore specific fields during auto-update [see issue Preserve some preexising data during update operation #327]
• Test cases for exportdb() API (@rachmadaniHaryono)
• Purge db file when all records are deleted
• Fix tests, which are taking long time to finish (thanks @rachmadaniHaryono)
• Import firefox exported json (title, url, tags, desc; parent folder if opted) [see API importdb()] (thanks @ckolumbus)
• Disable fetch from web during auto-import, import and merge
• Honor -n with -p (thanks @jpdasma)
• Support search options with --export
• Import "description" (into description field) and "keywards" (as tags) from HTML <meta> Tag
• Result pagination
• Convert bukuserver README from rst to markdown
• Browse a bookmark (possibly dead URL) on Wayback machine
• Show bookmarks to be deleted before deletion
• Support keyword filtering (records having keywords a and b but not c and d) (thanks @saltyCatfish)
• Support filtering by tags with search options (see Searching for tags and keywords #250) (thanks @saltyCatfish)
• Port feature from googler/ddgr - omniprompt key O
• Show results with most search keyword matches on top (thanks @mosegontar)
• Text-mode user agent for Buku
• Copy search result URL to cipboard
• Read default Firefox profile name from profiles.ini (see Firefox auto-import breaks if profile name does not end with .default #212, thanks @alex-bender)
• Support --format in search results (ref, thanks @mosegontar)
• API documentation (comments need to be in NumPy format) (thanks @mosegontar)
• Auto-import: optionally add parent folder name as tag, ask for unique tag [like importdb()]
• Support custom colours (thanks @shv-q3)
• Generate packages on Travis-CI using PackageCore (see Generate packages using PackageCore and Travis CI #189) (thanks @shaggytwodope)
• Search multiple tags, exclusion in tag search (thanks @mosegontar)
• Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
• Suggest tags those go together
• Append/overwrite/remove tags from prompt
• Add more tests (ongoing activity @rachmadaniHaryono)
• A browser plugin (thanks @samhh for bukubrow)
• Text editor support (thanks @ZwodahS)
• Need a PyPI maintainer (thanks @shaggytwodope)
• Make refreshdb faster using threads (record updates should be synchronized)
• Show usage count in tag list
• Proxy support (thanks @denisfalqueto)
• Continuous search at prompt
• Add prompt help
• Specify custom DB file to class BukuDb (library usage, no exposed option)
• Move to urllib3
• Handle redirects using referrer masking. Example URL. Fixed with urllib3.
• Support URL shortening. This helps to share URLs. (see Support adding shortened URLs to the database #92 for limitations)
• Make a bookmark title immutable via refreshdb()
• Markdown import/export
• Regex search
• Ubuntu PPA (thanks @shaggytwodope)
• Export specific tags to HTML
• Exact word match using REGEX. Make substring match optional.
• Delete all records based on a search result
• Delete multiple items, support combination of indices and ranges
• Append tags
• Travis CI integration
• Ubuntu deb package generation on new tag
• Merge bookmark database files (for users who work on multiple systems)
• Export bookmarks in FF or Chrome html format.
• Option to add folder names as tags while importing HTML (thanks @Mohammadkhalifa)
• Check and show upstream version
• Anything else which would add value (please discuss in this thread)

[rachmadaniHaryono]

Fix tests, which are taking long time to finish

this is finished

[rachmadaniHaryono]

anyone ever have experience database with empty record but but not 0 filesize?

i tried open it with db browser for sqlite but it also show no records

[jarun]

~$ cp ~/.local/share/buku/bookmarks.db ~/bookmarks.db -rvf
'/home/vaio/.local/share/buku/bookmarks.db' -> '/home/vaio/bookmarks.db'
1 files (204.0 KiB) copied in 0.0 seconds ( 12.7 MiB/s).
~$ buku -d
Remove ALL bookmarks? (y/n): y
All bookmarks deleted
~$ ll .local/share/buku/bookmarks.db 
-rw-rw-r-- 1 vaio vaio 207872 Jan  1 21:22 .local/share/buku/bookmarks.db
~$ ll ~/bookmarks.db 
-rw-rw-r-- 1 vaio vaio 207872 Jan  1 21:22 /home/vaio/bookmarks.db
~$ buku -p
[ERROR] 0 records
~$

The sqlite operation used is DROP TABLE.

[rachmadaniHaryono]

is that empty database at first or there is already records in that?

$ ll ~/.local/share/buku/bookmarks.db
-rw-r--r-- 1 q q 10907648 Jan  2 00:03 /home/q/.local/share/buku/bookmarks.db

that is the file (10.4mb), yet no record either from buku or from sqlite db browser

[jarun]

There were records at first. I have around 477 bookmarks in my personal file.

I think it would be a better option to just remove the file instead of dropping the table. That would be fast and reclaim the space. Added to the ToDo list.

Or maybe use the same code from delete_rec_all().

[rookiejet]

A "-r" reverse sort option for import (and export) could be useful to maintain chronology.

[jarun]

reverse sort based on which criteria?

[rookiejet]

Date added.

[jarun]

Buku doesn't store that information.

[rookiejet]

Entry/record order could work as well (atleast it is enough for my use case). Pinboard export maintains chronology of entries in descending order of most recent, and when imported to buku it is reversed.

[jarun]

Please read the introduction: no hidden history, obsolete records, usage analytics or homing

[rookiejet]

So would bookmark entry order falls under obsolete records? I don't follow. Buku already records and surfaces that. Sorry, I may have miscommunicated what I intended.

[jarun]

So would bookmark entry order falls under obsolete records?

No, the date added. It's possible to figure out patterns from that.

Generally speaking, we leave out any king of usage history.

[jarun]

You mentioned "chronology". If you are asking for just printing in reverse order, I think you can add a filter to do that while listing. If you really need it, please raise the PR.

[jabagawee]

Thoughts on adding type hints?

[rachmadaniHaryono]

while it could be done, it is better to put the effort into making new feature

[jarun]

Please feel free to add to the ToDo list.

[jabagawee]

What's the best way to chat? Seems like the gitter.im instance is inactive, and I'm not sure what way works best for you.

[jarun]

I pinged you now. I didn't get the noti from gitter.

[dertuxmalwieder]

A Chocolatey package would be awesome. I might have some time to try to build one some time in 2019...

(Packed calendar currently.)

[jarun]

@dertuxmalwieder whenever you have the time!

[jarun]

@rachmadaniHaryono can you please add the test cases for exportdb()?

[rachmadaniHaryono]

@jarun python 3.4 eol is last month

also there is warning from circleci

DEPRECATION: Python 3.4 support has been deprecated. 
pip 19.1 will be the last one supporting it. 
Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).

until when do we support it?

[jarun]

I'll remove it. :)

[rachmadaniHaryono]

@jabagawee i started added annotation on the pr and some function/method on buku

it is not perfect yet but it is a start

[EncryptedCurse]

Would it be possible to add a bookmark cleanup function of sorts (e.g. find dead links)?

Also, why doesn't the interactive prompt accept the same arguments as running Buku directly?

[bitozoid]

How about is the url is slightly modified? The "link" from one entry to the other would be missing. Tags relate bookmarks, but they do not point other specific entries.

[jarun]

other specific entries

Can you please elaborate?

[bitozoid]

The point is to create a "brain" of bookmark entries linked among them, where every bookmark entry has a description and internal links to other bookmark entries. A bookmark entry may have tags, but its tags do not work as pointers to other named bookmark entries (they would refer to every entry that has that tag). Is my point clearer? Anyway it is just an idea.

[jarun]

I think it will make the solution over-complicated. We have extensive search facilities already and I am of the opinion that the types of search is more than adequate to find a bookmark.

@rachmadaniHaryono what do you think?

[rachmadaniHaryono]

for buku itself imo kinda hard due to terminal user interface

for bukuserver it may have work by creating custom text-to-html, which will link any matching text to another entry (e.g. reddit r/subreddit text)

[Unknow0059]

Does bukuserver have a Grid View? Or even something like Windows' File Explorer List View?

The concept of an offline bookmark manager is one that I love because most alternatives are webapps intent on using your data and/or being a social platform, but I need an alternative because the list-based approach used by browsers to manage bookmarks doesn't show enough bookmarks at once, making managing difficult.

[jarun]

@rachmadaniHaryono do you have any plans for a grid-based view?

[rachmadaniHaryono]

grid based view

while it is possible, i'm not good enough in html to hack flask admin that. i may try but i will not promise anything

but i will love if bukuserver can what shiori does https://github.com/go-shiori/shiori

Windows' File Explorer List View

https://i.stack.imgur.com/BW7xN.png

i kinda confused with this because i have no idea how does each item will look like

doesn't show enough bookmarks at once, making managing difficult.

increase bookmarks per page config here https://github.com/jarun/buku/tree/master/bukuserver#configuration

[Unknow0059]

but i will love if bukuserver can what shiori does https://github.com/go-shiori/shiori

Yeah it'd be nice. I tried shiori but it just offers nothing when it comes to actual management.

Windows' File Explorer List View

https://i.stack.imgur.com/BW7xN.png

i kinda confused with this because i have no idea how does each item will look like

Wdym? You mean you're not sure how to implement the max length of each column list? You can truncate it to a variable limit.
e.g. a limit of 20 (or any chosen number) characters per item, then after that you just get ellipsis.

[rachmadaniHaryono]

Wdym? You mean you're not sure how to implement the max length of each column list? You can truncate it to a variable limit.

e.g. a limit of 20 (or any chosen number) characters per item, then after that you just get ellipsis.

so just url (or shortened one or click/hover to see full url) on single line. that maybe simple enough

[Unknow0059]

so just url (or shortened one or click/hover to see full url) on single line. that maybe simple enough

Title, then URL displayed on hover

[klartext]

Any idea on using different database-files?
Found a feature in the list, which supports it internall, but not for usage.
Planned to add that?

OK, found it in the operational notes. It was not mentioned in the output of --help.
This is intended (because of "Manual usage is prone to issues arising from human error.")?

[jarun]

Yes, it is intended. Use at your own risk.

[klartext]

Yes, it is intended. Use at your own risk.

I think -d switch to delete the whole database is rather a risk to destroy the data than -db to use a different database-file.

[2ynn]

I use bookmark folders extensively on firefox (rarely tags) and use the "Add parent folder names as tags" option when importing automatically from firefox. I am interested in preserving my folder tree structure and have it reflected in the sequence of comma separated tags. I therefore removed sorted() in the parse_tags function. buku line 3687: unique_tags = set(orig_tags)

Is there any side effects from doing this?

Thanks for the great work!

[jarun]

Is there any side effects from doing this?

I am not sure. Never tested it this way. If your search results are fine maybe it works.

[2ynn]

Thanks for your reply. Actually my solution doesn't work as set() does not preserve the insertion order from the orig_tags list. Instead, I am now directly returning the orig_tags list (wrapped with delimiter). My search results seem to work fine. I suppose I could run into issues if I use subfolders which have the same name as their parents.... but I just won't

[jarun]

You can have this modification in your fork. At this stage of the project I do not expect too many changes so it shouldn't be difficult to merge at a later point.

[rachmadaniHaryono]

@jarun i want to update this page https://github.com/jarun/buku/wiki/Buku-as-a-library

---

buku is developed as a powerful python library for bookmark management. All functionality are available through carefully designed APIs. main() is a good usage example. It's also possible to use a custom database file in multi-user scenarios. Check out the documentation for the following APIs which accept an optional argument as database.

>>> import buku
>>> buku.BukuDb.initdb(dbfile=None)
>>> buku.BukuCrypt.encrypt_file(iterations, dbfile=None)
>>> buku.BukuCrypt.decrypt_file(iterations, dbfile=None)

NOTE: This flexibility is not exposed in the program.

An example to use buku as library

>>> import buku
>>> buku.__version__
'4.4'
>>> bdb = buku.BukuDb(dbfile='temp.db')
>>> bdb.get_rec_all()
[]
>>> bdb.add_rec(url='https://example.com', title_in='example title', tags_in='tag1', desc='example desc')
1
>>> bdb.get_rec_all()
[(1, 'https://example.com', 'example title', ',tag1,', 'example desc', 0)]
>>> # id, url, title, tags, description, flags
>>> bdb.add_rec(url='https://example.com/1.jpg')
1
>>> bdb.get_rec_all()
[(1, 'https://example.com', 'example title', ',tag1,', 'example desc', 0),
 (2, 'https://example.com/1.jpg', None, ',', '', 0)]
>>> # Note: if record don't have tags, it will return ',' (comma character)

The bukuserver has several wrapper web APIs. Feel free to update if you need them.

---

changes

• python syntax for first example code
• replace api with bukuserver
• remove example to print http status code for basic buku add and print record
• add buku version

[jarun]

Sure thing!

[Clarkinia]

On fedora 33, copying a link in a search (c #id) throws this error:
Failed to locate suitable clipboard utility
I was thinking maybe it's a Wayland issue? When I right-click on the link in terminal I am able to copy.

The reason I want to copy is to be able to open in a browser other than Firefox (or the browsers listed in the Python webbrowser controller). Perhaps this is too much of a pipe dream, but I would like to choose a different browser with one letter. For example:
O 5 f Open index 5 in firefox
O 5 c Open index 5 in chrome

[jarun]

You may have to install wl-copy. See function copy_to_clipboard() for a list of supported copiers.

Switching browsers on the fly is not supported.

[Clarkinia]

Thanks! That worked. I had also installed buku from fedora dnf and the code was older. I looked at the file in /usr/bin/buku and the "wl-copy" info was not there yet. So I downloaded the latest from the buku github repository and replaced the buku file. And I installed wl-clipboard from the fedora's package manager. Now copy works. Thanks.