khard: mutt mode not working well #75

0-wiz-0 · 2016-07-02T11:51:09Z

I've updated to 0.9.0 and am trying to use khard with mutt.
The addresses come from Apple software, synced via vdirsyncer.

I set up muttrc to use "khard email -p %s" like the documentation says.

When I type the first letters of a name and type ^T, I have to wait 25 seconds. I think that's much too long. (There are ~900 cards in my address book.)
The completion adds not only the email address, but also the name and the TYPE of the email. Also, the order is weird.
As (obfuscated) example, i get:

email@address.comFirstLastHOME, INTERNET, pref

for a card that has

N:Last;First;;;
FN: First Last
EMAIL;TYPE=INTERNET,HOME,pref:email@address.com
...

which is not usable without manually deleting the end.

The previous version I used, 0.4.0, just pasted email@address.com (but also took too long for that).

The text was updated successfully, but these errors were encountered:

scheibler · 2016-07-02T12:43:57Z

For the second problem, please look here: #67 (comment).

The first one is much more difficult. However 25 Seconds seem a bit long, even for around 1000
contacts. I guess, that some of your contacts contain photos?

The next release will improve the parsing speed if you select a specific address book with -a but in
general, I can't do much about that. Sorry.

0-wiz-0 · 2016-07-02T13:08:12Z

Thanks, removing the -p fixed the second issue for me.

Yes, about 20 or so have photos. The file sizes are not so bad in general, 3 files >1MB, another 6 >100kB, another 50 > 10kB and the rest below, 12MB total. Perhaps you could add some cache to make lookups faster? Or 'grep' for candidates and only parse those?

scheibler · 2016-07-07T12:45:17Z

The latest commit in the developer branch adds the new option -c / --search-in-source-files. This
should solve your first problem or at least increase performance significantly. You can try out the
development branch at https://github.com/scheibler/khard/tree/python3support. But beware, that khard
switched to python3 and therefore you have to setup a python3 environment first (see instructions in
the readme file).

0-wiz-0 · 2016-07-12T13:14:22Z

Thank you for working on this!
I've tried it out today.
I set up the virtualenv and sourced the activate script; then (in an updated git checkout of khard) I did:

python3.5 setup.py build
...
python3.5 setup.py install
...(downloaded and installed its dependencies)
~/.virtualenvs/bin/khard
(add merge_editor to config)
~/.virtualenvs/bin/khard
Traceback (most recent call last):
  File "/home/wiz/.virtualenvs/bin/khard", line 9, in <module>
    load_entry_point('khard==0.11.0', 'console_scripts', 'khard')()
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/khard.py", line 1525, in main
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/config.py", line 299, in get_address_book
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/carddav_object.py", line 88, in from_file
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/carddav_object.py", line 64, in __init__
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1129, in readOne
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1093, in readComponents
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 650, in setBehaviorFromVersionLine
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 168, in setBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 154, in autoBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/vcard.py", line 137, in decode
AttributeError: 'str' object has no attribute 'decode'

I'm in a UTF_8 environment, in case it matters.

scheibler · 2016-07-12T13:49:32Z

Tried again with the following commands:

# create python3 virtual environment
virtualenv -p python3 ~/.virtualenvs/khard3
# clone developer branch
git clone https://github.com/scheibler/khard --branch python3support --single-branch
# change into created source folder
cd khard
# install
~/.virtualenvs/khard3/bin/python setup.py install
# start
~/.virtualenvs/khard3/bin/khard

For me it's working. I don't use the virtualenv activate script. Maybe that causes the problem with
the vobject library. Or there is a problem with your old khard config file. Please try to use the
newly shipped version from the folder misc/khard/.

0-wiz-0 · 2016-07-12T14:04:24Z

I followed your instructions. Replaced config file with example (changed path, editor and merge_editor to match my setup), and otherwise just used the commands you just wrote (no activate). Same error.
Perhaps vobject does not like one of my vcards?
How can we get more information on what khard is trying to do, which file it wants to open?

scheibler · 2016-07-12T14:41:58Z

Thomas Klausner notifications@github.com schrieb am 12.07.2016, 7:04 -0700:

Perhaps vobject does not like one of my vcards?

Seems so.

How can we get more information on what khard is trying to do, which file it wants to open?

Try the following patch. That should give you the file name of the problematic vcard.

diff --git a/khard/carddav_object.py b/khard/carddav_object.py
index 3b38bb1..d874c26 100644
--- a/khard/carddav_object.py
+++ b/khard/carddav_object.py
@@ -62,14 +62,16 @@ class CarddavObject:
             # create vcard object
             try:
                 self.vcard = vobject.readOne(contents)
-            except vobject.base.ParseError:
+            except (AttributeError, vobject.base.ParseError):
                 # if creation fails, try to repair vcard contents
                 try:
                     self.vcard = vobject.readOne(
                         self.filter_invalid_tags(contents))
                     self.write_to_file(overwrite=True)
-                except vobject.base.ParseError:
+                except (AttributeError, vobject.base.ParseError):
                     raise
+            # fix organisation values
+            self.get_organisations()

     #######################################
     # factory methods to create new contact

0-wiz-0 · 2016-07-12T22:11:13Z

Thanks for the patch. The same test now results in:

Traceback (most recent call last):
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/carddav_object.py", line 64, in __init__
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1129, in readOne
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1093, in readComponents
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 650, in setBehaviorFromVersionLine
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 168, in setBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 154, in autoBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/vcard.py", line 137, in decode
AttributeError: 'str' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wiz/.virtualenvs/khard/bin/khard", line 9, in <module>
    load_entry_point('khard==0.11.0', 'console_scripts', 'khard')()
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/khard.py", line 1525, in main
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/config.py", line 299, in get_address_book
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/carddav_object.py", line 90, in from_file
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/khard-0.11.0-py3.5.egg/khard/carddav_object.py", line 69, in __init__
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1129, in readOne
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 1093, in readComponents
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 650, in setBehaviorFromVersionLine
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 168, in setBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/base.py", line 154, in autoBehavior
  File "/home/wiz/.virtualenvs/khard/lib/python3.5/site-packages/vobject-0.9.2-py3.5.egg/vobject/vcard.py", line 137, in decode
AttributeError: 'str' object has no attribute 'decode'

scheibler · 2016-07-12T22:53:40Z

Sorry. I forgot something. Here is the full patch:

diff --git a/khard/carddav_object.py b/khard/carddav_object.py
index 3b38bb1..d874c26 100644
--- a/khard/carddav_object.py
+++ b/khard/carddav_object.py
@@ -62,14 +62,16 @@ class CarddavObject:
             # create vcard object
             try:
                 self.vcard = vobject.readOne(contents)
-            except vobject.base.ParseError:
+            except (AttributeError, vobject.base.ParseError):
                 # if creation fails, try to repair vcard contents
                 try:
                     self.vcard = vobject.readOne(
                         self.filter_invalid_tags(contents))
                     self.write_to_file(overwrite=True)
-                except vobject.base.ParseError:
+                except (AttributeError, vobject.base.ParseError):
                     raise
+            # fix organisation values
+            self.get_organisations()

     #######################################
     # factory methods to create new contact
diff --git a/khard/config.py b/khard/config.py
index 959c15b..a1ef8fc 100644
--- a/khard/config.py
+++ b/khard/config.py
@@ -301,7 +301,7 @@ class Config:
                                 print("Error: Could not open file %s\n%s"
                                       % (filename, e))
                                 error_counter += 1
-                            except vobject.base.ParseError as e:
+                            except (AttributeError, vobject.base.ParseError) as e:
                                 print("Error: Could not parse file %s\n%s"
                                       % (filename, e))
                                 error_counter += 1

0-wiz-0 · 2016-07-12T23:08:57Z

That patch helped!
It prints a lot of file names like:

Error: Could not parse file /home/wiz/...
'str' object has no attribute 'decode'

and then

58 of 902 vcard files could not be parsed

I checked, and all 58 of them have a PHOTO line, perhaps that's the problem?
They look like this:

PHOTO;ENCODING=b;TYPE=JPEG;X-ABCROP-RECTANGLE=ABClipRect_1&0&0&218&218&TRVX2uC2...

or

PHOTO;ENCODING=b;TYPE=JPEG:/9j/4AAQ...

or

PHOTO;BASE64:....

scheibler · 2016-07-12T23:37:26Z

Understood. Unfortunately that is related to the underlying vcard library vobject. Seems, that the
library still has some python3 compatibility issues: skarim/vobject#20.

0-wiz-0 · 2016-07-13T06:55:16Z

How annoying.
If you want me to test the feature, can you please send a patch that ignores the 58 files it can't parse and just operates on the remainder? I tried .virtualenv/khard/bin/khard $SomeCommonName but it didn't show anything, though it should.
Thank you!

scheibler · 2016-07-13T17:21:45Z

Please try out the latest commit from the python3support branch and have a look into the
khard.conf.example file. I've added the option skip_unparsable in the vcard section, which should
do what you need.

0-wiz-0 · 2016-07-15T13:43:45Z

Thanks, that new flag works.
I tested search-in-source-files on my local address book. It was faster and returned more relevant results.
For example, searching for 'Anna' I got 11 less hits than without search-in-source-files. However, all of the "missing" ones look incorrect. The family name ends with '-mann' and they shouldn't be found. (15s vs 4s)
Looking for 'Gerd' returns two entries in either case. Without search-in-source-files it takes 14s, with search-in-source-files it still takes 4s. Much better!

I still wonder why it takes 4s for two matches even with search-in-source-files -- it should only parse two (in this case small) vcards.

scheibler · 2016-07-15T18:20:26Z

Thomas Klausner notifications@github.com schrieb am 15.07.2016, 6:43 -0700:

I tested search-in-source-files on my local address book. It was faster and returned more relevant results.
For example, searching for 'Anna' I got 11 less hits than without search-in-source-files.
However, all of the "missing" ones look incorrect.

You've encountered a small bug in the search function. Will be fixed in the next commit.

I still wonder why it takes 4s for two matches even with search-in-source-files -- it should only parse two (in this case small) vcards.

I don't know either. My 50 contacts are parsed in around 0.40 seconds. Another tester told me, that
on his system, several hundred contacts are processed in 3 seconds (without the
search-in-source-files option).

How long does it take to show khards help page (-h)? Please also try to load a single empty address
book. If that also takes longer than a half second, we can rule out the vcard parsing process.

0-wiz-0 · 2016-07-15T21:13:18Z

The first run of khard -h takes 0.801s, the second and further runs take 0.436s.
If I run khard on an empty contacts dir, it takes 0.433s before showing Found no contacts.
With a search string with no matches, it takes (with search_in_source_files) 1.471s, with 17 matches 2.552s; with 845 matches 8.213s.
The 4s numbers earlier were on a very loaded system. I think the numbers are ok now (with search_in_source_files) for low number of hits.
Thank you for the improvements!

scheibler closed this as completed Jul 18, 2016

scheibler mentioned this issue Jul 19, 2016

[v0.11.0] Parsing error with photo included #80

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

khard: mutt mode not working well #75

khard: mutt mode not working well #75

0-wiz-0 commented Jul 2, 2016 •

edited

scheibler commented Jul 2, 2016

0-wiz-0 commented Jul 2, 2016

scheibler commented Jul 7, 2016

0-wiz-0 commented Jul 12, 2016 •

edited

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 13, 2016

scheibler commented Jul 13, 2016

0-wiz-0 commented Jul 15, 2016

scheibler commented Jul 15, 2016

0-wiz-0 commented Jul 15, 2016 •

edited

khard: mutt mode not working well #75

khard: mutt mode not working well #75

Comments

0-wiz-0 commented Jul 2, 2016 • edited

scheibler commented Jul 2, 2016

0-wiz-0 commented Jul 2, 2016

scheibler commented Jul 7, 2016

0-wiz-0 commented Jul 12, 2016 • edited

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 12, 2016

scheibler commented Jul 12, 2016

0-wiz-0 commented Jul 13, 2016

scheibler commented Jul 13, 2016

0-wiz-0 commented Jul 15, 2016

scheibler commented Jul 15, 2016

0-wiz-0 commented Jul 15, 2016 • edited

0-wiz-0 commented Jul 2, 2016 •

edited

0-wiz-0 commented Jul 12, 2016 •

edited

0-wiz-0 commented Jul 15, 2016 •

edited