Error using Kobo module

[ashfaqfarooqui]

Thanks for this package, it is something I've been wishing for a long time.

Firstly, I've not been very involved with using Python, so it was not straight forward to me how to use the different packages you have (as requirements) to run Orger. Specifically getting my_configuration in the correct paths. I hope you can make this a self-contained package which does not require additional packages to be explicitly installed. [Maybe, I'm totally wrong and have somehow misread your instructions, and doing something wrong; I'm not sure].

When I got everything to work together, I get the following error while importing kobo annotations

Traceback (most recent call last):
  File "/home/ashfaqf/Code/orger/modules/kobo.py", line 35, in <module>
    KoboView.main()
  File "/home/ashfaqf/.local/lib/python3.7/site-packages/orger/org_view.py", line 64, in main
    inst._run(to=args.to)
  File "/home/ashfaqf/.local/lib/python3.7/site-packages/orger/org_view.py", line 72, in _run
    org_tree = self.make_tree()
  File "/home/ashfaqf/.local/lib/python3.7/site-packages/orger/org_view.py", line 86, in make_tree
    for p in self.get_items():
  File "/home/ashfaqf/Code/orger/modules/kobo.py", line 20, in get_items
    for page in get_books_with_highlights():
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 724, in get_books_with_highlights
    highlights = get_highlights(**kwargs)
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 675, in get_highlights
    return list(sorted(_iter_highlights(**kwargs), key=lambda h: h.created))
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 657, in _iter_highlights
    if h not in yielded:
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 212, in __hash__
    return hash(self._key)
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 206, in _key
    return (self.dt, self.text, self.annotation)
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 154, in dt
    return unwrap(_parse_utcdt(self.row['DateCreated']))
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/common.py", line 12, in unwrap
    assert x is not None
AssertionError

What am I doing wrong here?

[karlicoss]

Hi @ashfaqfarooqui . Glad you tried it!
I think you've done the setup right actually!

The exception has something to do with database structure, in particular DateCreated row is none for some reason, which I've never seen on my Kobo. Could you please insert a debug print here, in dt function, like this:

        print(self.row)
        return unwrap(_parse_utcdt(self.row['DateCreated']))

, and post the result please? That'll help me understand how to fix it properly.

In the meantime, I think I'll make processing more defensive and if datetime is missing in the database just display the highlight without it. I'll let you know when I'm done. I think you can also get around it for now if you modify the same function I mentioned above to something like:

    @property
    def dt(self) -> datetime:
        res = _parse_utcdt(self.row['DateCreated'])
        if res is None:
            return datetime.now(tz=pytz.utc) # temporary fallback to current time
        else:
            return res

P.S. Sorry for complicated setup with all the python packages etc. I've spent a while thinking about it and it's simplest what I got so far (considering you need side packages, private configuration etc). Can you please share what would be your ideal way of configuring/setting up? Vague thoughts are fine too even if you don't have something specific in mind. That would help me a lot in understanding how to simplify it further :)

[ashfaqfarooqui]

This is what I get with the print statement:

OrderedDict([('BookmarkID', '270d0b05-51ff-4311-a678-3b1079c22aaf'), ('VolumeID', '2f91da6c-aac3-4ac2-906b-f7fdacad52c2'), ('ContentID', '2f91da6c-aac3-4ac2-906b-f7fdacad52c2!OEBPS!ch8499.html'), ('StartContainerPath', 'span#kobo\\.31\\.4'), ('StartContainerChildIndex', -99), ('StartOffset', 117), ('EndContainerPath', 'span#kobo\\.31\\.5'), ('EndContainerChildIndex', -99), ('EndOffset', 141), ('Text', '<blah blah quote>'), ('Annotation', None), ('ExtraAnnotationData', None), ('DateCreated', None), ('ChapterProgress', 0.75), ('Hidden', 'false'), ('Version', '2519072456649581271'), ('DateModified', '2017-05-17T21:05:35Z'), ('Creator', None), ('UUID', None), ('UserID', '61a97132-4c62-4105-9c66-4f0286a95fc7'), ('SyncTime', '2017-05-21T16:46:17Z'), ('Published', 'false')])

Regarding the installation, the best case scenario would be to install the different packages via the pip package manager, and then provide the configuration file(json/yaml/ any standard config template) as an input parameter.
A quicker fix, I think, would be to bundle up all the repositories into one (maybe, as git-submodules). In this way, the user would need to clone a single repository without having to think about relative paths and stuff...
Then again, this is from a usage perspective; I don't know what impact it would have if you use it as a library.

PS: The program could run with the function you provided.

[ashfaqfarooqui]

I guess this issue should be in Kobuddy, anyway, I got a similar error while running kobuddy to when querying the progress:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__main__.py", line 55, in <module>
    main()
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__main__.py", line 45, in main
    print_progress()
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 788, in print_progress
    for bevents in get_books_with_events(**kwargs):
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 777, in get_books_with_events
    return list(sorted(iter_books_with_events(**kwargs), key=lambda be: be.last))
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 772, in iter_books_with_events
    for book, events in group_by_key(evts, key=lambda e: e.book).items():
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/common.py", line 36, in group_by_key
    for i in l:
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 689, in iter_events
    for x in _iter_events_aux(**kwargs):
  File "/home/ashfaqf/Code/orger/my/my_configuration/repos/kobuddy/src/kobuddy/__init__.py", line 526, in _iter_events_aux
    assert zz[1::2] == b'eventTimestamps'
AssertionError

[karlicoss]

Ok, so I did some enhancements:

1. more defensive Highlight.dt handling. #2 here I am falling back for dt method, so hopefully you wouldn't need 'temporary fallback to current time' I posted above anymore when you run orger module.

2. Defensive strategy for handling Kobo events #3 I added a special flag for choosing error handling strategy, e.g. you can run ./kobuddy --errors return --db /path/to/db progress, and you should hopefully get most of your stuff processed. It prints errors at the end (instead of throwing immediately as the default does), you can suppress them by shell redirection (2>/dev/null). That should help while I'm working on fixing errors properly.

3. And as for fixing your progress error properly, it would be hard to diagnose just by assert, so I added the bytearray to exception context. If you use the command above it would log something like

   2019-10-15 23:27:19 - kobuddy - ERROR - bytearray(b'\x00\x00\x00\x01\x00\x00\x00\x1e\x00e\x00v\x00e\x00n\x00t\x00T\x00i\x00m\x00e\x00s\x00t\x00a\x00m\x00p\x00s\x00\x00\x00\t\x00\x00\x00\x00\x02\x00\x00\x00\x03\x00[WB\x0e\x00\x00\x00\x03\x00]\x99\xf2%')
   

   Can you post it if you don't mind? It shouldn't be anything sensitive, just bunch of timestamps. Or feel free to email me if you're paranoid about publishing it here.

By the way, what's the Kobo device you are using? Would be good to keep track which ones the library supports.

Thanks for spending time on this, it helps me and other people :)

[ashfaqfarooqui]

1. That fixed the timestamp error.

2. Here is the errors that were thrown for progress:

2019-10-16 08:38:26 - kobuddy - ERROR - bytearray(b'\x00\x00\x00\x03\x00\x00\x004\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00T\x00i\x00m\x00e\x00E\x00l\x00a\x00p\x00s\x00e\x00d\x00\x00\x00\n\x00\x00\x00\x00\x02\x000\x00\x00\x00(\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00C\x00o\x00u\x00n\x00t\x00\x00\x00\x02\x00\x00\x00\x00\x03\x00\x00\x00.\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00R\x00e\x00a\x00d\x00i\x00n\x00g\x00S\x00e\x00c\x00o\x00n\x00d\x00s\x00\x00\x00\n\x00\x00\x00\x00\x02\x000')
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - bytearray(b'\x00\x00\x00\x03\x00\x00\x004\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00T\x00i\x00m\x00e\x00E\x00l\x00a\x00p\x00s\x00e\x00d\x00\x00\x00\n\x00\x00\x00\x00\x02\x000\x00\x00\x00(\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00C\x00o\x00u\x00n\x00t\x00\x00\x00\x02\x00\x00\x00\x00\x03\x00\x00\x00.\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00R\x00e\x00a\x00d\x00i\x00n\x00g\x00S\x00e\x00c\x00o\x00n\x00d\x00s\x00\x00\x00\n\x00\x00\x00\x00\x02\x000')
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 47), ('EventCount', 28), ('LastOccurrence', '2019-04-10T22:00:08.596'), ('ContentID', '6b68b81e-f0fa-430f-b687-be547a1f9b87'), ('Checksum', 'ce85437e3a05405ecd52cbfcf8041a4a'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E007400000002000000001C0000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 48), ('EventCount', 9), ('LastOccurrence', '2019-04-10T23:00:06.710'), ('ContentID', '6b68b81e-f0fa-430f-b687-be547a1f9b87'), ('Checksum', 'c2e4285c9ce7f69278f09578a67e2f02'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000090000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 48), ('EventCount', 3), ('LastOccurrence', '2019-04-10T23:27:26.890'), ('ContentID', '881e921f-6b35-45e9-a808-ee61854e8e9b'), ('Checksum', 'a0a2f93155fd9e8d6ad26d1897b58dc7'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000030000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 56), ('EventCount', 1), ('LastOccurrence', '2019-04-11T07:09:37.718'), ('ContentID', '6b68b81e-f0fa-430f-b687-be547a1f9b87'), ('Checksum', 'ecb2e1b887f64967b8ba1870ae8d6d85'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000010000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 67), ('EventCount', 1), ('LastOccurrence', '2019-04-11T18:24:28.818'), ('ContentID', '6b68b81e-f0fa-430f-b687-be547a1f9b87'), ('Checksum', '3e958727a90fe629afe5c3b5aeac1cef'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000010000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 67), ('EventCount', 1), ('LastOccurrence', '2019-04-11T18:45:21.253'), ('ContentID', 'b7341500-3ca2-466c-b27c-40978837fd28'), ('Checksum', '180e10076261afc6457af3b9dae0e934'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000010000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 70), ('EventCount', 1), ('LastOccurrence', '2019-04-10T21:35:50.046'), ('ContentID', '6b68b81e-f0fa-430f-b687-be547a1f9b87'), ('Checksum', '117d3a193bd79ef8973131dc5aceccc1'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000010000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - bytearray(b'\x00\x00\x00\x03\x00\x00\x004\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00T\x00i\x00m\x00e\x00E\x00l\x00a\x00p\x00s\x00e\x00d\x00\x00\x00\n\x00\x00\x00\x00\x02\x000\x00\x00\x00(\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00S\x00y\x00n\x00c\x00e\x00d\x00C\x00o\x00u\x00n\x00t\x00\x00\x00\x02\x00\x00\x00\x00\x0b\x00\x00\x00.\x00E\x00x\x00t\x00r\x00a\x00D\x00a\x00t\x00a\x00R\x00e\x00a\x00d\x00i\x00n\x00g\x00S\x00e\x00c\x00o\x00n\x00d\x00s\x00\x00\x00\n\x00\x00\x00\x00\x02\x000')
NoneType: None
2019-10-16 08:38:26 - kobuddy - ERROR - OrderedDict([('EventType', 67), ('EventCount', 6), ('LastOccurrence', '2019-05-23T18:22:46.766'), ('ContentID', '09542b83-294d-4e59-bd1a-34f7ade8783c'), ('Checksum', '15c47c07b5b03f1f6166c3be6a56b9b1'), ('hex(ExtraData)', '000000030000003400450078007400720061004400610074006100530079006E00630065006400540069006D00650045006C006100700073006500640000000A000000000200300000002800450078007400720061004400610074006100530079006E0063006500640043006F0075006E00740000000200000000060000002E00450078007400720061004400610074006100520065006100640069006E0067005300650063006F006E006400730000000A00000000020030')])
NoneType: None

Hope that helps.

I am using the Kobo Aura one.

Glad to be of help :)

[karlicoss]

Thanks. From a quick glance doesn't look like these don't contain anything useful (you can read ExtraDataSyncedTimeElapsed/ExtraDataSyncedCount/ExtraDataReadingSeconds and all the other bytes are zeros. I'll see how to handle it properly so it doesn't complain.

[karlicoss]

Improved decoding so hopefully progress wouldn't spam with errors anymore! #4