Skip to content

Fix Pinboard JSON duplicate timestamps error#30

Merged
pirate merged 2 commits intoArchiveBox:masterfrom
bardisty:fix-pinboard-json-timestamps-error
Jul 1, 2017
Merged

Fix Pinboard JSON duplicate timestamps error#30
pirate merged 2 commits intoArchiveBox:masterfrom
bardisty:fix-pinboard-json-timestamps-error

Conversation

@bardisty
Copy link
Copy Markdown
Contributor

@bardisty bardisty commented Jul 1, 2017

If the JSON exported by Pinboard contains duplicate timestamps, Python
returns a TypeError exception:

TypeError: argument of type 'float' is not iterable

This is because time.mktime() returns a floating point number.
Encasing time.mktime() in str() fixes the data type not being
iterable.

time.mktime() has also been encased in int() to remove the
unnecessary decimal value (.0) that gets returned for each time value,
and to keep the script consistent with the other export functions.

Full Error:

$ env CHROME_BINARY=google-chrome-stable ./archive.py ~/tmp/test
[+] [2017-07-01 02:57:35] Starting archive from /home/bah/tmp/test export file.
Traceback (most recent call last):
  File "./archive.py", line 468, in <module>
    create_archive(export_file, service=export_type, resume=resume_from)
  File "./archive.py", line 412, in create_archive
    links = uniquefied_links(links)         # fix duplicate timestamps, returns sorted list
  File "./archive.py", line 283, in uniquefied_links
    link['timestamp'] = next_uniq_timestamp(seen_timestamps, link['timestamp'])
  File "./archive.py", line 251, in next_uniq_timestamp
    if '.' in timestamp:
TypeError: argument of type 'float' is not iterable

JSON test data:

[{"href":"https:\/\/pushover.net\/","description":"Pushover: Simple Notifications for Android, iOS, and Desktop","extended":"","meta":"1e68511234d9390d10b7772c8ccc4b9e","hash":"bb93374ead8a937b18c7c46e13168a7d","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"app android"},
{"href":"http:\/\/www.reddit.com\/r\/Android","description":"r\/android","extended":"","meta":"18a973f09c9cc0608c116967b64e0419","hash":"910293f019c2f4bb1a749fb937ba58e3","time":"2014-06-14T15:51:42Z","shared":"no","toread":"no","tags":"reddit android"}]

Output after fix:

$ env CHROME_BINARY=google-chrome-stable ./archive.py ~/tmp/test           
[+] [2017-07-01 04:09:17] Starting archive from /home/bah/tmp/test export file.
[*] [2017-07-01 04:09:17] Created archive index with 2 links.
[*] Checking Dependencies:
/bin/google-chrome-stable
/bin/wget
/bin/curl
[+] [1402786302 (2014-06-14 15:51:42)] "Pushover: Simple Notifications for Android, iOS, and Desktop": pushover.net/
    - Downloading Full Site
    - Printing PDF
      Exception: Exception Failed to print PDF
    - Snapping Screenshot
    - Submitting to archive.org
    - Fetching Favicon
[+] [1402786302.1 (2014-06-14 15:51:42)] "r/android": www.reddit.com/r/Android
    - Downloading Full Site
    - Printing PDF
      Exception: Exception Failed to print PDF
    - Snapping Screenshot
    - Submitting to archive.org
    - Fetching Favicon
[√] [2017-07-01 04:09:39.035009] Archive update complete.

If the JSON exported by Pinboard contains duplicate timestamps, Python
returns a TypeError exception:

`TypeError: argument of type 'float' is not iterable`

This is because `time.mktime()` returns a floating point number.
Encasing `time.mktime()` in `str()` fixes the data type not being
iterable.

`time.mktime()` has also been encased in `int()` to remove the
unnecessary decimal value (`.0`) that gets returned for each time value,
and to keep the script consistent with the other export functions.
@pirate
Copy link
Copy Markdown
Member

pirate commented Jul 1, 2017

Awesome PR, thanks so much!

Would you mind actually also commiting your test file under examples/pinboard.json. It's a useful testcase to have in the repo.

@pirate pirate merged commit 9f5fe72 into ArchiveBox:master Jul 1, 2017
@bardisty bardisty deleted the fix-pinboard-json-timestamps-error branch July 1, 2017 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants