Use better pagination (and implement progress bar)

[simonw]

Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow.

We can do better by implementing pagination using count and offset.

[simonw]

Prototype code:

offset = 0
fetched = []
size = 500
while True:
    page = requests.get("https://getpocket.com/v3/get", {
        "consumer_key": consumer_key,
        "access_token": access_token,
        "sort": "oldest",
        "detailType": "complete",
        "count": size,
        "offset": offset,
    }).json()
    print(offset)
    fetched.append(page)
    offset += size
    if not len(page["list"]):
        break

[simonw]

For a progress bar I need to know how many total items there are.

I found an undocumented API for this! /v3/stats which returns:

{
    "count_list": 7394,
    "count_read": 1016,
    "count_unread": 6378,
    "status": 1
}

I guessed this based on the documented v2 API: https://getpocket.com/api/v2_docs/#stats

[simonw]

I keep getting 503 errors even though I appear to be staying within the rate limit:

{'Date': 'Fri, 27 Mar 2020 21:35:57 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding':
'chunked', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location':
'get.php', 'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...;  httponly',
'X-Frame-Options': 'SAMEORIGIN', 'Status': '200 OK', 'X-Limit-Key-Limit': '10000',
'X-Limit-Key-Remaining': '9960', 'X-Limit-Key-Reset': '282', 'X-Source': 'Pocket',
'P3P': 'policyref="/w3c/p3p.xml", CP="ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE"'}

  [##----------------------------------]    6%  06:49:27

{'Date': 'Fri, 27 Mar 2020 21:36:06 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Content-Length':
'23', 'Connection': 'keep-alive', 'Server': 'Apache/2.4.25 (Debian)', 'Content-Location': 'get.php',
'Vary': 'negotiate', 'TCN': 'choice', 'Set-Cookie': '...', 'X-Frame-Options': 'SAMEORIGIN',
'X-Error': 'Pocket is currently under heavy load. Please wait a moment and try again.',
'X-Error-Code': '199', 'Status': '503 Service Unavailable', 'X-Source': 'Pocket',
'P3P': 'policyref="/w3c/p3p.xml", CP="ALL CURa ADMa DEVa OUR IND UNI COM NAV INT STA PRE"'}

I'm going to try doing a few automatic retries any time I see a 503 error.

[simonw]

Or maybe it was because of the current Google Cloud outage? https://news.ycombinator.com/item?id=22706677