New API endpoints

[Varstahl]

Amazon introduced new api endpoints that could be used to scrape instead of the HTML counterparts. Some information are missing, so we need to evaluate whether moving the system to JSON makes sense or not. Note to self: absolutely avoid last time's bloodshed with the mix of Amazon-JSON-UI and HTML, should they decide to remove or block access to the API endpoints, again.

Watchlist, ignoring non-available content
Homepage

[Paco8]

I think they added this for searching:

https://www.primevideo.com/gp/video/api/search?phrase=the%20expanse&startIndex=0&queryToken=&pageId=&queryPageType=&infiniteScrollEnabled=1&totalItems=33&isRetail=0

[Paco8]

It's also possible to get the listings of the sections in the home page (the "see more" links) using that search URL, passing the queryToken from those links.

[Varstahl]

Neat. Had stuff to do with new servers and technologies, I still plan to take a deep look at this, it might just take me a while to get my other stuff done first.

[Varstahl]

I just started working on this, wip branch located here. Let's see where this bumpy ride leads us, hopefully to significantly faster load times.

[Varstahl]

So… found a few nice points:

• /api/storefront/?contentType=watchlist completely breaks their backend :D
• /api/storefront/?contentType=movie works
• /api/storefront/?contentType=tv works
• /api/storefront/?contentType=kids doesn't work

So this means that:

1. if it doesn't paginate, it doesn't have an api endpoint
2. everything is initially downloaded as HTML
3. from pagination, stuff is (mostly) managed with api endpoints
4. series/movies page are only rendered in HTML

So, at this point, is it even worth the time?

Also, while doing a bit of reversing I found the following homepages, not available everywhere:

• DEFAULT("default", ActivityMarkers.HOMEPAGE_DEFAULT),
• HOME("home", ActivityMarkers.HOMEPAGE_HOME),
• KIDS("kids", ActivityMarkers.HOMEPAGE_KIDS),
• MOVIE("movie", ActivityMarkers.HOMEPAGE_MOVIE),
• ORIGINALS("originals", ActivityMarkers.HOMEPAGE_ORIGINALS),
• TV("tv", ActivityMarkers.HOMEPAGE_TV);

Honestly the API don't seem very useful so far, although they have a couple nice features we can't simply use them. Which begs the question why change in the first place, aside the fact it's currently still a mess. Don't know, I'll sleep on it.

Edit: actually the home pages have the JSON embedded in a var args, could parse that. Still the problem remains for single destination pages and navigation.

Edit 2: menu is embedded in a <script type="text/template">{"props":{"mainMenu":{}}</script>.

Edit 3: most of the metadata, including stuff that isn't actually shown to the user, can also be found in several props text/template. Guess we can finally go JSON only…

[Varstahl]

Amazon, being Amazon and in pure Amazon fashon, didn't include basic info needed to determine the basics of movie/season, which means that on average we're looking at 15 calls to the server per page (at a minimum). Oh well, we can (probably) defer mostl parsing though, so it should still be ok? :\

[Varstahl]

If someone feels brave enough to try, I have a (mostly) working PoC live on my fork. You can grab primevideo.py from here and replace it in your addon installation. You should probably backup your cache files as well if you want to be able to revert, since the new version will wipe out all cache.

[Paco8]

I guess the watchlist isn't implemented yet, is it?
It's empty for me.

[Varstahl]

It was working at some point, then it stopped working. I'm unsure of what I did (or didn't do), but it should eventually come back, this is super super early.

[Paco8]

I'm afraid this version is not working for me at all. The watchlist is still empty. When trying to browse the other sections I always get this error:

21:53:04.028 T:324   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.KeyError'>
                                            Error Contents: u'type'
                                            Traceback (most recent call last):
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\default.py", line 5, in <module>
                                                EntryPoint()
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\resources\lib\startup.py", line 58, in EntryPoint
                                                elif 'browse' == verb: g.pv.Browse(path)
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\resources\lib\primevideo.py", line 389, in Browse
                                                node, breadcrumb = self._TraverseCatalog(path)
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\resources\lib\primevideo.py", line 312, in _TraverseCatalog
                                                self._LazyLoad(node[nodeName], pathList[0:1 + i])
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\resources\lib\primevideo.py", line 993, in _LazyLoad
                                                page = next((x['href'] for x in cnt['pagination']['paginator'] if x['type'] == 'NextPage'), None)
                                              File "C:\Users\Paco\AppData\Roaming\Kodi\addons\plugin.video.amazon-test\resources\lib\primevideo.py", line 993, in <genexpr>
                                                page = next((x['href'] for x in cnt['pagination']['paginator'] if x['type'] == 'NextPage'), None)
                                            KeyError: u'type'
                                            -->End of Python script error report<--

[Varstahl]

I didn't complete the watchlist. I don't have that error, but as I said previously I might have a working version. Really had no time to complete this, I'll get back to working when I can.

[Paco8]

Maybe I could help (although my knowledge about python is almost zero).

I think I fixed that crash replacing the line

page = next((x['href'] for x in cnt['pagination']['paginator'] if x['type'] == 'NextPage'), None)

with

page = None
for x in cnt['pagination']['paginator']:
    if ('type' in x and x['type'] == 'NextPage'): page = x['href']
    elif ('*className*' in x and x['*className*'] == 'atv.wps.PaginatorNext'): page = x['href']

[Varstahl]

Thanks for the fix, I'll see if it applies to the current build when I can.

[Goommer]

Hi,
first at all thanks for your works
seconds a little disclaimer: My english, python and kodi knowledge are very poor

With this patch i can view series, seasons and episodies and also i can play it.
For sure it has a lot of errors but it works for me
A great problem is that into each season, except the first, all other seasons show as one more episode

I hope that helps
series_working_but.diff.txt

[Varstahl]

Thanks for the patch. I wanted to complete this new version a couple months ago, but I've been overwhelmed so far. I hope to check it asap.

[Goommer]

Don't worry, now I can play series and I can wait quietly for your new version.

I have fork your repository to submit changes, it will surely be more comfortable.

Varstahl/xbmc@add-pv-api...Goommer:goommer_mods

Thanks a lot

[Varstahl]

Ugh, seems that while I was gone for work like literally everything broke. Gotta get grok'in'.

[Goommer]

Hi again

I release my last modifications.
I guess it won't help you too much but at least it worked for me the last few weeks.

what's add:

• watchlist
• watchlist management, add to and delete from watchlist
• new, special and live series and movies
• refresh, everything except new, special and live series and movies (last commit)

Varstahl/xbmc@add-pv-api...Goommer:goommer_mods

I look forward to your release to learn by comparing it with mine.

Thanks a lot

[Varstahl]

Nice, I'll take a look. At the moment I'm getting demolished by python3 compatibility, with the new release everything broke but I'll make sure to include the new features in.

[Varstahl]

@Goommer I just got around to a workable state after the adaptation for the python 3 support, and I wanted to thank you for your effort. I didn't check and implement everything yet, and I don't know if I can merge everything as is, but the pagination fix is already in. It might take me a while to check everything, but I want to stress that your help is really appreciated.

Edit: before I forget, the current code with the new endpoints is now here.