Core: Add Basic JSON Parsing from API #12424
Conversation
There was a problem hiding this comment.
Looking good. I like what I see so far.
I've updated the yts.yml to see how close I could get the results matching the yts.cs
next step would be to:
- introduce filters
- allow fields without selectors (such as needed for downloadvolumefactor
- add a requestdelay parm as it seems most api's rate limit their intake and we may need to regulate this in the yml
Later I could start adding more test yml and that may help see what other features may be required.
|
Thanks a lot for the testing @garfield69. Thanks to you, some quality features are being added :) |
|
Nice! Little niggle found. Searching yts-yml for {{"status":"ok","status_message":"Query was successful","data":{"movie_count":0,"limit":50,"page_number":1},"@meta":{"server_time":1634408056,"server_timezone":"CET","api_version":2,"execution_time":"0 ms"}}apparently YTS do not provide the expected empty set bit-titan in comparison returns {"siteUrl":"https:\/\/bit-titan.net\/","siteName":"BiT-TiTAN","filesFound":"0","pages":0,"viewResults":"0 to 100 from 0","results":[]}nice of them to provide the empty set TPB returns [{"id":"0","name":"No results returned","info_hash":"0000000000000000000000000000000000000000","leechers":"0","seeders":"0","num_files":"0","size":"0","username":"","added":"0","status":"member","category":"0","imdb":"","total_found":"1"}]which is different again, clearly we cannot rely on sites' api providing empty sets (like bit-titan),. Maybe we can trap when the row selector is not found but some other field is provided that indicates that no results were returned, to distinguish genuine row selector not found error where the yml was coded incorrectly or the api was changed after the indexer was coded ? In the mean time I shall work on a couple more conversions that we can test with ;-) |
|
Great find!, I'll add a countSelector field to the block and add it before the row parser. |
|
@garfield69 Please care the dot notation according to the latest commit. The arrow notation was resulting in inconsistency so I removed it. |
|
Been working on try to get TPB-yml working for testing ... [{"id":53167695,"info_hash":"EE5F3CB04A68EF27A72825D77561FBCB3742C3DF","category":505,"name":"NubileFilms 21 10 16 Lilly Bella Take A Bite 720p mp4","status":"trusted","num_files":1,"size":485878644,"seeders":0,"leechers":0,"username":"Mesoglea","added":1634416932,"anon":0,"imdb":null},{"id":53167694,"info_hash":"906B1091590B494DDAF61B455B9EF2E7A81FC6BB","category":101,"name":"Time Life Music - The Complete 80's (+Bonus) 320k musicfromrizzo","status":"trusted","num_files":250,"size":2494136715,"seeders":0,"leechers":0,"username":"moviesbyrizzo","added":1634416698,"anon":0,"imdb":null},{"id":53167693,"info_hash":"AC3862636D51D761608CE87067AB8AEB2BA017CD","category":205,"name":"Mordkommissionen S02E01-07 svensk text","status":"trusted","num_files":7,"size":4184299787,"seeders":0,"leechers":0,"username":"Anonymous","added":1634416556,"imdb":null},{"id":53167692,"info_hash":"D3CC939FE6B84762EAC709D5410ADB4FB9E28052","category":505,"name":"AlettaOceanLive 21 10 15 Delivery Guy 720p mp4","status":"trusted","num_files":1,"size":315586774,"seeders":0,"leechers":0,"username":"Mesoglea","added":1634416506,"anon":0,"imdb":null},{"id":53167691,"info_hash":"CAA7062E573CBFC693C59093C8A5C4657BDCF910","category":205,"name":"Young Justice S04E02 XviD-AFG","status":"vip","num_files":6,"size":220151170,"seeders":0,"leechers":0,"username":"TvTeam","added":1634416290,"anon":0,"imdb":null},{"id":53167690,"info_hash":"34D527B41C914F1EC0F754A95AF9B458F110CA60","category":205,"name":"Young Justice S04E02 480p x264-mSD","status":"vip","num_files":6,"size":128636917,"seeders":0,"leechers":0,"username":"TvTeam","added":1634416282,"anon":0,"imdb":null},{"id":53167689,"info_hash":"6F03FDC69CBEFD910A37142D81E50A9CA8C90F61","category":205,"name":"Young Justice S04E01 XviD-AFG","status":"vip","num_files":6,"size":241661904,"seeders":0,"leechers":0,"username":"TvTeam","added":1634416275,"anon":0,"imdb":null},{"id":53167688,"info_hash":"719F9F79442CE5746AF393767D7458A1121B29FD","category":205,"name":"Young Justice S04E01 480p x264-mSD","status":"vip","num_files":6,"size":132407509,"seeders":0,"leechers":0,"username":"TvTeam","added":1634416268,"anon":0,"imdb":null},{"id":53167687,"info_hash":"E6F3B0548D28BC31C0C7AB73E20FC0ABF15C2001","category":205,"name":"Young.Justice.S04E01.XviD-AFG[TGx]","status":"vip","num_files":3,"size":240926475,"seeders":0,"leechers":0,"username":"Anonymous","added":1634416252,"imdb":null},{"id":53167686,"info_hash":"DA732064FB91D2C20702C4697DB1ECBEFC9D69D6","category":205,"name":"Young.Justice.S04E02.XviD-AFG[TGx]","status":"vip","num_files":3,"size":219299499,"seeders":0,"leechers":0,"username":"Anonymous","added":1634416039,"imdb":null},{"id":53167685,"info_hash":"4759BCED8A093816F10F4F9EF9D8C3F9024388A4","category":208,"name":"Magnum.P.I.2018.S04E03.Texas.Wedge.720p.AMZN.WEBRip.DDP5.1.x264-","status":"vip","num_files":2,"size":2027626373,"seeders":0,"leechers":0,"username":"Anonymous","added":1634416027,"imdb":null},
etc etc etcso for the row what do I specify as a selector in this case? also, regarding the count, if I need to access a field from the root, that is to say, a field that is before the set referenced by the row selector, how can I reference it? |
|
I think a simple Count selector refers the root object only, not the rows unless specified as a dot notation path. |
|
I've loaded thepiratebay.yml so you can take a looksee. It's WIP. rows:
selector: .caused error |
|
\o/ got a fully working version of bit-titan.yml |
|
\o/ full working version of milkie.yml |
|
additional feature request: standard:
selector: ..standard
case:
0: ""
1: "8K"
2: "2160p"
3: "1080p"
4: "1080i"
5: "720p"
6: "SD"
downloadvolumefactor:
selector: ..sp_state
case:
2: 0 # free
4: 0 # 2x free
5: 0.5 # 50% free
6: 0.5 # 2x 50% free
7: 0.3 # 30% free
"*": 1
uploadvolumefactor:
selector: ..sp_state
case:
3: 2 # 2x
4: 2 # 2x free
6: 2 # 2x 50% free
"*": 1Thanks. |
Sorry, |
|
Honestly, I don't have much idea for case block. I'm thinking it as more of multiple matches and replace things and have accordingly pushed a commit. Please test it and if I'm wrong, can you please post some examples too so I can know what can be achieved through the case block. |
add default for case block
|
Re: #1774 their API will always add a blank result at the end which results in the error Adding Similarly, adding When there are no results (e.g. searching |
|
@ilike2burnthing can I see the indexer you've made for this? |
|
|
@ilike2burnthing I understand now what you are saying. @mikeoscar2006 This would be quite a powerful feature to add. but this might be quite a large undertaking to develop. Does the JToken from Newtonsoft.Json.Linq have comparators and query capabilities? so xbit generates a response like this: {
"dht_results": [{"ID":"16273799",
"NAME":"www.ds1024.xyz 极品无毛一线天白虎萝莉从蛋蛋后直播大秀 穿着可爱睡衣",
"MAGNET":"magnet:?xt=urn:btih:8904e34eb8331463927a82aac181a043504656cc&dn=www.ds1024.xyz+%E6%9E%81%E5%93%81%E6%97%A0%E6%AF%9B%E4%B8%80%E7%BA%BF%E5%A4%A9%E7%99%BD%E8%99%8E%E8%90%9D%E8%8E%89%E4%BB%8E%E8%9B%8B%E8%9B%8B%E5%90%8E%E7%9B%B4%E6%92%AD%E5%A4%A7%E7%A7%80+%E7%A9%BF%E7%9D%80%E5%8F%AF%E7%88%B1%E7%9D%A1%E8%A1%A3",
"SIZE":"1.15GB",
"DISCOVERED":"2021-10-19 10:37:29" },
{}]}where they include an empty set at the end of the results. So perhaps an alternative might be to just ignore an empty set if we have already processed one other set previously? |
|
@garfield69 Can you give few examples so I can understand what exactly we can achieve with PS: Loving the response. Didn't know so much things would be solved with this addition 🤩 |
|
examples: {
"data": [
{
"name": "a regular title with a magnet and freeleech",
"magnet": "magnet:?xt=urn:btih:8904e34eb8331463927a82aac181a043504656cc",
"free": "1"
},
{
"name": "a VIP title with a magnet and not free",
"magnet": "magnet:?xt=urn:btih:8004e34eb8331463927a82aac181a043504656cc",
"free": "0",
"vip": "true"
},
{
"name": "a title without a magnet",
"free": "0"
}
]
}you could code rows:
selector: data:has(magnet):not(vip):has(free:contains("1"))
fields:
title:
selector: name
download:
selector: magnet
donwloadvolumefactor:
selector: free
case:
0: 1 # not freeleech
1: 0 # freeleechand this would drop any row that does not include a selector with the name I hope I have not made any mistakes in the presentation and that is make sense. the :has(), :not(), :contains() would also operate at the field selector level to allow fields with the same name to be processed or skipped (in combination with the rows:
selector: data
fields:
date:
selector: published:contains("-")
optional: true
filters:
- name: dateparse
args: "2006-01-02"
date:
selector: published:not(:contains("-"))
optional: true
filters:
- name: timeagowhile the latter is strictly a violation of yaml guidelines (duplicate date keys) its in line with existing yaml indexers that have this type of logic. |
|
Got it. AFAIK, There are filter expressions that are not as simple as Can only think of these two options rn. lmk which one do you think is better and should we implement it in this PR or a separate one after merging this. |
I think the current PR is ready for general use, and will allow us to publish the bulk of those json indexer that work within the current PR framework.
the former has the advantage of looking similar to the existing HTML based indexers so that would be my preferred option, but I can see how a filter block might be easier to implement and TBH as long as either option works I am not too fussed if the latter method is implemented instead. selector: name
filters:
- name: contains
args: "string"and similarly for albeit, I don't see how the |
|
Same as you, if they're equally easy to implement then |
|
Well if the former approach is preferable then I'll definitely start with that and if not implementable, I'll fall back to the latter approach. |
|
Looks great. I'm almost done with the overhaul. I'll be publishing it today for your final review and then we can merge this. |
|
Looks good. Should we bump to v0.19 for this? |
Resolves #12334.
Note: It is still in a PoC state. After adding more features, a final overhaul will be done to improve the code quality.