core: categories cleanup

[ngosang]

• swap categories 2050 Movies/BluRay / 2060 Movies/3D
• swap categories 6050 XXX/Pack / 6070 XXX/Other
• swap categories 7010 Books/Mags / 7020 Books/EBook / 7030 Books/Comics
• category validation is case sensitive
• renamed some categories to follow Newznab specs

[ngosang]

@garfield69 take a look at this PR.
Category renames are fine but I have found more categories not following this specs => https://newznab.readthedocs.io/en/latest/misc/api/#predefined-categories

[ngosang]

@Taloth Sorry to bother you but we need an expert in Newznab / Torznab.
Recently we have found that Jackett categories are different from this specification => https://newznab.readthedocs.io/en/latest/misc/api/#predefined-categories

• Should we follow those specs?
• Torznab specs say something different about categories?
• We have some "unofficial" categories like 1180, "Console/PS4" because Newznab is a bit outdated. Is there a new version of Newznab / Torznab specification with newer categories?

[Taloth]

The best approach there is to look at newznab and nzedb for those special categories and just pick something that's best. (https://github.com/nZEDb/nZEDb/blob/b485fa326a0ff1f47ce144164eb1f070e406b555/resources/db/schema/data/10-categories.tsv)
For example nzedb has PS3 releases as 1080, and 1180 for PS4.
You can also look at existing newznab indexers and see if they already have the category and use that.
I checked and one private one uses 1085 for PS4, and 1090 for XBOX ONE.
Another one, oznzb seems to be newznab-based and has 1100 as PS4.
I suspect nzedb started using 11xx for 'new' categories to avoid conflicting with new newznab ones in the near future, but that's gonna overlap pretty soon.
Regardless, this is one of the reasons I added the dropdown in Sonarr, so it doesn't really matter.

For the categories specified in the newznab documentation you definitely should try to follow it, because it means defaults like 5070 will work properly.
It's also the reason why when Jackett first started out I recommended to put the indexer specific categories in the 100000 range, which is being done for the bulk of jackett indexers. Each indexer specific category then gets aliased to one or more main categories like 5070. (Categories are like tags, a release can have multiple, in fact they always do, because releases in 5070 are also in 5000) It's rarely a 1 to 1 mapping.

The torznab spec refers to newznab and only adds a few new attributes for seeds, not categories. So there's no 'one truth'. It's perhaps possible to contact bb and see if the official list can be extended. But there's already some divergence so it probably doesn't matter much.
Note that nzedb is essentially a fork/competitor of newznab so it might not be easy to unify something.

Edit to add: I just saw you swapped 2050 and 2060... That's clearly newznab vs nzedb. nzedb has them inverted.

[ngosang]

The best approach there is to look at newznab and nzedb for those special categories and just pick something that's best.
Edit to add: I just saw you swapped 2050 and 2060... That's clearly newznab vs nzedb. nzedb has them inverted.

Since there is not a clear standard I think we should follow Newznab as close as possible and add new categories when required. This PR is moving in that direction. @garfield69 We should update the Wiki to add reference links to make it clear what specs we are following.

You can also look at existing newznab indexers and see if they already have the category and use that.

I don't have access, but I think it's not really important since they are not standard. Will be interesting to see if they have more new categories. @Taloth could you share the caps of some newznab indexers so we can add newer cats?

It's also the reason why when Jackett first started out I recommended to put the indexer specific categories in the 100000 range, which is being done for the bulk of jackett indexers. Each indexer specific category then gets aliased to one or more main categories like 5070. (Categories are like tags, a release can have multiple, in fact they always do, because releases in 5070 are also in 5000) It's rarely a 1 to 1 mapping.

We are already adding custom categories in most trackers but there are some bugs in the code like #9746 I'm working on it.

Offtopic: the Torznab caps in Jackett are wrong too. In one indexer with only 2 categories (Movie/HD/Bluray and Movie/Xvid) we are generating this tree.

<categories>
  <category id="100048" name="Movie/HD/Bluray"/>
  <category id="100007" name="Movie/Xvid"/>
  <category id="2000" name="Movies">
    <subcat id="2010" name="Movies/Foreign"/>
    <subcat id="2020" name="Movies/Other"/>
    <subcat id="2030" name="Movies/SD"/>
    <subcat id="2040" name="Movies/HD"/>
    <subcat id="2045" name="Movies/UHD"/>
    <subcat id="2050" name="Movies/3D"/>
    <subcat id="2060" name="Movies/BluRay"/>
    <subcat id="2070" name="Movies/DVD"/>
    <subcat id="2080" name="Movies/WEBDL"/>
  </category>
  <category id="2030" name="Movies/SD"/>
  <category id="2040" name="Movies/HD"/>
</categories>

It's wrong because we are adding all subcategories of Movies (most of them are not supported) and some categories like Movies/SD are duplicated. With the fix I'm going to push soon the new tree will be:

<categories>
  <category id="2000" name="Movies">
    <subcat id="2030" name="Movies/SD"/>
    <subcat id="2040" name="Movies/HD"/>
  </category>
  <category id="100048" name="Movie/HD/Bluray"/>
  <category id="100007" name="Movie/Xvid"/>
</categories>

This should solve some issues in Sonarr category tree.

@garfield69 If you agree you can merge this PR.

[Taloth]

I don't have access, but I think it's not really important since they are not standard. Will be interesting to see if they have more new categories. @Taloth could you share the caps of some newznab indexers so we can add newer cats?

I could, but most indexers do not require an apikey:

https://nzbgeek.info/api?t=caps (newznab 1.0.0, not sure if that's an actual newznab version)
https://api.oznzb.com/api?t=caps (newznab 0.2.3p)
https://drunkenslug.com/api?t=caps (nzedb 0.8.21)
https://nzb.cat/api?t=caps (nzedb 0.8.19)

We are already adding custom categories in most trackers but there are some bugs in the code like #9746 I'm working on it.

Right, so you have to map the string category to an internal number, and alias that to an official newznab category.
Why I mentioned this is because those 100000 categories show up in the dropdown, and could be useful for indexers that don't have categories at all or odd categories that don't map 1-to-1. Like the 8000 problem. You could add 100001 "All" and then alias those releases in both Movies and TV.
It doesn't really matter since the users has to change the category anyway because Sonarr uses 5030 and 5040 as default.

Here's what we did for hdaccess (no long exists, but was one of the two trackers that natively supported torznab).

Offtopic: the Torznab caps in Jackett are wrong too. In one indexer with only 2 categories (Movie/HD/Bluray and Movie/Xvid) we are generating this tree.
...

It's wrong because we are adding all subcategories of Movies (most of them are not supported) and some categories like Movies/SD are duplicated. With the fix I'm going to push soon the new tree will be:

<categories>
  <category id="2000" name="Movies">
    <subcat id="2030" name="Movies/SD"/>
    <subcat id="2040" name="Movies/HD"/>
  </category>
  <category id="100048" name="Movie/HD/Bluray"/>
  <category id="100007" name="Movie/Xvid"/>
</categories>

This should solve some issues in Sonarr category tree.

It's not necessarily wrong to have subcategories that don't contain any releases, in fact it could be useful in some cases. It's like "Yes I recognize this category". In fact, by default Sonarr only selects 5030 and 5040.
But yes, the dupe would be an issue.
Are those Movie/HD/Bluray releases actually transcoded bluray (blurip) or raw untouched bluray? afaik that's what determines if it's 2040 or 2060.

[garfield69]

https://github.com/Jackett/Jackett/wiki/Jackett-Categories

[garfield69]

the dashboard continued to use the old names for categories until I used Shift-F5 to flush the Chrome browser cache
do we need to add a dummy ?changed=2020103001 to a script src reference in the index.html to force a refresh for the next update?

[ngosang]

do we need to add a dummy ?changed=2020103001 to a script src reference in the index.html to force a refresh for the next update?

That's not going to work and it's not necessary. Maybe some of them need to close and reopen the browser to see the new cats but it's not a big deal. 😃