Add template filter to parse human-readable file size to bytes

[mnixry]

Motivation for changes:

Some of the plugin, for example rss does not produce the torrent_size entry field, but for some sites, it actually contains human-readable size like 114.5MB, 14GB, or 1.91TB in the torrent title.
In some cases, we need to filter out the torrents that are larger or smaller than we expected size. In this case, a filter to parse human-readable string to actual bytes number is necessary.

Detailed changes:

• Added a jinja filter parse_size to support parse the string like 114.5GB to 122943438848 bytes int

Config usage if relevant (new plugin or updated schema):

tasks:
  download_filtered_size:
    rss: https://example.com/?token=xxxx
    accept_all: yes
    no_entries_ok: yes
    if: # rejects torrents size which are larger than 100GB
      - "title|re_search('\d+(\.\d+)?\s*\w+B')|parse_size > 100 * (1024**3)": reject

To Do:

Current PR is a proof of concept, if you have any ideas about it, welcome comment.

• tests
• document

[gazpachoking]

I like the idea. Maybe we should remove the anchors from the regex so that less regex in the template itself is needed? And maybe we make an optional parameter to control whether it assumes SI units or not, e.g. 1 kb|parse_size == 1024, but 1 kb|parse_size(si=true) == 1000. kib would of course always be 1024.

[mnixry]

Maybe we should remove the anchors from the regex so that less regex in the template itself is needed? And maybe we make an optional parameter to control whether it assumes SI units or not

@gazpachoking Updated. Please check again and feel free to comment :)

P.S. should I add a specified test for this filter? I found most of the filters have no specified test except stripyear.

[gazpachoking]

Couple small comments, but looks good to me! And I think a few tests would be a great idea if you are up for it.

[gazpachoking]

I thought of one more question... should it be case sensitive? I'm thinking people will only use this filter on fields that are very likely to have a size in it, so we probably don't need to be so strict about the casing.

[mnixry]

I thought of one more question... should it be case sensitive? I'm thinking people will only use this filter on fields that are very likely to have a size in it, so we probably don't need to be so strict about the casing.

Updated. I've added an argument that can let user match case-insensitive, and false by default. Case-insensitive matching will significantly increase the wrong match rate.

Considering a title like this, which are really common in most RSS feed:

Symphony No.9 - Ludwig van Beethoven [FLAC][96kbps][5.78G]

A case-insensitive matching will match the sample rate 96kb instead of the actual size 5.78G.

[gazpachoking]

That's a good point. We probably could have gone with it several iterations ago, but I can't help tweaking. 😄 What about a negative lookahead at the end of the regex to eliminate that kind of error? Something like (?![A-Za-z]). Or maybe just use a word boundary at the end \b

[mnixry]

What about a negative lookahead at the end of the regex to eliminate that kind of error? Something like (?![A-Za-z]). Or maybe just use a word boundary at the end \b

I adjusted the regex with lookahead syntax, it should work without wrong match:

(?P<digit>\d+(?:\.\d+)?)\s*(?P<unit>[KMGTPE]?i?B(?!\w))

If you think there is no problem with this regexp, please reply and I will push the new commit later.

---

I believe this regex should work for 99% of entries. And for scenarios that this filter can't handle, users can chain the re_search and other jinja syntax (and match_re for this filter are also able to change) to make it work for their scenarios. I think more tweaking may not really necessary, I guess :(

[gazpachoking]

Awesome, thanks!