[Plugin] Introduce `merge_tags` subroutine for plugins

[psilabs-dev]

This came up when I tried looking for a solution to #1140.

Currently, the method of tagging archives via metadata plugins goes through two stages: exec_metadata_plugin and set_tags. When exec_metadata_plugin is called, it finds tags for an archive which are not already in the archive, and this is done by logic in the subroutine.

For example, if existing tags are [a, b, c, d, e] and new tags are [c, d, e, f, g], then the tags returned by exec_metadata_plugin will be the set difference, [f, g].

set_tags then appends these new tags onto the old tags. Although its default behavior is to replace the archive's tags with its arguments completely, this append option is true in all calls accompanying exec_metadata_plugin.

Under this current code, the "merge policy" of metadata plugin calls is "append if not exists". For example, if existing tags are [a, b, artist_name:c, d, e] and new tags are [artist_name:f], then calling a metadata plugin will lead to the archive with tags [a, b, artist_name:c, d, e, artist_name:f]. Deletions will never occur with this policy.

While there is nothing unintended with the above feature, certain data sources have strict constraints. For instance in Pixiv, artist names and upload dates may change, but there can only be one artist name/upload date.

Although I can add logic to handle this specifically for Pixiv, such logic would be outside the scope of namespace-specific plugin contributions. I believe this logic can be added to all namespaced metadata code. Also, other plugins may benefit from being able to implement their own tag constraints.

Proposed is a low-impact, low-difficulty refactoring to add something like the following to all custom plugin files,

# Plugin ABC

sub get_tags {
    # get_tags logic
}

sub merge_tags($old_tags, $new_tags) {
    # plugin-specific tag merging logic, or fall back to "append if not exists" merging
    return $merged_tags;
}

...

Notice that actively implementing a merge_tags subroutine is not necessary; by default, "append if not exists" will be used. This subroutine should be offered in a common module.

Then, for setting tags, we have something like this:

%plugin_result = exec_metadata_plugin( $plugin, $id, %args );
my $old_tags = $redis->hget( $id, "tags" );
my $new_tags = $plugin_result{new_tags};
my $merged_tags = $plugin->merge_tags($old_tags, $new_tags);
set_tags( $id, $merged_tags ); # no need to append, replace with these tags.

In the case of Pixiv, the merge policy will look like this:

sub merge_tags($old_tags, $new_tags) {
    my @merged_tags;
    foreach my $tagtoadd (@tagarray) {
        # check if tag starts with "artist:" or "upload_date:", then check and remove from old tags
        # do usual check to see if tag is not in old tags
    }
}

We may also choose to include arguments in merge_tags which arise from plugin settings, that's up for separate discussion.

[psilabs-dev]

To follow up on this, I'm planning to add features for the pixiv and nhentai metadata plugins: a checkbox option to include collection of bookmark counts for pixiv artworks, and favorites for nhentai archives (I'm assuming these exist in the html doc). I'm sure similar metrics exist for others as well.

What applications do these metrics have? One is archive deduplication. A site like nhentai can have up to 10-20 duplicates for a given archive, with varying quality of translations. It could be reasonable to find, for a given language, the archive with the highest favorite/star count to keep, if such data was available.

A method would be needed to avoid multiple occurrences of "favorite:*", "like:*" tags per archive, which plugin-specific tag merging will solve.

[IceBreeze]

I wouldn't use a new method, but I would add a parameter to the returned hash like this:

my %r = ( 
  title => $title,
  tags => $tags,
  replace_tags => 1
)

This will cause a change inside exec_metadata_plugin which will have to pass the new information to the caller, maybe like this:

%returnhash = ($newmetadata{replace_tags}) ? ( tags => $newtags ) : ( new_tags => $newtags );
# or 
%returnhash = ( new_tags => $newtags, replace => $newmetadata{replace_tags} );

with preference for the latter and then save the tags as:

set_tags( $id, $plugin_result{new_tags}, ! $newmetadata{replace_tags} );

or something like that.

[psilabs-dev]

I can get behind this. Basically this allows us to always prioritize the data provided by the most up-to-date metadata.

[psilabs-dev]

Actually, there might be a problem for users who use multiple metadata plugins, regardless of whether the implementation is a global configuration or plugin-specific.

If I use metadata plugins A -> B -> C, and C has replace_tags = 1, then the metadata retrieved by A and B are erased, this would make chaining plugins which do different things pointless (though personally I don't use multiple plugins).

We can change the logic to replace only namespaced tags, where only tags that are key value pairs are replaced by most recent ones, and all other tags are left alone/appended onto:

# old metadata
"author:author1,author:author2,last_updated:2025-01-01,tag_a,tag_b,tag_c,source:source"

# new metadata
"author:author1,author:author2-updated,last_updated:2025-01-02,tag_a,tag_b,tag_c,tag_d,source:source"

and also replace other non-tag metadata, like summary and title.

[psilabs-dev]

resolved downstream with API