Adds script to remove duplicates, refs 2883

[mwjames]

This PR is made in reference to: #2883

This PR addresses or contains:

• Adds the removeDuplicateEntities.php maintenance script to remove all duplicate entities with non-residual references (meaning those that have no reference in any other table) from the entity table

This PR includes:

• Tests (unit/integration)
• CI build passed

Fixes #

[mwjames]

The script will output something similar to the example below with the disposed section containing those that have been removed and untouched that persist due to being the one with an active reference. In case both would carry active references (which should not but can happen) then the both remain and the user has to resolve the issue manually by inspecting the listed entity and purge (or save) the page [0].

Found: 1 duplicates

.

Log

{
    "disposed": {
        "311800": {
            "smw_title": "_wpg",
            "smw_namespace": "102",
            "smw_iw": "",
            "smw_subobject": ""
        }
    },
    "untouched": {
        "189135": {
            "smw_title": "_wpg",
            "smw_namespace": "102",
            "smw_iw": "",
            "smw_subobject": ""
        }
    }
}

[0] https://www.semantic-mediawiki.org/wiki/Help:Duplicate_entities

[mwjames]

@kghbln Happy testing. (PS: Remember the [0] list is cached!)

[0] https://sandbox.semantic-mediawiki.org/w/index.php?title=Sp%C3%A9cial:SemanticMediaWiki&action=duplookup

[kghbln]

@mwjames I just ran the script (log). Interestingly this log seems to provide some hints about the whereabouts of #2001 ("This_page_supports_semantic_in-text_annotations ..."). Since the log is rather lengthy it will perhaps be an idea allow logging to a file similarly to what rebuildData.php does.

[mwjames]

Since the log is rather lengthy it will perhaps be an idea allow logging to a file similarly to what rebuildData.php does.

Can you create a follow-up task on this?

[kghbln]

Can you create a follow-up task on this?

Done with #2946

[mwjames]

@kghbln FYI

whereabouts of #2001 ("This_page_supports_semantic_in-text_annotations ...").

Here the analysis:

First, check:

Second, find the ID (use the full name as input, it will find the IDs) and reference the ID is used.

With the result of:

{
    "21692": [],
    "21843": [],
    "21881": {
        "smw_di_wikipage": {
            "s_id": "3378"
        }
    }
}

Looking at 3378 will reveal:

After looking at Utilisateur:Lalquier, you will find:

{{#set:Description=This page supports semantic in-text annotations (e.g. "Is specified asWorld Heritage Site") to build structured and queryable content provided by Semantic MediaWiki. For a comprehensive description on how to use annotations or the #ask parser function, please have a look at the getting started, in-text annotation, or inline queries help page.This page supports semantic in-text annotations

I'd say, kick the text from the user page!

[kghbln]

I'd say, kick the text from the user page!

Yeah, done! This was driving me nuts.

[kghbln]

Documented