Implement digippres.org changes and min sig length

[ross-spencer]

A filter has been added to the default query to reduce the number of results relevant to us. This reduces the Wikidata query service query time as well as the amount of time required to generate provenance.

Additionally, because of TrID issues, we implement a minimum signature length that reduces time even further.

Connected to: ffdev-info/wikidp-issues#32
Connected to: #183
Connected to: ffdev-info/wikidp-issues#38
Co-authored-by: @anjackson

nb. default roy is not able to harvest from Wikidata on my computer anymore, at least not within 10 mins. Maybe if left longer.

With filter

time ./roy harvest --wikidata
2024/07/13 14:10:48 Roy (Wikidata): Harvesting Wikidata definitions: lang 'en'
2024/07/13 14:10:48 Roy (Wikidata): Harvesting definitions from: 'https://query.wikidata.org/sparql'
2024/07/13 14:10:48 Roy (Wikidata): Harvesting revision history from: 'https://www.wikidata.org/'
2024/07/13 14:15:37 Roy (Wikidata): Harvesting Wikidata definitions '/home/r0ss/siegfried/wikidata/wikidata-definitions-3.0.0' complete

real	4m48.907s
user	0m10.730s
sys	0m2.906s

./roy build -wikidata
2024/07/13 15:21:55 Roy (Wikidata): Congratulations: doing something with the Wikidata identifier package!
2024/07/13 15:21:55 Roy (Wikidata): Opening Wikidata definitions: /home/r0ss/siegfried/wikidata/wikidata-definitions-3.0.0
2024/07/13 15:22:40 {
  "AllSparqlResults": 16749,
  "CondensedSparqlResults": 13720,
  "SparqlRowsWithSigs": 10609,
  "RecordsWithPotentialSignatures": 9133,
  "FormatsWithBadHeuristics": 60,
  "RecordsWithSignatures": 9073,
  "MultipleSequences": 12,
  "AllLintingMessages": [
    "Use the `-wikidataDebug` flag to build the identifier to see linting messages"
  ],
  "AllLintingMessageCount": 253,
  "RecordCountWithLintingMessages": 199
}
2024/07/13 15:22:40 Roy (Wikidata): Building identifiers set from PRONOM
2024/07/13 15:22:50 Roy (Wikidata): In Infos()... length formats: '13720' no-pronom: 'false'
2024/07/13 15:22:50 Roy (Wikidata): Adding Glob signatures to identifier...
2024/07/13 15:22:50 Roy (Wikidata): Adding container signatures to identifier...
2024/07/13 15:22:51 Roy (Wikidata): Adding container signatures to identifier...
2024/07/13 15:22:52 Roy (Wikidata): Adding Wikidata Byte signatures to identifier...

With siglen min 6 (3 bytes) signature length:

time ./roy harvest -wikidata
2024/07/13 15:54:37 Roy (Wikidata): Harvesting Wikidata definitions: lang 'en'
2024/07/13 15:54:37 Roy (Wikidata): Harvesting definitions from: 'https://query.wikidata.org/sparql'
2024/07/13 15:54:37 Roy (Wikidata): Harvesting revision history from: 'https://www.wikidata.org/'
2024/07/13 15:57:21 Roy (Wikidata): Harvesting Wikidata definitions '/home/r0ss/siegfried/wikidata/wikidata-definitions-3.0.0' complete

real	2m43.534s
user	0m6.724s
sys	0m1.961s

./roy build -wikidata
2024/07/13 15:57:54 Roy (Wikidata): Congratulations: doing something with the Wikidata identifier package!
2024/07/13 15:57:54 Roy (Wikidata): Opening Wikidata definitions: /home/r0ss/siegfried/wikidata/wikidata-definitions-3.0.0
2024/07/13 15:58:09 {
  "AllSparqlResults": 9284,
  "CondensedSparqlResults": 8118,
  "SparqlRowsWithSigs": 9284,
  "RecordsWithPotentialSignatures": 8118,
  "FormatsWithBadHeuristics": 46,
  "RecordsWithSignatures": 8072,
  "MultipleSequences": 12,
  "AllLintingMessages": [
    "Use the `-wikidataDebug` flag to build the identifier to see linting messages"
  ],
  "AllLintingMessageCount": 195,
  "RecordCountWithLintingMessages": 156
}
2024/07/13 15:58:09 Roy (Wikidata): Building identifiers set from PRONOM
2024/07/13 15:58:19 Roy (Wikidata): In Infos()... length formats: '8118' no-pronom: 'false'
2024/07/13 15:58:19 Roy (Wikidata): Adding Glob signatures to identifier...
2024/07/13 15:58:19 Roy (Wikidata): Adding container signatures to identifier...
2024/07/13 15:58:19 Roy (Wikidata): Adding container signatures to identifier...
2024/07/13 15:58:20 Roy (Wikidata): Adding Wikidata Byte signatures to identifier...

[ross-spencer]

@richardlehane cc. @thorsted -- is there any chance we can build a release candidate with these changes to trial them?

[richardlehane]

@ross-spencer @thorsted these changes now on the develop branch and built as a release candidate (Version 1.11.2-rc0)