search wikicommons by sha1

[jhpoelen]

Came up with conversation with @Daniel-Mietchen

On uploading an image to Media Commons, hashes are calculated and used in the media commons ecosystem.

[jhpoelen]

feature is used by Media uploading to prevent duplicates.

[jhpoelen]

Searching images in wikidata with available checksum

SELECT ?item ?image ?checksum ?method WHERE {
  ?item p:P4092 ?checksum;
        wdt:P18|wdt:P117 ?image .
  ?checksum pq:P459 wd:Q13414952 .
}
LIMIT 10

search by specific checksum

SELECT ?item ?image ?checksum ?method WHERE {
  ?item wdt:P4092 ?checksum;
        wdt:P18|wdt:P117 ?image .
  ?item wdt:P4092 "2bef4f65032f34d70957bb123560fddb63e5686c" .
}
LIMIT 10

@Daniel-Mietchen any idea how how to do structured searched on wiki mediacommons statements ?

For some reason the hash://sha1/8a0c61a0ac8bc4adaecb0a39d63f8817cee3c789 did not yield any hits via

SELECT ?item ?image ?checksum ?method WHERE {
  ?item p:P4092 ?checksum;
        wdt:P18|wdt:P117 ?image .
  ?checksum pq:P459 wd:Q13414952 .
  ?item wdt:P4092 "8a0c61a0ac8bc4adaecb0a39d63f8817cee3c789" .
}
LIMIT 10

, despite the structured data I added to the media commons object https://commons.wikimedia.org/wiki/File:Agapostemon_texanus_killed_by_Peucetia_viridans_-_iNaturalist_56389401.jpg .

[jhpoelen]

Here's an example of a SPARQL query with structured checksum constraints -

SELECT ?item ?referenceURL WHERE {
  ?item p:P4092 ?checksum .
  ?checksum ps:P4092 "2bef4f65032f34d70957bb123560fddb63e5686c" .
  OPTIONAL { 
    ?checksum prov:wasDerivedFrom ?refnode. 
    ?refnode pr:P854 ?referenceURL .
  }
}
LIMIT 10

And, not that, for this specific example, the related referenceUrl value is a pointer to a meta-data file that documented the sha1 hash for a specific file, not the file itself. Meaning that curl -L {referenceUrl} | sha1sum will not be 2bef4f65032f34d70957bb123560fddb63e5686c .

[jhpoelen]

Looks like OpenRefine/OpenRefine#4472 (comment) @multichill contains a method to search wikimedia commons by their sha1 hash.

Ask the API for any files with the hash, for example https://commons.wikimedia.org/w/api.php?action=query&list=allimages&aisha1=b12eb7f6b43a3cc1eefc54db90ec5ae1504c3f17

[jhpoelen]

with

curl "https://commons.wikimedia.org/w/api.php?action=query&aisha1=b12eb7f6b43a3cc1eefc54db90ec5ae1504c3f17&list=allimages&format=json"\
 | jq.

yielding:

{
  "batchcomplete": "",
  "query": {
    "allimages": [
      {
        "name": "Johann_Adam_Klein_-_Cossacks_Eat_a_Meal_in_the_Field_(1819),_Thorvaldsens_Museum_E721,6.jpg",
        "timestamp": "2022-05-03T19:38:38Z",
        "url": "https://upload.wikimedia.org/wikipedia/commons/c/ce/Johann_Adam_Klein_-_Cossacks_Eat_a_Meal_in_the_Field_%281819%29%2C_Thorvaldsens_Museum_E721%2C6.jpg",
        "descriptionurl": "https://commons.wikimedia.org/wiki/File:Johann_Adam_Klein_-_Cossacks_Eat_a_Meal_in_the_Field_(1819),_Thorvaldsens_Museum_E721,6.jpg",
        "descriptionshorturl": "https://commons.wikimedia.org/w/index.php?curid=117655456",
        "ns": 6,
        "title": "File:Johann Adam Klein - Cossacks Eat a Meal in the Field (1819), Thorvaldsens Museum E721,6.jpg"
      }
    ]
  }
}

`
and, as expected:

curl "https://upload.wikimedia.org/wikipedia/commons/c/ce/Johann_Adam_Klein_-_Cossacks_Eat_a_Meal_in_the_Field_%281819%29%2C_Thorvaldsens_Museum_E721%2C6.jpg"\
 | sha1sum

b12eb7f6b43a3cc1eefc54db90ec5ae1504c3f17 or hash://sha1/b12eb7f6b43a3cc1eefc54db90ec5ae1504c3f17

[jhpoelen]

with recent change, content in WikiMedia commons can now be retrieved by sha1 using Preston via e.g.,:

preston cat\
 --remote https://wikimedia.org\
 hash://sha1/b12eb7f6b43a3cc1eefc54db90ec5ae1504c3f17\
> some.jpg

producing attached image:

[jhpoelen]

and, the original example was retrieved via:

preston cat\
 --remote https://wikimedia.org\
 hash://sha1/8a0c61a0ac8bc4adaecb0a39d63f8817cee3c789\
 > iNaturalist_56389401.jpg

with retrieved image attached

[jhpoelen]

Now also available via https://linker.bio/hash://sha1/8a0c61a0ac8bc4adaecb0a39d63f8817cee3c789

[jhpoelen]

and also, running on wikimedia servers,

https://beehind.toolforge.org/hash://sha1/8a0c61a0ac8bc4adaecb0a39d63f8817cee3c789

and attached screenshot

[jhpoelen]

fyi @cboettig @mbjones this issue contains a method to integrate with sha1's from the WikiMedia Commons. I believe these contain all images on wikipedia and associated projects.

[mbjones]

Nice.