Crawl data from Paris Musees using their API

[rtroncy]

We have one more source to crawl and to integrate in the Knowledge Graph coming from Paris Musees.

See the gdoc documentation for the details.

@ehrhart Can you add this new source in the SILKNOW crawler?

Next, we would need to transform this collected data in the SILKNOW data model to be inserted in the knowledge graph. @mpuren is preparing the mapping table which will be very similar to the Joconde one

[mpuren]

I have prepared the mapping tables in this gdoc. I have mapped the data as they are online, so we will have to verify that the collected data have the same structure. I think some fields identifying the museums will be added (for example an address) ; and maybe some fields will be missing (such as "Contenus en relation", "Droits sur les images", "Documentation", "Expositions".)

[mpuren]

We are currently discussing the best strategy to crawl the data from the Paris Musées collections. So for now don't add this source in the SILKNOW crawler ; I will keep you posted.

[rtroncy]

Does this "hold" refer to the mail thread continued by Jorge (Focus of KG contents)?

[mpuren]

Yes, it was referring to this email thread.

But after the the conversation we had this morning, I think we can resume the process.

[ehrhart]

I cannot manage to get high quality photos using the GraphQL api. It only returns the thumbnails for some reason.

For example: https://parismuseescollections.paris.fr/node/106152 has 2 photos.
GraphQL Explorer: https://apicollections.parismusees.paris.fr/explorer
The following query returns 2 thumbnails but the HD photos are null:

{
  nodeQuery(filter: {conditions: [
    {field: "nid", value:"106152"}
  ]}, limit: 1) {
    count
    entities {
      entityUuid
      ... on NodeOeuvre {
        absolutePath
        fieldVisuels {
          entity {
            mid
            uuid
            vid
            revisionCreated
            revisionLogMessage
            status
            name
            created
            changed
            publicUrl
            vignette
            defaultLangcode
            revisionDefault
            revisionTranslationAffected
            contentTranslationOutdated
            entityLabel
            entityId
            entityBundle
            entityChanged
            entityUuid
            entityType
            entityCreated
            entityPublished
            fieldCopyright
            fieldImageDroits
            fieldImageLegendeCplementaire
            fieldImageVenteTirage
            fieldLegende
            fieldImageLibre
          }
        }
        fieldHdVisuel {
          entity {
            mid
            uuid
            vid
            revisionCreated
            revisionLogMessage
            status
            name
            created
            changed
            publicUrl
            vignette
            defaultLangcode
            revisionDefault
            revisionTranslationAffected
            contentTranslationOutdated
            entityLabel
            entityId
            entityBundle
            entityChanged
            entityUuid
            entityType
            entityCreated
            entityPublished
            fieldCopyright
            fieldImageDroits
            fieldImageLegendeCplementaire
            fieldImageVenteTirage
            fieldLegende
            fieldImageLibre
          }
        }
      }
    }
  }
}

[ehrhart]

It looks like we're able to guess the HD url by replacing the base url, for example:

• Thumbnail: https://apicollections.parismusees.paris.fr/sites/default/files/styles/thumbnail/collections/atoms/images/CAR/manz_carns00015_001.jpg?itok=4nVFHI-_
• HD version: https://www.parismuseescollections.paris.fr/sites/default/files/styles/pm_diaporama/public/atoms/images/CAR/manz_carns00015_001.jpg

The file name stays the same, and the token doesn't seem to matter.

[ehrhart]

Implemented using the GraphQL API and the 3 search strategies for dates, as described per the gdoc.

Note: I've tried mapping as much data as possible within our structure (label-value), but in order to not miss any potential informations I've also added the original JSON response as a raw property in the record file.

List of known fields for Paris Musées.

Records: paris-musees_records_20201202_1.tar.gz

Files: paris-musees_files_20201202_1.tar.gz