Spike: metadata completeness

[murdo-moj]

Following on from #62

Create a report/dashboard which pulls from postgres (DataHub entity store) to display metadata completeness

Could be:

• Superset dashboard
• A page on find-moj-data which live queries postgres
• A scheduled workflow on find-moj-data which feeds a static file for a page, to avoid loading the database so much

[seanprivett]

Investigate how the analytics measures are generated in DataHub - any reliability issues
Present analytics from tile in DataHub UI
Use HTML rather than bespoke tools
Not plugged in to the front end at this stage

[murdo-moj]

some of the graphql queries for analytics are here
https://github.com/datahub-project/datahub/blob/1b3bbf8234adf6c04b90fb6d0f62ce0254b7ec09/datahub-web-react/src/graphql/analytics.graphql

these are for the charts though, it is missing the info in the top tile
https://datahub-catalogue-dev.apps.live.cloud-platform.service.justice.gov.uk/analytics

[murdo-moj]

eg

    getAnalyticsCharts {
        groupId
        title
        charts {
            __typename
        }
    }
}

[tom-webber]

We want to know how the analytics from the top tiles on the DataHub Analytics page are generated.

These tiles are generated from the getHighlights operation. Resulting graphql:

query getHighlights {
  getHighlights {
    value
    title
    body
    __typename
  }
}

value

Resolver:

Retrieves the Highlights to be rendered of the Analytics screen of the DataHub application.

Relevant note (Feb 2023):

  /** TODO: Config Driven Charts Instead of Hardcoded. */

[tom-webber]

There was interest in tracking frontend activity with Google Analytics. This is something that's possible to do with DataHub as well. It's currently only available if we maintain a fork of DataHub.

[tom-webber]

The 'Datasets' count widget includes the following text:

"0.00% have owners, 83.42% have tags, 0.00% have glossary terms, 100.00% have description, 99.41% have domain assigned!"

The description number isn't matching with the descriptions we're seeing on the platform, as much less than 100% of the entities have populated descriptions. The numerator for this percentage comes from here in the analytics derivation code.

According to a response in the DataHub slack:

there are 2 descriptions in DataHub: native descriptions and source descriptions (those authored in external platform). This hasDescription is only matching against the source description, e.g. that which comes from the original platform

If we want to view the count for the in-catalogue description, we need to inspect a different property: the editableProperties.description, rather than properties.description (as per slack):

DataHub always maintains 2 distinct descriptions - one from the underlying system and another that is "editable" inside of DataHub. (properties.description and editableProperties.description), respectively.
Please use the editableProperties.description field to read the DataHub-native description that was edited in the UI

It may be that more complex logic is required for calculation of the documentation numerator, as there doesn't appear to be an equivalent hasDescription boolean for the editable description, only the editedDescription field.

[seanprivett]

Investigation complete: team to have a discussion about outcome and which tickets need to be created off the back of this.