Resource page with Data Explorer or preview

[lauragift21]

A page for viewing an individual resource exists and it previews the data (if a preview for that data type exists).

In particular, for CSV etc and data in CKAN DataStore it displays the Data Explorer.

NOTE: ⚠️ there is quite a bit of thinking here about how we refactor the data previewing and data explorer. Read the notes in analysis section below. Note, in particular, that the acceptance and tasks are not yet updated to reflect this (we will probably need to refactor this issue into multiple issues).

Acceptance

• Resource page works from the mocks, e.g., localhost:3000/@world-bank/gdp/r/gdp loads and displays resource metadata WIP: https://portal.datopian1.now.sh/@dataset/asdsad/r/gettyimages-954867550.jpg
• For CSV we get a data preview (as per data explorer)
• Ditto for data in CKAN DataStore

Tasks

• Implement /[org]/[dataset]/r/[resource] page that renders necessary components for the resource page:
  • Navbar
  • About resource component, e.g., metadata table with info such as resource name, title, description, format, created/modified date and link to downloading
• Data Explorer or Data Preview component based on given resource type:
  • If in datastore, we can show the data explorer that has preview + query builder + map/chart builder + pagination
  • If not in datastore, we only render a preview based on format:
    • tabular data => table
    • image/website/pdf => web view
    • geo data => map
    • anything else => nothing

Analysis

Re-using existing Data Explorer library might not be the best way to go here because:

• Data Explorer we've built using React + Redux wasn't the easiest library to maintain, for example, if you need to fix something in one of the components (eg, Map Builder), you have to go there and do:
  • fix it
  • if unit tests there, run them
  • test with fixtures (cosmos)
  • test with yarn start to load complete app
  • build the dist version by yarn build:package
  • push to github
  • publish to NPM
  • install it in Data Explorer and build the bundle
  • import the bundles (normally huge ~>2-3MB due to dependencies such as Plotly) into the HTML page

As we can see, it is quite a long process and over time we've seen little feature development but only bug fixes. I believe the process was discouraging for developers in general.

Other points:

• Currently, using Data Explorer as a React component is not clear as it was always been used as bundles that doesn't get called explicitly with arguments, but inits on load and traverses the DOM to find the HTML tag with data-pacakge attribute. It parses it and uses it as props for child components.
• If we want to use Data Explorer and its components in NextJS, we have to refactor it as it isn't compatible right away, for example, importing CSS, see https://github.com/vercel/next.js/blob/master/errors/css-npm.md

How would I build Data Explorer today with NextJS?

• Make the Data Explorer a component rather than a library
• Control components such as Query/Map/Chart builder from NextJS so basically our frontend app becomes the Data Explorer
• Use something like SWR for client-side data fetching - https://swr.now.sh/

http://tech.datopian.com/data-explorer

graph TD

DataExplorer
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
Redux

subgraph old
  Redux -.- DataExplorer
  DataExplorer --> DataPackageViews
  DataExplorer --> DataStoreQueryBuilder
  DataExplorer --> ChartBuilder
  DataExplorer --> MapBuilder
end

graph TD

NextJSPage
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
GraphQL

subgraph new
  NextJSPage --init--> DataPackageViews
  NextJSPage --init--> DataStoreQueryBuilder
  NextJSPage --init--> ChartBuilder
  NextJSPage --init--> MapBuilder
  GraphQL -.- NextJSPage
  GraphQL -.- DataPackageViews
  GraphQL -.- DataStoreQueryBuilder
  GraphQL -.- ChartBuilder
  GraphQL -.- MapBuilder
end

• User: GET a resource page - /@org/{dataset-name}/r/{resource-name}
• NextJS (server side) fetches DMS API to get resource metadata which is then cached by GraphQL
• NextJS responses with HTML + JSON for hydrating the tempalte
• Browser renders the page with components
• DataPackageViews component fetches data from the source url (data can be in filestore [or datastore])
  • OR: could load the data server side and pass down ...
• Renders the data in a table (for tabular data)
  • TODO: other renderings ...``

First steps:

• Review current Data Explorer lib [0.5h]
• Review datapackage views component [0.5h]
• Refactore NextJS page for resource - https://github.com/datopian/portal.js/blob/master/pages/%5Borg%5D/%5Bdataset%5D/r/%5Bresource%5D/index.tsx
  • we already implemented server side data fetching (resource object)
  • we next need to create DataPackageView component which should:
    • Remove DataExplorer component and create DataPackageView compoent instead
    • Copy over implementation of DataExplorer
    • fetch data in browser (eg, use https://swr.vercel.app/) [4h]
    • render table based on that data [4h]
      • we can re-use existing datapackage-views-js here

Notes

• Resource pages do simple (pre)view by default
• Explorer page for rich exploration (requires)

# url structure
/dataset/myname/ => [implicitly] /dataset/myname/show/default/ # default-branch-latest-revision/
/dataset/myname/resource/xxx => [implicitly] renders  /dataset/myname/show/default/{resource-id}

# explicit revisions
/dataset/myname/show/{revref}/              # dataset showcase
/dataset/myname/show/{revref}/{resource-id} # resource showcase

[anuveyatsu]

Re-using existing Data Explorer library might not be the best way to go here because:

• Data Explorer we've built using React + Redux wasn't the easiest library to maintain, for example, if you need to fix something in one of the components (eg, Map Builder), you have to go there and do:
  • fix it
  • if unit tests there, run them
  • test with fixtures (cosmos)
  • test with yarn start to load complete app
  • build the dist version by yarn build:package
  • push to github
  • publish to NPM
  • install it in Data Explorer and build the bundle
  • import the bundles (normally huge ~>2-3MB due to dependencies such as Plotly) into the HTML page

As we can see, it is quite a long process and over time we've seen little feature development but only bug fixes. I believe the process was discouraging for developers in general.

Other points:

• Currently, using Data Explorer as a React component is not clear as it was always been used as bundles that doesn't get called explicitly with arguments, but inits on load and traverses the DOM to find the HTML tag with data-pacakge attribute. It parses it and uses it as props for child components.
• If we want to use Data Explorer and its components in NextJS, we have to refactor it as it isn't compatible right away, for example, importing CSS, see https://github.com/vercel/next.js/blob/master/errors/css-npm.md

[anuveyatsu]

How would I build Data Explorer today with NextJS?

• Make the Data Explorer a component rather than a library
• Control components such as Query/Map/Chart builder from NextJS so basically our frontend app becomes the Data Explorer
• Use something like SWR for client-side data fetching - https://swr.now.sh/

[anuveyatsu]

@rufuspollock I've added couple of comments above, please, take a look.

[rufuspollock]

@anuveyatsu ok - for now we build the page without preview and move one whilst we decide what we want to do here 😄

I've moved this to blocked / waiting for for now.

[anuveyatsu]

This is how resource page looks like at the moment:

https://portal.datopian1.now.sh/@dataset/asdsad/r/gettyimages-954867550.jpg

[rufuspollock]

WONTFIX. Closing for now as Catalog related and we aren't going to do much morer right now. Note we have implemented views support in the basic portal app in #72.