Make fresh data available regularly

[jvwong]

Background: Currently we retrieve the article data manually and save it to local file (e.g. data-search.js). On build (e.g. npm run build or npm run watch), the data file is directly imported as a JS module and the associated data is bundled with the App.

Problem: The data is currently fixed. The intention is for the App to show updated article information (e.g. daily). To accomplish this, it will be necessary to make bundling and serving the App (i.e. on Cloudflare) separate from the article data it uses.

Goal: Make the App retrieve the article data file from an external source.

---

Q: What are the applicable constraints, e.g. compatibility or performance?

A: The source of the data should enable browsers to access the information. For instance, a binary file associated with a GitHub release does not provide for Cross-Origin Resource Sharing (CORS). A temporary solution is to point the app at a file in the remote (e.g. https://raw.githubusercontent.com/PathwayCommons/hyper-recent/main/example-data/data.json) and find a suitable data store (see below).

• References
  • Data repository to check: https://www.cloudflare.com/en-ca/products/workers-kv/

[maxkfranz]

Logging some options here. Feel feel to incorporate into the top post of the issue, as needed.

• (1) Commit the files to the repo
  • Pros: simple, easy-to-access history
  • Cons: will blow up the repo size and make things slow and expensive to manage in the long term
• (2) Store the files in Cloudflare KV using a Worker
  • Key: date
  • Value: the whole file json
  • Worker: a small single-function web service that just shoves the file in KV, probably protected with an API key
  • Pros: effective, efficient in the long term
  • Con: a couple more moving parts (but we need to use CF anyway)
• (3) Make the file processing generation part of the Cloudflare pages build script
  • Github actions could kick off a build via a deploy hook
  • CF pages has a 20 minute build limit time -- probably enough
  • Pros: even simpler than (1)
  • Cons: handling history of files requires an extra step or you need to build N days of data each time if the dashboard goes N days back

[jvwong]

• (3) Make the file processing generation part of the Cloudflare pages build script

  • Github actions could kick off a build via a deploy hook
  • CF pages has a 20 minute build limit time -- probably enough
  • Pros: even simpler than (1)
  • Cons: handling history of files requires an extra step or you need to build N days of data each time if the dashboard goes N days back

I did a couple of runs of data-search.js locally and it's about 2.5 minutes (0.23 MB file for 1 popular topic "alzheimers" over 1 month). Maybe we can give the (3) CF-build a try.

[maxkfranz]

Probably good enough for now.

For future: Remote builds are usually slower than local. I wouldn't be surprised if alzheimers/1mo took ~10 min.

[jvwong]

PR #42 build took 3m 48s to complete (1 topic)