Hacker News Datasette write demo

[simonw]

Idea is to have my existing scraper at https://github.com/simonw/scrape-hacker-news-by-domain also write to my private Datasette Cloud account, then create an atom feed from it.

[simonw]

https://simon.datasette.cloud/data/hacker_news_posts_atom.atom is working now.

[simonw]

This is the SQL view for the atom feed:

CREATE VIEW hacker_news_posts_atom as select
  id as atom_id,
  title as atom_title,
  url,
  commentsUrl as atom_link,
  dt || 'Z' as atom_updated,
  'Submitter: ' || submitter || ' - ' || points || ' points, ' || numComments || ' comments' as atom_content
from
  hacker_news_posts
order by
  dt desc
limit
  100;

[simonw]

I created the hacker_news_posts table like this:

export ROWS=$(
  jq -n --argjson rows "$(cat simonwillison-net.json)" \
  '{ "table": "hacker_news_posts", "rows": $rows, "pk": "id", "replace": true }'
)
# Use curl to POST some JSON to a URL
curl -X POST \
  https://simon.datasette.cloud/data/-/create \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DS_TOKEN" \
  -d $ROWS

[simonw]

But until I fix this issue:

• ignore:true/replace:true options for /db/-/create API #1927

I need to insert freshly scraped data like this:

export ROWS=$(
  jq -n --argjson rows "$(cat simonwillison-net.json)" \
  '{ "rows": $rows, "replace": true }'
)

curl -X POST \
  https://simon.datasette.cloud/data/hacker_news_posts/-/insert \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DS_TOKEN" \
  -d $ROWS

[simonw]

Here's the change that should submit data to Datasette Cloud: simonw/scrape-hacker-news-by-domain@848bb7e

I ran delete from hacker_news_posts against my instance so https://simon.datasette.cloud/data/hacker_news_posts is now empty.

[simonw]

Got it working!

https://simon.datasette.cloud/data/hacker_news_posts

https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/submit-to-datasette-cloud.sh

[simonw]

Wrote about this extensively here: https://simonwillison.net/2022/Dec/2/datasette-write-api/