Json parsing checklist

[betsybookwyrm]

This checklist is for the first pass at parsing all the objects in the json. It's okay if it misses some cases, but it should get it to the point where most of the data goes in the database and we can start doing testing to find things we've missed.

Twarc + twitter json:

• _twarc
• data
  • tweet fields
  • mentions
  • annotations
  • context annotations
  • references
• includes
  • media
  • places
  • tweets (as for data above)
  • users
    • user fields
    • urls
    • hashtags
    • mentions
    • other entites???
• meta
• errors

[betsybookwyrm]

I'm pasting the more complex jq commands I'm using to go through the twarc/twitter json structure on the wiki, so I/we don't have to figure them out again: https://github.com/QUT-Digital-Observatory/tidy_tweet/wiki

[betsybookwyrm]

To start with, I'm ignoring changes over time - initially it will only be able to accurately ingest one API result file per database. We'll change this later.