🎮 Data collector for SGDQ-Stats
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
misc
tools
utils
.gitignore
LICENSE
Makefile
README.md
backup.js
chat_collector.js
data_file.json
health_check.js
package.json
populate_firebase_games.js
sgdq_collector.js
start.js
start.sh
storage_link.js
test.js
tracker_collector.js
tweet_collector.js
tweet_sender.js

README.md

sgdq-collector

🎮 Backend data collector for SGDQ-Stats

Tools

  • NodeJS
  • Firebase for backend storage
  • Google Compute Engine as the hardware running the scripts
  • Google Cloud Storage as the cache location for the JSON dataset

Data Sources

  • All Twitch data is acquired with tmi.js. Viewership data is gained by polling the API every minute. The collector listens in on the chat and reports messages per minute, as well as emote data (by referencing the Twitch global emote list).
  • Twitter data is collected by listening to a Twitter stream with the terms sgdq, summergamesdonequick, sgdq2016, #sgdq2016.
  • Donations / Donator data is scraped from the SGDQ Donation Tracker.

Frontend

These scripts collect data for the data visualizations done in SGDQ-Stats.

Firebase Schema

  • data => Holds collection of data points
    • <epoch time> => Entry
      • m => Total donations ($)
      • d => Total num donators
      • v => Number of Twitch viewers
  • games => Holds collection of games
    • <epoch time> => Start time of game
      • title => Name of game
      • runner => Name of speed runner
      • duration => Planned duration of game's run
      • start_time => Start time of game's run in epoch time (Redundant...)
  • stats
    • avg_donation => Average donation amount ($)
    • games_played => Number of games played so far in the marathon
    • max_donation => Largest donation ammount ($)
    • num_donators => Number of individual donations (not a perfect 1-to-1 map, but close enough)
    • total_chats => Total number of chat messages sent in the Twitch Chat
    • total_donations => Sum of all donations ($)
    • total_emotes => Total number of emotes sent in the Twitch Chat
    • total_tweets => Total number of Tweets sent related to SGDQ
  • extras => Interesting, but less useful stats
    • <epoch time> => Entry
      • c => Number of Twitch chats sent
      • e => Number of Twitch emotes sent
      • t => Number of SGDQ-related Tweets sent