e621 Tag Statistics

About

Statistics on tags on e621 via their daily dump csv. Plots data using matplotlib or Google Charts via a Flask app.
Every image on e621 must be tagged with info describing what's in it (characters, artist, etc.). Using this information provided via the e621 API, we can plot the popularity of something.
Also: rule34 version.

---

File Description

• main.py The main statistics program. Chang list of tags to track in the script via tag_name
  
• downloader.py Downloads ALL images with certain tags and tag exclusion; No multithreading (10K ~5hr).
  
• app.py The Flask app Google Charts version of matplotlib. Does the same thing as main.py
  
• app_init.py Functions for the Flask app that does the counting. Change list of tags to track in the script via tag_name
  
• templates/index.htm The template page for the Flask app.
  
• Everything in /[OLD]/ Older versions of these scripts that used SQLite and JSON, before CSV dumps.

How To Use

• Download the CSV dump from e621. Place it in this directory.
• Open a Command Prompt and cd to this directory.
• Install requirements via pip to a venv. pip install -r requirements.txt
For main.py & app.py:
• Change tags you want to track via tag_name.
For downloader.py
• Change directory to download to via dld.
• Change tags to include via include_tags.
• Change tags to exclude via exclude_tags.
• Run the script.
• Profit?

Requirements

• Python311
• matplotlib
• pandas
• flask
• tqdm

TODO

• ☑ Downloader based on tag inclusion/exclusion (No multithreading to not overwhelm server).
• ☑ Google Charts via Flask app
• ☑ Use new CSV downloads.
• ☑ I want to download everything, every picture. and train a recognition algoritmn, or a new img generator based off of these.
• ☑ Update all files to use the new sqlite database.
• ⬜ Finish rule34 version of this.
• ☑ Optimize fetchall.py, so it won't take a day long to fetch.
• ☑ Find source of random 501s in fetchall.py.
• ☑ Extract created_at and tags to plot popularity of a tag over a time period.
• ☑ Create admin dashboard.
• ☑ omit_empty - Omit entries with value of 0.
• ☑ omit_final - Omit the final entry (which might skew the line low when run during the begining of the month).
• ☑ More attrative plots.
• ☑ Upload database to home server.
• ☑ Convert data-set to a db (json -> SQLite)..
• More Reports:
• ⬜ Compare Zootopia's influence on furry population with [a][s]'s data.
• ☑ How many liters/year based on cum? (Cum Counter TM)
• ☑ Species Explorer
• ☑ Jurassic Park Dino Dong.
• ⬜ Finish Species Explorer.
• ☑ Count general tags in a comic pool.

Expaned upon "explore621"* by [adjective][species] licensed under CC BY-NC-SA. Project source is licensed under the MIT License
*Domain no longer active, it's now a minecraft server? I woulda bought it, but they got to it fast.