FAL Scraper Script is a tool to help play Fantasy Anime League on MyAnimeList.
It works by polling MAL's API for data on requested seasonal shows and displaying the FAL-relevant bits in a compact, human-readable Excel table that is easy to work with. It can be plugged into the OS task scheduler for automatic (e.g. weekly, daily, or hourly) running or be run on demand.
Note: Requires MyAnimeList API key.
For more information, see:
Fantasy Anime League
moozooh's Comprehensive Guide to FAL
What's new
This is a major feature update. So major, in fact, that the code is now five times longer and all the old code aside from some sheet formatting bits has been rewritten from the ground up. Let's see what we have here...
Added:
- An optional secondary data sheet to track specific titles separately for easier data manipulation (e.g. if you're copying or parsing data from the Excel file to use elsewhere).
- An optional sheet with a full and 100% accurate forum post breakdown.
- Automatic season detection and forum post counting period heuristic with an optional date override.
- More data columns: Posts, Status, ID.
- Column hiding and sorting.
- Error handling and input validation (not exhaustive for now, but better than nothing).
- Adjustable console output verbosity levels and logging output to a file.
Improved:
- List formatting and color coding were adjusted and made more neat; certain columns including Watching and Completed are hidden by default now that the W+C column always displays relevant numbers.
- Status heuristics: now correctly detects early previews in almost every case. Known exceptions are shows with very low starting viewer count; some further tweaking might be warranted.
- Output naming: now includes a custom season name to resolve conflicts for cases when you're running multiple script instances, e.g. for current and future seasons, testing, etc.
- Almost every part of the code has been completely refactored with asynchronous calls, batching, other optimizations, and Python 3.12 compatibility. It's blazing fast now! Processing 30 seasonal shows together with all the post data and some duplicates on the alt sheet takes roughly 7 seconds.
- Code is much better documented now.
Compatibility notes:
- The script will no longer function on Python 3.8 correctly.
- 3.9 through 3.11 are untested but should work. 3.13 is untested and may work.
- 3.12 is fully tested and works well.
A note on post fetching
This function has been in the works for almost a year because I wanted to maximize accuracy and be able to see live data at any given moment. It took a lot of thinking, studying, and trial and error with the invaluable assistance of ChatGPT and Claude to get everything right. It works as follows:
- Visit the episode discussion page of a given anime's forum via an HTTP call and parse the links to discussion threads and the last post, collect all useful information along the way.
- Fetch the data from each thread via API calls.
- Fetch the last post via HTTP again (because the API doesn't expose it due to a bug).
- Rinse and repeat for all IDs specified.
- Compile all the data neatly on a separate sheet.
- Using the data collected, calculate and add the number of posts relevant to the current post counting period in a now visible column on the main sheet.
Post fetching is disabled by default to ensure it is never used unknowingly, because in a season with, say, 30 tracked shows and 12 episodes to discuss with an average of 40 posts each, and no client-side throttling, this will siphon about 14,400 posts off the site in a matter of seconds. That is not a very polite number of requests to suddenly slam a server with even if we do our best to batch them and whatnot, so please use this feature very sparingly and responsibly: remove unneeded IDs as the season goes on, never schedule automatic execution more frequent than once an hour, don't test operation with large numbers of IDs at a time, use the provided user variables for self-throttling. Failing to observe these recommendations risks having your MAL API key revoked.
A note on verbosity
There are now four levels of verbosity which I had to add to debug this unholy mess, controlled by the log_verbosity variable.
- Level 0 (silent; console-only): Print exit code only.
- Level 1 (basic; default): In addition to the above, print principal operation steps and status of important variables.
- Level 2 (verbose): In addition to the above, print every fetch request.
- Level 3 (debug; log-only): In addition to the above, print every response and the content of almost every populated dictionary and variable, redundancy be damned.
If you're automating continuous data collection and would like to keep track of scheduled script runs like a big boy/girl, level 1 should have you covered for most intents and purposes. God help you if you ever need to use level 3 for anything; I only left it there as a testament to my tenacity in seeing this release through without giving up. As demonstrated by the fact that it took almost a year to finish this, programming isn't remotely as fun as the idea of programming...
Future development
- A standalone GUI application (no, I have not given up on it, but it is low-priority for now).
- Better error handling.
- Bug fixes, if warranted.
- Optimizations, if found, aside from converting the code to Python 3.13.
How to run
- Request the API key from MAL.
- Install Python 3.12. Make sure it installs pip (the package manager) and environment variables.
- Open the command line and install the missing modules via
pip install aiohttp openpyxl beautifulsoup4. - Download the script, open it in any text editor or IDE, and configure the user variables (most importantly the API key).
- Save the script and run it (such as by double-clicking the file).
- Enjoy your data.