OverFast API provides comprehensive data on Overwatch 2 heroes, game modes, maps, and player statistics by scraping Blizzard pages. Developed with the efficiency of FastAPI and Beautiful Soup, it leverages nginx as a reverse proxy and Redis for caching. Its tailored caching mechanism significantly reduces calls to Blizzard pages, ensuring swift and precise data delivery to users.
- ✨ Live instance
- 🐋 Run for production
- 💽 Run as developer
- 👨💻 Technical details
- 🐍 Architecture
- 🤝 Contributing
- 🚀 Community projects
- 🙏 Credits
- 📝 License
The live instance is restricted to 30 req/s per IP (a shared limit across all endpoints). If you require more, consider hosting your own instance on a server 👍
- Live instance (Redoc documentation) : https://overfast-api.tekrop.fr/
- Swagger UI : https://overfast-api.tekrop.fr/docs
- Status page : https://stats.uptimerobot.com/E0k0yU1pJQ
Running the project is straightforward. Ensure you have docker and docker compose installed. Next, generate a .env file using the provided .env.dist template. Finally, execute the following command:
make upSame as earlier, ensure you have docker and docker compose installed, and generate a .env file using the provided .env.dist template. You can customize the .env file according to your requirements to configure the volumes used by the OverFast API.
Then, execute the following commands to launch the dev server :
make build # Build the images, needed for all further commands
make start # Launch OverFast API (dev mode)The dev server will be running on the port 8000. You can use the make down command to stop and remove the containers. Feel free to type make or make help to access a comprehensive list of all available commands for your reference.
Should you wish to customize according to your specific requirements, here is a detailed list of available settings:
APP_VOLUME_PATH: Folder for shared app data like logs, Redis save file and dotenv file (app settings)APP_PORT: Port for the app container (default is80).APP_BASE_URL: Base URL for exposed links in endpoints like player search and maps listing.
You likely won't need to modify other generic settings, but if you're curious about their functionality, consult the docstrings within the app/config.py file for further details.
The code quality is checked using ruff. I'm also using ruff format for imports ordering and code formatting, enforcing PEP-8 convention on my code. To check the quality of the code, you just have to run the following command :
make lint # Run ruff linter
make format # Run ruff formatterThe code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers classes, the common classes, but also on the commands (run in CLI) and the API views (using FastAPI TestClient class).
Running tests with coverage (default)
make testRunning tests with given args
make test PYTEST_ARGS="tests/common"The project is using pre-commit framework to ensure code quality before making any commit on the repository. After installing the project dependencies, you can install the pre-commit by using the pre-commit install command.
The configuration can be found in the .pre-commit-config.yaml file. It consists in launching 2 processes on modified files before making any commit :
rufffor linting and code formatting (withruff format)sourceryfor more code quality checks and a lot of simplifications
In player career statistics, various conversions are applied for ease of use:
- Duration values are converted to seconds (integer)
- Percent values are represented as integers, omitting the percent symbol
- Integer and float string representations are converted to their respective types
OverFast API integrates a Redis-based cache system, divided into two main components:
- API Cache: This high-level cache associates URIs (cache keys) with raw JSON data. Upon the initial request, if a cache entry exists, the nginx server returns the JSON data directly. Cached values are stored with varying TTL (Time-To-Live) parameters depending on the requested route.
- Parser Cache: Specifically designed for the API's parsing system, this cache stores parsing results (JSON objects) from HTML Blizzard pages. Its purpose is to minimize calls to Blizzard servers when requests involve filters. The cached values are refreshed in the background prior to expiration.
Here is the list of all TTL values configured for API Cache :
- Heroes list : 1 day
- Hero specific data : 1 day
- Roles list : 1 day
- Gamemodes list : 1 day
- Maps list : 1 day
- Players career : 1 hour
- Players search : 1 hour
sequenceDiagram
autonumber
participant Redis
participant Worker
participant Blizzard
Worker->>+Redis: Request expiring Parser Cache
Redis-->>-Worker: Return expiring Parser Cache
alt Some Parser Cache will expire
Worker->>+Blizzard: Request up-to-date data
Blizzard-->>-Worker: Return up-to-date data
Worker->>+Redis: Update cache values
end
To minimize requests to Blizzard servers, a Refresh-Ahead cache system has been deployed.
Upon the initial request for a player's career page, there may be a slight delay (approximately 2-3 seconds) as data is fetched from Blizzard. Following this, the computed data is cached in the Parser Cache, which is subsequently refreshed in the background by a dedicated worker, before expiration. Additionally, the final data is stored in the API Cache, which is generated only upon user requests.
This approach ensures that subsequent requests for the same career page are exceptionally swift, significantly enhancing user experience.
sequenceDiagram
autonumber
actor User
participant Nginx
participant Redis
participant App
User->>+Nginx: Make an API request
Nginx->>+Redis: Make an API Cache request
alt API Cache is available
Redis-->>Nginx: Return API Cache data
Nginx-->>User: Return API Cache data
else
Redis-->>-Nginx: Return no result
Nginx->>+App: Transmit the request to App server
App->>+Redis: Make Parser Cache request
alt Parser Cache is available
Redis-->>App: Return Parser Cache
else
Redis-->>-App: Return no result
App->>App: Parse HTML page
end
App-->>-Nginx: Return API data
Nginx-->>-User: Return API data
end
Utilizing docker compose, this architecture provides response cache saving into Redis. Subsequent requests are then directly served by nginx without involving the Python server at all. This approach strikes the optimal performance balance, leveraging nginx's efficiency in serving static content. Depending on the configured Blizzard pages, a single request may trigger multiple Parser Cache requests.
Contributions, issues and feature requests are welcome ! Do you want to update the heroes data (health, armor, shields, etc.) or the maps list ? Don't hesitate to consult the dedicated CONTRIBUTING file.
Projects using OverFast API as a data source are listed below. Using it in your project? Reach out via email with your project link, and I'll add it!
- Datastrike, analysis and results tracking tool (https://datastrike.cloud)
- Discord Bot OW2 for stats (https://github.com/polsojac/ow2discordbot)
- OverBot, the best Overwatch bot for Discord (https://github.com/davidetacchini/overbot)
- Overfast API client (https://github.com/Sipixer/overfast-api-client)
- Overwatch Career Profile (https://github.com/EliaRenov/ow-career-profile)
- OverwatchPy, a Python wrapper for the API (https://github.com/alexraskin/overwatchpy)
- Watch Over, mobile app by @Backxtar (https://play.google.com/store/apps/details?id=de.backxtar.watchoveroverwatch)
All maps screenshots hosted by the API are owned by Blizzard. Sources :
- Blizzard Press Center (https://blizzard.gamespress.com)
- Overwatch Wiki (https://overwatch.fandom.com/wiki/)
Copyright © 2021-2024 Valentin PORCHET.
This project is MIT licensed.