RoyaleAPI v4.0.0-beta

🚀 Anndromeda™ RoyaleAPI v4.0.0-beta

Release Date: 2025-02-13
IMPORTANT: This version of RoyaleAPI is deprecated due to Traderie changing its HTML structure. The release is provided solely for future development reference and is not functional with the current Traderie platform.
Note: Versions before v4.0.0-beta were documented under ProjectAnndromeda. This release marks the first official version on the Xelvanta account. Anndromeda™ operates as a sister company to Xelvanta Group and are part of the same parent organization.

📌 Highlights

• 🔹 Refactored Scraper Logic – Improved modular structure by breaking down tasks into smaller, more maintainable functions.
• 🔹 Optimized Concurrency – Added dynamic task management using asyncio.Semaphore to limit concurrent requests and reduce server load.
• 🔹 Enhanced Error Handling – Improved exception handling for network-related issues (net::ERR_NETWORK_CHANGED) and retries for failed attempts.
• 🔹 Cancellation Flag – Introduced a flag to stop scraping when no items are found, improving efficiency by avoiding unnecessary requests.

🛠️ Bug Fixes

• 🐞 Network Error Handling – Improved retry logic for net::ERR_NETWORK_CHANGED, ensuring network issues don’t prematurely stop the scraping process.
• 🐞 Timeout Handling – Increased timeout for page.goto() to 60 seconds, reducing unnecessary failures during slow page loads.
• 🐞 No More Items Found – Fixed the handling of empty pages by checking for "No results found" before proceeding with scraping.
• 🐞 Page Data Parsing – Fixed an issue where parsing errors (e.g., extracting item names or values) would cause empty results or incorrect data to be returned.

📦 Improvements

• 📈 Performance Boost – Reduced delay between requests (from 1s to 0.03s) and optimized task concurrency, significantly decreasing the total scraping time.
• 📈 Better Logging – Enhanced logging to provide more detailed error messages, task status updates, and debugging information.
• 📈 Improved Task Management – Implemented dynamic task scheduling, ensuring only a defined number of concurrent requests are active at any time.
• 📈 Early Return for No Results – Introduced early termination logic when no items are found on a page, conserving resources by halting further requests.

🔧 Known Issues

• ⚠️ Minor Timeout Issues – Occasionally, requests to extremely slow pages might still exceed the timeout despite extended limits. Monitoring is in place.
• ⚠️ Slow Scraping Due to Pagination – Scraping may be slower than expected due to the need to load multiple pages sequentially. Each page fetch introduces a delay, especially when handling large datasets across many pages. Optimization for more efficient pagination handling is under consideration.