Use existing job queuing system for downloads

[deldesir]

Describe the bug/problem

Calibre-Web currently has apscheduler for managing various tasks, including format conversions, uploads, and metadata-related activities. While the present job queuing system functions adequately, there's a need to further optimize and adapt this mechanism specifically for download tasks.

The existing job queuing mechanism, for its general functionality and reliability, can be retained and optimized to better accommodate and efficiently manage download-related tasks within the Calibre-Web environment per feature #7.

The following changes are proposed:

1. Fine-tuning for Downloads: Optimize the current job queue to better handle and prioritize download-related tasks.
2. Load Distribution: Explore methods to evenly distribute and manage the load when handling concurrent or multiple downloads.
3. Error Handling and Monitoring: Enhance error handling and monitoring within the job queuing system to ensure the smooth execution of download tasks.

To Reproduce

There is no specific bug to reproduce; this issue pertains to code consistency and best practices within the Calibre-Web project.

Expected behavior

By optimizing the current job queuing mechanism specifically for download tasks, the goal is to ensure smoother and more efficient download operations while maintaining the reliability and stability of the existing system.

Environment (please complete the following information):

• OS: Ubuntu 23.10
• Python version: 3.11.6
• Calibre-Web version: 0.6.21 - Romesa (customized per Support additional media upload and download  #7 )
• Docker container: No
• Special Hardware: No
• Browser: Firefox Nightly 121.0a1 (2023-10-31) (64-bit)

Additional Context

The intention is to build upon the strengths of the current job queuing system, ensuring it's well-suited for download-related tasks without requiring a complete overhaul. This approach aims to maintain compatibility and stability while improving the efficiency of download task management within Calibre-Web.

[deldesir]

Implemented in #10 and #18

[holta]

Possible follow-ups:

1. FFmeg for generating missing covers
2. Post-process videos/thumbnails ?
3. Banner auto-cropping, similar to youtube2zim ?

[holta]

Tangentially related questions around how educators/parents/curriculum designers can monitor their ongoing "Download to IIAB" downloads' progress:

• Log progress while downloading chapmanjacobd/library#26
• Download tasks don't seem to run simultaneously, causing delays #58