feat: added timeouts to requests to prevent blocking requests

[RRua]

Pull Request: Add Optional Timeout Parameters to HTTP Requests

Summary

This pull request introduces optional timeout parameters to all functions in the FirecrawlApp class that make HTTP requests. This enhancement improves the robustness and resilience of the application by ensuring that calls do not hang indefinitely, which helps prevent resource starvation and supports better error handling.

Key Changes

• Added timeout as an optional argument to the following functions:

  • scrape_url
  • crawl_url
  • async_crawl_url
  • check_crawl_status
  • ....

• Updated internal _post_request and _get_request methods to use the timeout value when provided.

• Default behavior remains unchanged if timeout is not specified, ensuring backward compatibility.

Benefits

• Prevents indefinite blocking on slow or unresponsive network requests.
• Enhances system reliability and resilience by managing resource utilization more effectively.
• Offers users the flexibility to customize timeout settings based on their specific needs.

[mogery]

LGTM, thanks!

[nickscamara]

• Update the docs
• Maybe update the name to not conflict with params.timeout