feat: LinkContentFetcher - replace requests with httpx, add async and http/2

[vblagoje]

Why:

Enhances the LinkContentFetcher functionality by introducing asynchronous content fetching capabilities and optional HTTP/2 support, replacing requests with httpx to improve performance and reliability.

What:

• Added asynchronous methods for fetching web content, significantly improving efficiency for parallel requests.
• Incorporated support for HTTP/2 using httpx with lazy loading for the h2 library.
• Replaced requests with httpx for both synchronous and asynchronous HTTP operations, including custom client configurations.

How can it be used:

• To fetch web content asynchronously:

  async def fetch_async():
      fetcher = LinkContentFetcher(http2=True)
      results = await fetcher.run_async(urls=["https://example.com"])
      return results["streams"]
  streams = asyncio.run(fetch_async())

• To enable HTTP/2:

  fetcher = LinkContentFetcher(http2=True)

How did you test it:

Implemented and utilized comprehensive unit and integration tests using pytest, focusing on both synchronous and asynchronous fetching paths, various content types, and error handling scenarios.

Notes for the reviewer:

Pay attention to the lazy import of HTTP/2 support and verify if error handling is comprehensive for network failures and import issues with h2. Review the updated test cases to ensure all edge cases are covered.

[coveralls]

Pull Request Test Coverage Report for Build 13929227857

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 25 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.02%) to 90.092%

---

Files with Coverage Reduction	New Missed Lines	%
components/fetchers/link_content.py	25	86.03%

Totals
Change from base Build 13920793695:	0.02%
Covered Lines:	9802
Relevant Lines:	10880

---

💛 - Coveralls

[vblagoje]

@julian-risch @dfokina I messed up a bit #9032 so this one is superseding it

[julian-risch]

Looks quite good to me already! I suggest merging the two test files. Other than that just minor comments. And I was wondering what you think about customizable headers as a separate issue?

[julian-risch]

I suggest we merge the two files into one.
It's more consistent with the rest of the code base (one file with all test cases per component) and less code duplication (HTML_URL, etc.)

[vblagoje]

Ahaha I wanted to have one to begin with but I made two files because I saw that we have separate files for asynchronous and synchronous in openai chat generator. See https://github.com/deepset-ai/haystack/tree/main/test/components/generators/chat Perhaps we can chat internally about it and whatever we decide - I'll adjust

[julian-risch]

Given that users reported issues using LinkContentFetcher, I think the component would benefit from making REQUEST_HEADERS customizable. Separate issue and PR maybe? What do you think?

[vblagoje]

Yes let's go for separate issue in the next sprint

[julian-risch]

only one small change request remaining: merging the two test files. Unless you see a good reason to keep them separate. https://github.com/deepset-ai/haystack/pull/9034/files#r1995051947