fix: update ChromeDriver options on restricted environments and add ChromeDriver options as function parameter

[danielbichuetti]

Related Issues

• fixes Crawler is crashing in some container environments #2946

Proposed Changes:

• Add a webdriver_options parameter, so the user can set the parameters that his specific scenario demands
• Automatically set some Webdriver options in Azure ML and Google Colab to avoid crashes
  • --disable-dev-shm-usage
• Automatically set some Webdriver options in scenario where Chrome will run as root to avoid crashes
• --no-sandbox
• Change default parameters:
  • --disable-gpu There is no need to enable GPU acceleration on a text crawler. GPU is expensive in the cloud.
  • --disable-dev-shm-usage Usually in container environments there is no access to shared memory, or it's set with the default size of 64 MB. Disabling its usage, Chrome will write to a temporary directory. Using shared memory may improve performance just in high testing workloads, which is not our use case.
    - --single-process As haystack doesn't support multi-tabs, there is no point to let Chrome spawn multiple processes. It will just increase the memory footprint.

How did you test it?

• Unit tests
• Manual tests based on python:3.9-slim-bullseye container image
• Manual tests inside Azure ML
• Manual tests inside haystack Dockerfiles

Notes for the reviewer

This changes essentially will make haystack Crawler node able to run in multiple environments without any user extra effort. In case any user has more specific needs, it will be possible using the parameter.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I've used the conventional commit convention for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

[agnieszka-m]

Hey, I added some minor language comments to comply with our writing guidelines. Thanks!

[agnieszka-m]

Thanks!

[sjrl]

@danielbichuetti This is looking good to me! Although I think we should have someone from the core-engineering team for the final approval.

[ZanSara]

Looks good, thank you!