feat: use manual proxy rotation for web scraping 🔁

[ramiAbdou]

Description ✏️

This PR will help reduce costs for web scraping by using a set of dedicated proxies instead of a browser instance that various SaaS companies offer. Now that we have a dedicated set of proxies, we need to use them with puppeteer.

The key here is we need to tell puppeteer we are using a proxy server, then we need to authenticate with it, ie:

const context = await browser.createBrowserContext({ proxyServer: proxy });

const page = await context.newPage();

await page.authenticate({
  password: OXYLABS_PASSWORD,
  username: OXYLABS_USERNAME,
});

This PR:

• Introduces the OXYLABS_USERNAME, OXYLABS_PASSWORD, and OXYLABS_PROXIES environment variables.
• Removes the DEFAULT_TIMEOUT on the puppeteer settings (already defaults to 30 seconds).
• Uses the user-agents package for more realistic and more random user agent headers.

Type of Change 🐞

• Feature - A non-breaking change which adds functionality.
• Fix - A non-breaking change which fixes an issue.
• Refactor - A change that neither fixes a bug nor adds a feature.
• Documentation - A change only to in-code or markdown documentation.
• Tests - A change that adds missing unit/integration tests.
• Chore - A change that is likely none of the above.

Checklist ✅

• I have done a self-review of my code.
• I have manually tested my code (if applicable).
• I have added/updated any relevant documentation (if applicable).