Concurrency Implementation Feedback

[Alexander-Welsh]

Problem Description

I have a puppeteer project that has been leveraging this project for a while, it's been pretty seamless. However, I recently attempted to integrate some proxying into the project and ran into issues while attempting to do this in an incognitoBrowserContext. The --proxy-server flag doesn't work as expected on launch when using an incognito context, a solution is to pass a proxyServer arg on each call of createIncognitoBrowserContext, which is (reasonably) obfuscated by this package.

Resolution

In order to solve this I used an "experimental" custom concurrency and extended the LaunchOptions interface to include the proxyServer.

// custom concurrency
export default class CustomProxyableIncognitoContext extends SingleBrowserImplementation {
  protected async createResources(): Promise<ResourceData> {
    const { proxyServer } = this.options as puppeteer.LaunchOptions & { proxyServer: string };
    const context = await (this.browser as puppeteer.Browser)
      .createIncognitoBrowserContext({ proxyServer });
    const page = await context.newPage();

    return {
      context,
      page,
    };
  }

  protected async freeResources(resources: ResourceData): Promise<void> {
    await resources.context.close();
  }
}

// Cluster Launch, with args
module.exports = Cluster.launch({
  puppeteer,
  concurrency: CustomProxyableIncognitoContext,
  maxConcurrency: MAX_CONCURRENCY,
  puppeteerOptions: {
    args: launchArgs,
    ignoreHTTPSErrors: true,
    defaultViewport: { width: VIEWPORT_WIDTH, height: VIEWPORT_HEIGHT },
    proxyServer: process.env.PROXY_URL,
  },
  // Library default timeout is 30000
  timeout: 60000,
  headless: true,
  monitor: MONITOR,
});

Questions/thoughts

• Do you have any objections to exporting the ResourceData interface and SingleBrowserImplementation class, so that I don't have to keep a copy locally or grab them from the node_modules?
• Just wanted to drop some feedback that even though this is "experimental" it works well and I find it valuable and would appreciate it if it stayed in the API