Something in crawlee is masking remote IP adresses

[matjaeck]

Consider following code:

{
  "type": "module",
  "dependencies": {
    "crawlee": "3.1.4",
    "playwright": "1.27.1"
  }
}

import { chromium } from "playwright"

console.log("Scraping server ip addresses using playwright only")
const browser = await chromium.launch()
const context = await browser.newContext()
const page = await context.newPage()
const session = await context.newCDPSession(page)
await setupResponseIpLogging(session)
await page.goto("https://google.de")
await browser.close()

console.log("Scraping server ip addresses using crawlee playwright crawler")

import { PlaywrightCrawler, RequestQueue } from "crawlee"
const navigationQueue = await RequestQueue.open()
await navigationQueue.addRequest({ url: "https://google.de" })
const crawler = new PlaywrightCrawler({
    requestQueue: navigationQueue,
    preNavigationHooks: [
        async ctx => {
            const context = ctx.page.context()
            const session = await context.newCDPSession(ctx.page)
            await setupResponseIpLogging(session)
        }
    ],
    requestHandler: () => { }
})
await crawler.run()
process.exit()

async function setupResponseIpLogging(session) {
    session.on("Network.responseReceived", (event) => {
        const ip = event.response.remoteIPAddress
        if (ip) console.log(ip)
    })
    await session.send("Network.enable")
}

This (npm i && node index.js) outputs something like this:

Scraping server ip addresses using playwright only
172.217.23.99
172.217.23.99
172.217.23.99
172.217.23.99
172.217.23.99
216.58.212.163
142.250.185.99
142.250.185.99
172.217.23.99
142.250.185.142
Scraping server ip addresses using crawlee playwright crawler
INFO  PlaywrightCrawler: Starting the crawl
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1
127.0.0.1

Obviously something in the engine is kind of masking the remote IP addresses, so that they always point to localhost. Does anybody have an idea how to stop it from doing that?

[B4nan]

Probably this, but I don't recall the reason from the top of my head.

https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/playwright/playwright-plugin.ts#L48-L52

[B4nan]

This is the commit that introduced it:

apify/browser-pool@c44dd8d

Which links to this:

https://github.com/microsoft/playwright/blob/2e4722d460b5142267e0e506ca7ea9a259556b5f/packages/playwright-core/src/server/browserContext.ts#L423-L427

[matjaeck]

Thank you, hmm I'll let you know if that causes the issue I described. The playwright workaround however seems to be platform-specific (os.platform() === 'win32)? Perhaps it could be made a platform thing in crawlee too...

[B4nan]

Perhaps it could be made a platform thing in crawlee too...

That makes sense for sure

[matjaeck]

Probably this, but I don't recall the reason from the top of my head.

https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/playwright/playwright-plugin.ts#L48-L52

Indeed, this is causing it.

[2tunnels]

Is there any plan to address this or provide an option to preserve the original values? I’d suggest creating an issue for it since it might be important for those relying on accurate CDP logs.

[janbuchar]

Hi, it's not currently planned, but feel free to open an issue so that it won't get lost here.