Crawl nested html documents loaded dynamically

[ab14p]

What is the current behavior?

#150

With robots.txt set to false

Crawler waits using WaitFor and a specified timeout of 10 seconds, but still not able to crawl nested documents. For example, I want to extract src of an <iframe> which is an AD (display advertisement).

Enabled screen shot option to check if the ad iframe has loaded before evaluatePage function was executed. I can see the ad in screen shot but function does not return the src from <iframe>.

What is the expected behavior?

Be able to crawl nested html documents such as ADs which are loaded dynamically by Java Script.

Can you please provide an example or solution for this..

Please tell us about your environment:

• Version: 1.5.0
• Platform / OS version:
• Node.js version: 6.11.2

[BubuAnabelas]

Maybe you could use the waitUntiloption in the crawler.queue() method in combination with waitFor, or passing a function to waitFor checking if the ad was loaded.

[yujiosaka]

@ab14p
Isn't it because it was blocked by Cross Origin Policy?
Please try passing --disable-web-security argument. See https://github.com/yujiosaka/headless-chrome-crawler#launch-options for more details.

[ab14p]

@yujiosaka @BubuAnabelas
I used waitUntil: 'networkidle0' and waitFor: { selectorOrFunctionOrTimeout: 20000 }
I did try setting up Cross Origin policy in launch options as shown below, but still no luck.
args: ['--disable-web-security']

[yujiosaka]

@ab14p
Still not encountered the same situation.
Can you provide the code?

[yujiosaka]

Closing this issue because no information is provided for a month.
Please feel free to reopen it if you have further questions.