-
Notifications
You must be signed in to change notification settings - Fork 284
Description
And here's an example for chromedp:
// The CSS selector for the element you must wait for
const selectorToWaitFor = "a"
var htmlContent string
const targetURL = "https://example.com"
log.Printf("Navigating to %s, waiting for '%s', and retrieving source...", targetURL, selectorToWaitFor)
err := chromedp.Run(ctx,
// 1. Navigate to the target URL
chromedp.Navigate(targetURL),
// 2. Add the waiting condition here
// This pauses execution until the element is visible in the DOM
chromedp.WaitVisible(selectorToWaitFor, chromedp.ByQuery),
// 3. Retrieve the OuterHTML of the entire 'html' element
chromedp.OuterHTML(`html`, &htmlContent, chromedp.ByQuery),
)It works for example.com, but not another site (with the same code) which uses JS to load its content (targetURL = "https://so.gitee.com/?q=Vue%20go", selectorToWaitFor = "a[aria-label=Next]") --
I'm using the latest github.com/chromedp/chromedp (v0.14.2), and
I set a generous overall timeout using context.WithTimeout (300 seconds) for the entire operation. However, during execution of specific actions (chromedp.WaitVisible), the program consistently fails with a "context canceled" error after approximately 7 seconds. The actual body returned is (for sure the element is not found):
<div class="d-flex justify-content-center align-middle" style="height: 80vh;"><div class="m-auto"><div class="spinner-grow spinner-grow-sm text-success" role="status"><span class="visually-hidden">Loading...</span></div> <div class="spinner-grow spinner-grow-sm text-success" role="status"><span class="visually-hidden">Loading...</span></div> <div class="spinner-grow spinner-grow-sm text-success" role="status"><span class="visually-hidden">Loading...</span></div></div></div></body></html>The site is heavy JS based, the initial load takes 6~7 seconds. then the browser should run JS to finish loading.
Using go-rod, and waiting for the same element, it can retrieve the whole JS loaded content.
The issue might be a lightpanda cloud browser issue, as I am using Google Chrome browser using browser=chrome parameter in the url for go-rod, yet when adding browser=chrome parameter in the url for chromedp, it failed with the following error (so I'm not sure where exactly the problem is):
Visiting 'https://example.com' (& wait for 'a') ...
2025/11/14 09:26:50 -> {"id":1,"method":"Target.createTarget","params":{"url":"about:blank","enableBeginFrameControl":false,"newWindow":false,"background":false,"forTab":false,"hidden":false}}
2025/11/14 09:26:51 <- {"id":1,"error":{"code":-32000,"message":"Failed to open new tab - no browser is open"}}
...
exit status 1