Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

There's currently no way to wait for a condition except with WaitFunc and a selector #680

Closed
ghost opened this issue Aug 7, 2020 · 4 comments

Comments

@ghost
Copy link

ghost commented Aug 7, 2020

What I want is to evaluate some JS repeatedly until it returns the expected result. There seems to be no other way to do that but with a clumsy code like this, using WaitFunc and a random selector:

chromedp.Query("random-selector",
	func(s *chromedp.Selector) {
		chromedp.WaitFunc(func(ctx context.Context, cur *cdp.Frame, ids ...cdp.NodeID) ([]*cdp.Node, error) {
			if len(ids) == 0 {
				return nil, nil
			}
			var result string
			if err := chromedp.Evaluate(someJavascript, &result).Do(ctx); err != nil {
				return nil, err
			} else if result != whatWeAreWaitingFor {
				return nil, nil
			}
			return []*cdp.Node{}, nil
		})(s)
	},
	chromedp.AtLeast(0),
)

It would be great if one could wait for a JS condition using a simpler action.

@ghost
Copy link
Author

ghost commented Aug 7, 2020

FWIW, puppeteer has waitForFunction:

https://devdocs.io/puppeteer/index#pagewaitforfunctionpagefunction-options-args

@yoyo00xx
Copy link

yoyo00xx commented Mar 6, 2021

There should be something like the puppeteer API

@ZekeLu
Copy link
Member

ZekeLu commented Mar 7, 2021

@opennota It seems that you just want the wait part of chromedp.Query. It's implemented like this:

chromedp/query.go

Lines 161 to 166 in e297055

for {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(5 * time.Millisecond):
}

Borrow this implementation, your code can be changed to (I have extended it to be runnable, and it's based on chromedp v0.6.5):

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/chromedp/chromedp"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprint(w, `
<html>
<script>
let v = 0;
setTimeout(()=>{v = 100}, 1000);
</script>
<body>
hello world
</body>
</html>
`)
	}))
	defer ts.Close()

	ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))
	defer cancel()
	ctx, cancel = context.WithTimeout(ctx, 5*time.Second)
	defer cancel()

	if err := chromedp.Run(ctx,
		chromedp.Navigate(ts.URL),
		chromedp.ActionFunc(func(ctx context.Context) error {
			for {
				select {
				case <-ctx.Done():
					return ctx.Err()
				// you can customize the interval below
				case <-time.After(5 * time.Millisecond):
				}
				var result bool
				err := chromedp.Evaluate(`v === 100`, &result).Do(ctx)
				if err != nil {
					return err
				}
				if result {
					return nil
				}
			}
		}),
	); err != nil {
		log.Fatal(err)
	}
}

The problem of this implementation is that it involves many roundtrips on the WebSocket connection. I have looked into the implementation of https://devdocs.io/puppeteer/index#pagewaitforfunctionpagefunction-options-args, and found that the wait part is totally happens in the browser, and there is just one roundtrip on the WebSocket connection.

The following implementation is borrowed from https://devdocs.io/puppeteer/index#pagewaitforfunctionpagefunction-options-args (It uses Runtime.callFunctionOn and requires an executionContextId, which is not exported by chromedp. We will use chrome.Query to get the executionContextId):

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"

	"github.com/chromedp/cdproto/cdp"
	"github.com/chromedp/cdproto/runtime"
	"github.com/chromedp/chromedp"
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprint(w, `
<html>
<script>
let v = 0;
setTimeout(()=>{v = 100}, 3000);
</script>
<body>
hello world
</body>
</html>
`)
	}))
	defer ts.Close()

	ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithDebugf(log.Printf))
	defer cancel()

	// copied from puppeteer
	waitForPredicatePageFunction := `async function waitForPredicatePageFunction(predicateBody, polling, timeout, ...args) {
    const predicate = new Function('...args', predicateBody);
    let timedOut = false;
    if (timeout)
        setTimeout(() => (timedOut = true), timeout);
    if (polling === 'raf')
        return await pollRaf();
    if (polling === 'mutation')
        return await pollMutation();
    if (typeof polling === 'number')
        return await pollInterval(polling);
    /**
     * @returns {!Promise<*>}
     */
    async function pollMutation() {
        const success = await predicate(...args);
        if (success)
            return Promise.resolve(success);
        let fulfill;
        const result = new Promise((x) => (fulfill = x));
        const observer = new MutationObserver(async () => {
            if (timedOut) {
                observer.disconnect();
                fulfill();
            }
            const success = await predicate(...args);
            if (success) {
                observer.disconnect();
                fulfill(success);
            }
        });
        observer.observe(document, {
            childList: true,
            subtree: true,
            attributes: true,
        });
        return result;
    }
    async function pollRaf() {
        let fulfill;
        const result = new Promise((x) => (fulfill = x));
        await onRaf();
        return result;
        async function onRaf() {
            if (timedOut) {
                fulfill();
                return;
            }
            const success = await predicate(...args);
            if (success)
                fulfill(success);
            else
                requestAnimationFrame(onRaf);
        }
    }
    async function pollInterval(pollInterval) {
        let fulfill;
        const result = new Promise((x) => (fulfill = x));
        await onTimeout();
        return result;
        async function onTimeout() {
            if (timedOut) {
                fulfill();
                return;
            }
            const success = await predicate(...args);
            if (success)
                fulfill(success);
            else
                setTimeout(onTimeout, pollInterval);
        }
    }
}
`

	var exeCtxID runtime.ExecutionContextID
	// run
	if err := chromedp.Run(ctx,
		chromedp.Navigate(ts.URL),
		// to get the runtime.ExecutionContextID
		chromedp.Query("random-selector",
			func(s *chromedp.Selector) {
				chromedp.WaitFunc(func(ctx context.Context, cur *cdp.Frame, id runtime.ExecutionContextID, ids ...cdp.NodeID) ([]*cdp.Node, error) {
					exeCtxID = id
					return []*cdp.Node{}, nil
				})(s)
			},
			chromedp.AtLeast(0),
		),
		chromedp.ActionFunc(func(ctx context.Context) error {
			predicateBody, err := json.Marshal(`return (()=>v === 100)(...args);`)
			if err != nil {
				return err
			}
			timeout, err := json.Marshal(5000)
			if err != nil {
				return err
			}
			polling, err := json.Marshal("raf")
			if err != nil {
				return err
			}
			// I just ignore the result below. It should parsed to check whether it's timeout
			_, _, err = runtime.CallFunctionOn(waitForPredicatePageFunction).
				WithExecutionContextID(exeCtxID).
				WithReturnByValue(false).
				WithAwaitPromise(true).
				WithUserGesture(true).
				WithArguments([]*runtime.CallArgument{
					{Value: predicateBody},
					{Value: polling},
					{Value: timeout},
				}).Do(ctx)
			return err
		}),
	); err != nil {
		log.Fatal(err)
	}
}

@ZekeLu
Copy link
Member

ZekeLu commented Apr 27, 2021

This issue should have been addressed by #766. Please file a new issue if you find bugs in that feature. Thank you!

@ZekeLu ZekeLu closed this as completed Apr 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants