[Question] Network route

[NikitaNesterenko]

Hello
When i use page.route(...) - next logic stopped and route is triggered after a certain time, it causes a lot of problems.
is there a way to deal with this?
For example exposeFunction good work and dont stopped next logic

[yury-s]

Sorry, it's hard to understand from your description what the problem is. Can you please attach your code snippet and point at the line which doesn't work as expected/breaks normal execution flow?

[NikitaNesterenko]

        page = context.newPage();
    
        page.route("**/**", route -> {
            logic... for example disable image
        });
        page.navigate("example.com");
        // navigate work
        login();  // dont work, in method logic ( use field page in class )

[yury-s]

What's inside route handler? What does login do? Note that if you do a blocking call inside page.route() it will block further nework traffic too, so may be that is your issue.

[NikitaNesterenko]

Login method contains logic to login ahaha
Click to button, fill input and etc.
In route logic -
if ( ... ) {

} else {
resume;
}
for example

I use example for disable images from doc
Logic stopped after page loaded

[yury-s]

I'm sorry, this is impossible to understand what your code does without seeing your code snippet. Please provide full code without replacing parts of it with comments explaining what it was doing, any sensitive information could be redacted but we need to see what exact calls you are making.

[NikitaNesterenko]

public class YoutubeTask extends Thread {

    private Page page;

    @Override
    public void run() {
         initialize();
    }

    private void initialize() {
        System.out.println("initializing...");
        try (Playwright playwright = Playwright.create()) {
            BrowserType browser = playwright.webkit();


            BrowserContext context = browser.launchPersistentContext(Path.of("nik_nesterenko_2020@mail.ru"),
                    new BrowserType.LaunchPersistentContextOptions()
                            .setHeadless(false)
                            .setArgs(Arrays.asList("--disable-blink-features=AutomationControlled", "--blink-settings=imagesEnabled=false"))
                            .setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15")
                            .setJavaScriptEnabled(true)
                            .setBypassCSP(true)
                            .setTimeout(100000)
            );

            page = context.newPage();

            page.route("**/*", route -> {
                if ("image".equals(route.request().resourceType()))
                    route.abort();
                else
                    route.resume();
            });

            page.navigate("https://www.youtube.com/");
            searchVideo();
        }
    }

    private void searchVideo() {
        // part of the method is executed, or not at all
        // Even sout doesn't work
        System.out.println("Searching video...");
        page.querySelector("#search").click();
        System.out.println("Searching cats...");   
        page.keyboard().type("cats", new Keyboard.TypeOptions().setDelay(300));
    }
}

[yury-s]

I modified your snippet to use chromium instead of webkit and changed run() to main(), it passed with no issues. I also ran it without "--blink-settings=imagesEnabled=false" argument to ensure that the branch that aborts requests gets executed. Everything worked fine. If printlns are not run it probaly means that page.navigate() doesn't finish, could it be that you are throttled by youtube and the navigation just takes longer? Try running it with DEBUG=pw:api to enable debug logging, it should give you some information on what's going on.

[NikitaNesterenko]

I did not refactor my code from threads to main, I deleted - "--blink-settings = imagesEnabled = false"
Everything is still the same
The YouTube page is fully loaded, even debugging requests are caught in the handler.
It can't be threads or Mac OS related?

[yury-s]

What does it print to console when launched with DEBUG=pw:api ?

It can't be threads or Mac OS related?

It shouldn't be. Do you run multiple threads with the same code?

[yury-s]

Does the task fail in isolation or only if you run them in parallel? You launch persistent context and chromium doesn't like when you have several processes sharing the same profile directory, it may cause undefined behavior. If you run several tasks in parallel either make sure that they use different profile directories (each of them should be accessed by one browser instance at a time) or better use launch() which uses ephemeral context.

[NikitaNesterenko]

for each thread its own hotel context is natural
tested only within threads
later I will test through pswm without streams and write about the results

[NikitaNesterenko]

I tested many options, it really works for some sites, and some get stuck or do not work correctly.
Is there another way to disable the loading of images on the site?

[yury-s]

The interception (page.route) should get all the network requests from the page, so this should work. You were going to collect some logs for the case where it failed with your example, did you succeed in it? We need more information about what's going on and what doesn't work, lack of the logs makes difficult for us to help you. Also if the code fails for youtube let's try to reproduce with that, otherwise it's hard to follow if you jump from one use case to another. Also what about using ephemeral contexts when you run in parallel, is there a reason you have to stick with persistent one?

[NikitaNesterenko]

I tested many options, no logs yet
This behavior is only seen when using webkit
If you use chronium, then everything is fine.