Error: No data found for resource with given identifier #260
Comments
|
This happens because AFAIK you're only allowed to call I implemented this in the following using a const CDP = require('chrome-remote-interface');
setTimeout(() => {
CDP(async (client) => {
const {Network, Page, Runtime} = client;
const requests = new Set(); // <---------- HERE
Network.requestWillBeSent(({requestId, request}) => {
if(request.url.indexOf("ct2/results/rpc") != -1){
console.log(`REQ [${requestId}] ${request.method} ${request.url} \n`);
requests.add(requestId); // <---------- HERE
}
});
Network.loadingFinished(async ({requestId}) => {
if (requests.has(requestId)) { // <---------- HERE
const {body, base64Encoded} = await Network.getResponseBody({requestId});
console.log(`RES [${requestId}] body: ${body} \n`);
}
});
try {
await Promise.all([Network.enable(), Page.enable()]);
await Page.navigate({url: 'https://clinicaltrials.gov/ct2/results?cond=Parents&term=&cntry1=&state1=&Search=Search&recrs=a#wrapper'});
await Page.loadEventFired();
await Runtime.evaluate({
expression: `document.querySelector('.paginate_button.next').click()`
});
} catch (err) {
console.error(err);
}
}).on('error', (err) => {
console.error(err);
});
}, 1000); |
|
Thank you very much for your help, only you are helping people) |
|
Wish I'd found this issue earlier - I actually reported this problem as a bug over on the chromium bug tracker: It still seems to fail occasionally but is much more reliable when called from My test code is here: |
|
Got a follow up on this. I'm seeing Again, the methodology I'm using:
I'm attempting to save ALL resources loaded by a particular site. I see this problem consistently when visiting cnn.com, for example. It seems like it might have something to do with proxy-related URLs? Here's an example of one resource (which is loaded when visiting CNN) having this problem: |
|
@pmurley that may also happen (if I recall correctly) when you navigate away from the URL then you call |
|
Wow, you're quick! Yeah, that makes sense, and it's something I should look into a bit more, but I don't think that's what is going on here - at least not in a straightforward way. I'm certainly only calling I guess it could be something caused by some sort of auto-navigation/redirection within a particular frame(?), but if anyone has any other ideas, I'd be grateful! |
|
@pmurley if you can come up with a minimal program that reproduces this issue I could take a look at it. |
|
Here's an example. This usually (but not every single time) prints "Why does this happen?" at least once. const CDP = require('chrome-remote-interface');
setTimeout(() => {
CDP(async (client) => {
const {Network, Page, Runtime} = client;
var req_ids = new Set();
Network.requestWillBeSent(({requestId, request}) => {
req_ids.add(requestId);
});
Network.responseReceived(async ({requestId, response}) => {
});
Network.loadingFinished(async ({requestId, response}) => {
if (req_ids.has(requestId)) {
try {
var response_body = await Network.getResponseBody({requestId});
} catch (err) {
console.log(err);
console.log('Why does this happen?');
}
} else {
// I am also confused as to why we sometimes we get here,
// but this is not my main concern.
console.log('requestId not seen before');
}
});
try {
await Promise.all([Network.enable(), Page.enable()]);
await Page.navigate({url: 'http://cnn.com'});
await Page.loadEventFired();
} catch (err) {
console.error(err);
}
}).on('error', (err) => {
console.error(err);
});
}, 10000); |
|
@pmurley thanks, why the 10s delay though? So I think the problem here is that you're reusing the same tab for multiple page loads, so you end up with unprocessed events coming from the previous instance that reference stale objects. In fact, I consistently get that error if I run the script against a tab that has been used for a previous page load, and never with a blank new tab.
Because they are served from the cache, you can catch them with Here's what I mean: const CDP = require('chrome-remote-interface');
async function test() {
try {
// this is basically the new part //////////////////////////
const target = await CDP.New();
const client = await CDP({target});
////////////////////////////////////////////////////////////
const {Network, Page, Runtime} = client;
const req_ids = new Set();
Network.requestWillBeSent(({requestId}) => {
console.log(`${requestId} Network.requestWillBeSent`);
req_ids.add(requestId);
});
Network.requestServedFromCache(({requestId}) => {
console.log(`${requestId} Network.requestServedFromCache`);
});
Network.responseReceived(async ({requestId}) => {
console.log(`${requestId} Network.responseReceived`);
});
Network.loadingFinished(async ({requestId}) => {
console.log(`${requestId} Network.loadingFinished`);
if (req_ids.has(requestId)) {
try {
var response_body = await Network.getResponseBody({requestId});
} catch (err) {
console.log(`${requestId} Network.getResponseBody: FAILED`);
}
} else {
console.log(`${requestId} UNKNOWN`);
}
});
Network.loadingFailed(async ({requestId}) => {
console.log(`${requestId} Network.loadingFailed`);
});
await Network.enable();
await Page.enable();
await Page.navigate({url: 'http://cnn.com'});
await Page.loadEventFired();
console.log('Page.loadEventFired');
} catch (err) {
console.error(err);
}
}
test();You should handle the errors better and possibly close Bonus: it might be wise to load each page in a new browser context (incognito-like), if that's the case take a look here. |
How to filter url when i use loadingFinished. |
|
@xgj1988 Actually ,I use webContens.debugger of electron ,I couldn't find how to get response body .I found the issue from this page , So i think you know the electron too. Could you tell me how to get the response body in the electron ? |
|
@xgj1988 in the same way AFAIK, the API should be the same. Anyway I don't know electron at all. :) |
|
@cyrus-and Network.getResponseBody how to get requetId? |
|
@xgj1988 as I told you, it's the one that you get in the |
|
@cyrus-and ok I got it . THANKS |
|
@pmurley @xgj1988 @cyrus-and did you figure out what causes problem getting response for some requests? i am also facing this in electron. Could you have a look https://stackoverflow.com/questions/66101799/electron-browserwindow-cannot-get-response-when-debugger-is-attached |
|
@maklimcz could it be about caching as mentioned above? |
|
@cyrus-and nay, I think it isn't loaded from cache because I don't get a
|
|
@maklimcz see if you can reproduce this without Electron. |
|
The conclusion that I came to was that any attempt to "sniff"[^1] network packets using the devtools protocol is not reliable. The My solution has been to use workaround code - e.g if I need data in the response body and fail to get it then either I find the data on the rendered html page, or I send a fetch request from the app code directly (i.e. rather than asking devtools to ask chrome to fetch it ... and then trying to sniff the response). I expect that Hope it helps. [^1] like |
|
@cyrus-and @ilanc it seems i managed to do what I wanted. However I didn't use Also remember to send
https://chromedevtools.github.io/devtools-protocol/tot/Fetch/#event-requestPaused |
@maklimcz can you please elaborate how to run your example. |
|
One thing I found was that it was necessary to ignore 'preflight' requests when calling Network.on("responseReceived",(async (params) => {
if (params.type == 'Preflight') return
const {body} = await Network.getResponseBody(params);
// Do something with the body
})); |
and others
Perhaps you can help me with this, I try to get body from some ajax request on the page, but I all time getting
Error: No data found for resource with given identifierand looks like problem only with this request, maybe I'm missing somethingThanks.
The text was updated successfully, but these errors were encountered: