bad proxy exits node without any error #739
Labels
Comments
|
I've faced the same issue. Didn't use custom timeout or attempted to read the response, just wanted to know if the request through proxy was successful (no timeout etc.) |
|
I also face sometimes the node.js process getting stuck after firing the request with proxy/agent. I attempt to read the status of the response but the execution never gets there. |
|
same for me |
|
same. It will exits node without any error if proxy is unavailable |
|
I think the error might be due to |
ghost
mentioned this issue
|
I am encountering a similar (the same?) issue with just I believe in my case it is triggered because the connection is very unstable and has high packet loss (5% or more). Could this also be the case for the proxies you are seeing problems with? (I'm not saying this isn't a bug in either node or node-fetch; just trying to narrow down possible causes) |
|
Can you try running the script with |
|
I'll give it a try. Might take a while before it next crashes. The output from |
|
Spoke too soon; it already crashed! I'll have the logs here soon. |
That's where the logs end. Process exits with exit code 0. I know for sure it isn't a clean exit because there are a lot more logs it would print if that were the case (a lot more API calls too). Last stdout line (any stderr output is in the logs above): The calling code (sorry it's a bit messy, I've been adding lots of special-case error handling to diagnose what's going on here): function getText(res) {
return res.text().catch(() => '<failed to get response text>');
}
export async function callAPI(api, params) {
const url = `${API_BASE}/${api}.json?${new URLSearchParams({ api_key: API_KEY, ...params })}`;
while (true) {
await apiThrottle.check();
console.info(`calling ${url}`); // <-- this is the last line we see printed
// [this is simplified from the real code to remove paths which I know are not taken based on the log output]
const res = await fetch(url); // <-- node-fetch call
// I added a console log line here in a later experiment and it DOES get printed before dying
if (res.ok) {
try {
const json = await res.json(); // <-- problem seems to be here
// I added a console log line here in a later experiment and it DOES NOT get printed
return json; // <-- all paths from here print messages which we do not see in the logs
} catch (e) {
console.log('failed to parse JSON', e); // NOT printed
throw e;
}
}
const text = await getText(res);
// [...cut since not relevant...] (error handling; all paths print or throw, none observed in output)
}
}Finally, to give an idea of the quality of the network connection (which I think is relevant here since it seems to exit after hanging on a request for a while): It seems to be particularly bad at the moment! Which might explain why the error is reproducing so readily. |
|
@davidje13 Thank you for all the details! This is very helpful. I believe the problem you are hitting is a content-length or chunked transfer-encoding bug. I actually have this on my hit list for tomorrow, so hopefully I'll have some kind of resolution by then. In the meantime, you may have some luck if you switch to Node.js 12 from 14/15 or from 12 to 14. I've noticed slightly different behaviors in tests (e.g #1056). If you do see a change in switching (positive or negative), please post the results. |
|
Glad you've got a suspected cause! A few final points for now:
|
|
@davidje13 I have created a branch with a hopeful fix. Please give it a try when you can: tekwiz:node-fetch/fix-chunked-transfer-premature-close-1. You should be able to install it with npm install 'github:tekwiz/node-fetch#fix-chunked-transfer-premature-close-1' |
|
Yes, that branch works better; instead of exiting silently it now throws (still using Node15): Which seems reasonable. I can catch this error and retry the request. Out of curiosity: did you find out what was causing the node process to die before? I don't see any smoking gun in your diffs (I suspect there is also a bug to report in NodeJS itself here). |
4 tasks
4 tasks
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've been using node-fetch with different proxies and
https-proxy-agentfor years. Never had a problem. I switched to another proxy service and every 50 requests or so, the node.js process exits without warning. I know it's the proxy because if I switch to a different proxy it works perfectly.How can I debug this issue if it just exits without warning and I don't have access to the logs of the proxy service?
This is the code I'm using for testing:
It always exits without warning between 30-50 iterations with that proxy
The text was updated successfully, but these errors were encountered: