chunks of html missing from request body #266
Comments
|
can you dump the headers from curl and the headers from request so that we can compare? Mikeal Rogers On Thursday, June 7, 2012 at 12:17 PM, jmonster wrote:
|
|
Upon trying to reproduce to pluck out the headers, curl is now failing exactly the same. There must be something I did in my browser (with my session/cookies) that triggered the page to work properly... sorry for the distraction; I'll re-open the issue if I can better isolate the problem |
|
This is so weird. I think maybe request is shitting the bed with a cookie? Can I explicitly disallow chunked encoding? Once I click the 'continue' button via Google Chrome and try the cURL request again it works, but the node request still fails exactly as it did before. What's interesting is that what I'm calling a failure is this partial HTML response ... well, before I clicked 'Continue' in chrome even google chrome got that incomplete html blob. Now that I've pressed continue, cURL gets the full page, Chrome gets the full page, but node-request still only gets the partial page. request headers: {
"date": "Thu, 07 Jun 2012 21:18:51 GMT",
"server": "Apache/2.0.64 (Unix)",
"cache-control": "no-cache",
"pragma": "no-cache",
"p3p": "CP=\"PHY ONL CAO CURa ADMa DEVa TAIa PSAa PSDa IVAo IVDo CONo HISa TELo OTPo OUR DELa STP BUS UNI COM NAV INT DEM OTC\",policyref=\"/w3c/p3p.xml\"",
"x-powered-by": "Servlet/2.5 JSP/2.1",
"vary": "Accept-Encoding",
"x-ua-compatible": "IE=EmulateIE7",
"connection": "close",
"transfer-encoding": "chunked",
"content-type": "text/html; charset=ISO-8859-1"
}curl headers: HTTP/1.1 200 OK
Date: Thu, 07 Jun 2012 21:19:49 GMT
Server: Apache/2.0.64 (Unix)
Cache-Control: no-cache
Pragma: no-cache
P3P: CP="PHY ONL CAO CURa ADMa DEVa TAIa PSAa PSDa IVAo IVDo CONo HISa TELo OTPo OUR DELa STP BUS UNI COM NAV INT DEM OTC",policyref="/w3c/p3p.xml"
X-Powered-By: Servlet/2.5 JSP/2.1
Vary: Accept-Encoding
X-UA-Compatible: IE=EmulateIE7
Connection: close
Content-Type: text/html; charset=ISO-8859-1 |
|
Any thoughts on the exclamation point in the cookie string? for the curl request I have to escape it, !, node-request yields the same results regardless. If I omit it then dollargeneral fails to match my session at all. |
|
Apparently this isn't a node-request issue because if I go a level lower I get the same thing: var request = require('request')
var cookie = "browser_id=155739558874; ClrOSSID=1337963428563-8545; ClrSCD=1337963428563; emailPopUp_cookie=emailPopUp_cookie; s_vi=[CS]v1|27DFD9D285013860-4000010B20135E2B[CE]; __g_u=207498766274588_3_1_1_5_1339343191532_0; _csuid=X282b88a3b23560; JSESSIONID=8C7nPRgMjzSMXLpVyvkGHgDlFKh4QTrPsrGgphD4z91grvtFj3Jzp!1171214651; __g_c=a%3A1%7Cb%3A50%7Cc%3A207498766274588%7Cd%3A3%7Ce%3A1%7Cf%3A1%7Ch%3A0%7Cr%3A%7Cg%3A1; ClrSSID=1337963428563-8545; WT_FPC=id=74.98.195.63-2987874064.30227091:lv=1339102004792:ss=1339101820391; s_pers=%20s_nr%3D1337963520412%7C1340555520412%3B%20s_lastvisit%3D1339105420428%7C1433713420428%3B%20gpv_p6%3DPayPal%2520Express%2520GSI%2520Transition%2520Page%7C1339107404832%3B; mt.v=2.674285215.1337963428813; s_sess=%20s_cp_persist12%3DCMS_HP_B1_Circular_Corp_PriceSavings_Circ_060312%3B%20s_cc%3Dtrue%3B%20s_sq%3D%3B%20s_ppv%3D-%252C79%252C79%252C647%3B; PrefID=42-1514422448"
var options = {
host: 'www.dollargeneral.com',
port: 443,
path: '/checkout/index.jsp?process=address',
method: 'GET',
headers: {cookie:cookie}
};
var http = require('https')
var req = http.request(options, function(res) {
console.log('STATUS: ' + res.statusCode);
console.log('HEADERS: ' + JSON.stringify(res.headers));
res.setEncoding('utf8');
res.on('data', function (chunk) {
console.log('BODY: ' + chunk);
});
});
req.on('error', function(e) {
console.log('problem with request: ' + e.message);
});
req.end();I'm leaving the issue open for the moment incase you have any advice but otherwise feel free to close it :( |
|
i can also debug it if it's a core problem. You gave me the response headers, what I need is the request headers. In curl they should show up in debug mode, for request. just do var r = request(url, function () { console.log(r.headers) }) |
|
my bad, always wondered how to get the request headers! var R = request({
uri:'https://www.dollargeneral.com/checkout/index.jsp?process=address',
headers:{cookie:cookie},
followRedirect:false
}, function(e,r,b) {
console.log(R.headers)
})yields { cookie: 'JSESSIONID=DJQ9PRHDJYGYjsrxL4SBnCJHyM21JQjQBDk1h1gvJ5DklMXJY4ms!-1374481206;',
'content-length': 0 }Doesn't look terribly useful? Am I doing something wrong? CURL: |
|
and whattaya know, it's the user agent. you are awesome. thank you, may not have thought to check it without the debugging prompts! |
|
great to hear :) |
I've spent about 8 hours trying to figure out what I might be doing wrong, and finally I've isolated it. I've saved my cookies as a text file and tried an identical request with cURL and it works. I open a node console and make the request with the same set of cookies and it's missing a huge chunk of data.
the result is missing most of the page, it ends abruptly with:
<div class="wrap">curl, on the other hand, returns what i expect (and matches a browser):
You'll need to establish your own cookies for this to work -- what I did was start an order, pay with paypal, go back to the site ... you dont have to actually complete/authorize any purchases.
this is the ONLY page I've had this problem with
The text was updated successfully, but these errors were encountered: