Adding support for getting body of requests or responses

[ghost]

saleh...@gmail.com commented:

Adding support for accessing body of requests (useful for http post requests) in the onResourceRequested call back.

Which version of PhantomJS are you using? 1.2

Disclaimer:
This issue was migrated on 2013-03-15 from the project's former issue tracker on Google Code, Issue #158.
🌟   31 people had starred this issue at the time of migration.

[ariya]

ariya.hi...@gmail.com commented:

I'm not sure how we would encode the content. Typed array comes to mind, but that's not widely supported yet.

[ariya]

ariya.hi...@gmail.com commented:

 

 
Metadata Updates

• Title updated: Adding support for getting body of requests or responses

[ariya]

jgorn...@gmail.com commented:

It seems the body would be part of the "text" property in the content object of the HAR response object: http://www.softwareishard.com/blog/har-12-spec/#content

[ariya]

jgorn...@gmail.com commented:

After doing some more research, wouldn't it be feasible to add the content of the response to a text property in the data emitted when listening to the NetworkAccessManager::handleFinished and performing a readAll() on the reply argument? Then taking that byte array and converting to a QTString?

Currently, the signal is coming from: http://doc.qt.nokia.com/4.6/qnetworkaccessmanager.html#finished
Then the reply object is: http://doc.qt.nokia.com/4.6/qnetworkreply.html#finished
Where we could call readAll to : http://doc.qt.nokia.com/4.6/qiodevice.html#readAll

Just wanted to check if I am on the right path here.

[ariya]

ariya.hi...@gmail.com commented:

I'm not sure readAll is wise there. Doing that definitely leaves WebKit with data at all?

[ariya]

jgorn...@gmail.com commented:

Are you saying that if you call readAll on the response object, it won't leave that data available for WebKit to read?

[ariya]

ariya.hi...@gmail.com commented:

Yes, I am quite sure calling readAll will eat all the received data and leave nothing.

[ariya]

jgorn...@gmail.com commented:

Off the top of your head, would there then be a way to read a copy of response data? If you can point me in the right direction, I can start experimenting with it.

[ariya]

jgorn...@gmail.com commented:

I did a little more research and found out that it might be possible to write a proxy for the QNAM and the NetworkReply. This way you could capture the data received and still have it accessible after the read has finished. An example I found pointed me to: http://gitorious.org/qtwebkit/performance/blobs/master/host-tools/mirror/main.cpp

Am I on the right track here?

Thanks!

[ariya]

ariya.hi...@gmail.com commented:

Yes, proxying both the network access manager and network reply is the right approach.

[ariya]

jgorn...@gmail.com commented:

I'm on my way to figuring this out but have run into a snag. Keep in mind I'm not that familiar with C++ but after adding in the reply proxy, I get a compile error of:

Undefined symbols:
"vtable for NetworkReplyProxy", referenced from:
NetworkReplyProxy::NetworkReplyProxy(QObject_, QNetworkReply_)in networkaccessmanager.o

Here is my latest gist of the NetworkAccessManager:
https://gist.github.com/1131280

Any help would be greatly appreciated!

[ariya]

jgorn...@gmail.com commented:

Pull request: #124

We'll definitely need to go through a code review on this one :)

[ariya]

ariya.hi...@gmail.com commented:

See 578aa6c.

We need an option to enable and disable (by default) this data gathering (since it introduces some performance penalty).

 
Metadata Updates

• Milestone updated: Release1.3 (was: ---)
• Status updated: Accepted

[ariya]

ariya.hi...@gmail.com commented:

Hmm, some ownership issue and/or race condition in the proxy might provoke a crash.

[ariya]

jgorn...@gmail.com commented:

Ariya, I'm sorry, but I'm not following. Can you elaborate a little more?

Also, when I get some time, I can add in a patch to disable the use of the proxy.

[ariya]

ariya.hi...@gmail.com commented:

I fixed the potential crash in 37404ba.

[ariya]

jgorn...@gmail.com commented:

Awesome!

[ariya]

ariya.hi...@gmail.com commented:

Closing this one as it is implemented already.

For the API/settings improvement, head to issue 236.

 
Metadata Updates

• Status updated: Fixed

[ariya]

ariya.hi...@gmail.com commented:

Reopened. Likely needs to be rescheduled soon.

It causes regression, see issue 238.

 
Metadata Updates

• Status updated: Accepted

[ariya]

ariya.hi...@gmail.com commented:

 

[ariya]

ariya.hi...@gmail.com commented:

I can't find a quick way to resolve this. Probably I will revert it for the time being.

[ariya]

ariya.hi...@gmail.com commented:

Unfortunately I have to revert this, see eb25581

We need to have a better working solution for 1.4.

 
Metadata Updates

• Milestone updated: Release1.4 (was: Release1.3)

[ariya]

jgorn...@gmail.com commented:

Ariya, do you have any other information on what needs to be added to the proxy so we can get this back in? I'd like to be able to help and get this resolved.

[ariya]

ariya.hi...@gmail.com commented:

I have not investigated further. Basically the fix should not regress netlog and netsniff examples (easy to test). See also issue 238.

[ariya]

jgorn...@gmail.com commented:

Sounds good. I will try to look into this and see if I can find a solution. I will also look into adding the flag to disable the proxy for performance reasons.

[ariya]

ariya.hi...@gmail.com commented:

Not enough time to resolve for 1.4. Postpone to 1.5.

 
Metadata Updates

• Milestone updated: Release1.5 (was: Release1.4)

[ariya]

ariya.hi...@gmail.com commented:

No activity, reschedule.

 
Metadata Updates

• Milestone updated: FutureRelease (was: Release1.5)

[ariya]

ariya.hi...@gmail.com commented:

Issue 422 has been merged into this issue.

[ariya]

ariya.hi...@gmail.com commented:

Issue 501 has been merged into this issue.

[ariya]

webmas...@webmaster.ms commented:

I think, the embedded phantomjs server can act as a proxy which intersects and saves the content of the dowloading files and the sending POST.
The only problem is its unability to parse queries like "GET http://www.google.com/ HTTP/1.1".
The server responses with "Error 400: Bad Request, Cannot parse HTTP request: [GET]".
It seems that fixing the server would be an easier and faster solution for the people wanting to access the bodies of requests and responses.

[rockswang]

Are there any documentation about this feature?

[vanillycpp]

I didn`n found documentation
just this http://stackoverflow.com/questions/22058321/status-of-phantomjs-onresourcereceived-response-body-work

[arrowing]

How long I can use reponse body in released version?

I will use proxy for it. :(

[MichalBu]

This fix was removed from 2.1.1 - why?
How can I get the build with this feature?

[vitallium]

@MichalBu You can find more information here #13922

[jbruggeman]

I was able to hack together some code together to re-add this functionality. Basically I edited QNetworkReply to overload read/readAll, intercept the data, and dump it to a QByteArray. I then exposed and Base64 encoded it.

It's not nice (one might even call it terrible), but it works.
https://github.com/jbruggeman/phantomjs
https://github.com/jbruggeman/qtbase

[jeromekleinen]

@jbruggeman As I really would like this feature, could you explain how to set this up using the sources you provided? I am not sure how I would get the build to pick up the recompiled qtbase for example (preferably without overwriting the system wide qtbase).

[jbruggeman]

Hi,

Just clone the repos and follow the normal build steps for phantom.

On Tuesday, 30 August 2016, jksecurity notifications@github.com wrote:

@jbruggeman https://github.com/jbruggeman As I really need this feature
urgently, could you explain how to set this up using the sources you
provided? I am not sure how I would get the build to pick up the recompiled
qtbase for example.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10158 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFMBizZV5fa1ANGsV6NCE6622IVTvp4ks5qk9fdgaJpZM4AgfdU
.

[Frogmouth]

@jbruggeman ... Can you explain the process however? I'm not skilled in "build process" but I need the "body response" :( I have already build the Ariya's version with the method that i find i this page: http://phantomjs.org/build.html... but I don't think it's correct also for yours release.

[lifuchao]

@jbruggeman would you please supply a compiled phantomjs with onResourceReceived response.body?thanks a lot.my e-mail is :12682082@qq.com

[lifuchao]

@MichalBu can you use onResourceReceived response.body now ? I encountered this issue,can you help me ?

[artemeus]

@ariya @Paxa Could anyone please help me understand why we have been unable to add this feature to master in 5+ years? Really need it (and less mature frameworks like SlimerJs already have it), yet can't make myself commit to hacking together a build..

[Paxa]

I don't know, this situation feels weird to me. I already switched to slimerjs, it's also faster and not require few hours to compile

[vitallium]

The problem is that this feature broke AJAX requests. That's why we reverted it.

[artemeus]

@Paxa @vitallium Thank you for quick replies gentlemen!
I guess it is a dead end then.
I will try replacing PhantomJS executable with SlimerJs in Java with Selenium WebDriver code and see whether I can execute all the same unit tests. Hoping I don't have to jump extra hoops to get headless experience on Linux with it.

[developez]

Hello. I also need the body with the content to download some pdf files from a website. I am sorry, I think this is an obvious comment but here on file 'networkaccessmanager.cpp' on line 454 (https://github.com/ariya/phantomjs/blob/master/src/networkaccessmanager.cpp#L454) the body property is assigned to emtpy string so how can we expect that be filled? Why can't we take the content from QNetworkReply or that object hasn't got the content?

[tasoss]

Can somebody pretty please update on this issue?

[dmitriyse]

I also require this "must have" feature.

[tasoss]

I don't understand why don't you answer.Because you believe that the answer is already mentioned above?
Which means that it can't be implemented.It breaks ajax requests so you can't fix it right?Can you please lock the thread please?

[amanda94]

I find that I can get postData in requestData for POST requests by listening the event onResourceRequested.But for PUT or PATCH methods, postData do not exist , is there anyway I can get the data for PUT or PATCH？or I should add some extra settings ? Also when I find the method is a "?" when I try to send PATCH or DELETE Ajax requests
something added , I am using the 2.1 version

[artemeus]

Set up ChromeDriver with headless Chrome. PhantomJS is dead.

…

On Wed, Nov 15, 2017 at 11:28 PM, amanda ***@***.***> wrote: I find that I can get postData in requestData for POST requests by listening the event onResourceRequested.But for PUT or PATCH methods, postData do not exist , is there anyway I can get the data for PUT or PATCH？or I should add some extra settings ? Also when I find the method is a "?" when I try to send PATCH or DELETE Ajax requests — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10158 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABf9Ve35fSUiQvj50_Qhj_ymgexAj2DGks5s2-QogaJpZM4AgfdU> .