Skip to content
This repository has been archived by the owner. It is now read-only.

Adding support for getting body of requests or responses #10158

Closed
ghost opened this issue Jul 4, 2011 · 103 comments

Comments

Projects
None yet
@ghost
Copy link

commented Jul 4, 2011

saleh...@gmail.com commented:

Adding support for accessing body of requests (useful for http post requests) in the onResourceRequested call back.

Which version of PhantomJS are you using? 1.2

Disclaimer:
This issue was migrated on 2013-03-15 from the project's former issue tracker on Google Code, Issue #158.
🌟   31 people had starred this issue at the time of migration.

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 4, 2011

ariya.hi...@gmail.com commented:

I'm not sure how we would encode the content. Typed array comes to mind, but that's not widely supported yet.

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 16, 2011

ariya.hi...@gmail.com commented:

 

 
Metadata Updates

  • Title updated: Adding support for getting body of requests or responses
@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 16, 2011

jgorn...@gmail.com commented:

It seems the body would be part of the "text" property in the content object of the HAR response object: http://www.softwareishard.com/blog/har-12-spec/#content

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 21, 2011

jgorn...@gmail.com commented:

After doing some more research, wouldn't it be feasible to add the content of the response to a text property in the data emitted when listening to the NetworkAccessManager::handleFinished and performing a readAll() on the reply argument? Then taking that byte array and converting to a QTString?

Currently, the signal is coming from: http://doc.qt.nokia.com/4.6/qnetworkaccessmanager.html#finished
Then the reply object is: http://doc.qt.nokia.com/4.6/qnetworkreply.html#finished
Where we could call readAll to : http://doc.qt.nokia.com/4.6/qiodevice.html#readAll

Just wanted to check if I am on the right path here.

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 23, 2011

ariya.hi...@gmail.com commented:

I'm not sure readAll is wise there. Doing that definitely leaves WebKit with data at all?

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 23, 2011

jgorn...@gmail.com commented:

Are you saying that if you call readAll on the response object, it won't leave that data available for WebKit to read?

@ariya

This comment has been minimized.

Copy link
Owner

commented Jul 25, 2011

ariya.hi...@gmail.com commented:

Yes, I am quite sure calling readAll will eat all the received data and leave nothing.

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 5, 2011

jgorn...@gmail.com commented:

Off the top of your head, would there then be a way to read a copy of response data? If you can point me in the right direction, I can start experimenting with it.

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 8, 2011

jgorn...@gmail.com commented:

I did a little more research and found out that it might be possible to write a proxy for the QNAM and the NetworkReply. This way you could capture the data received and still have it accessible after the read has finished. An example I found pointed me to: http://gitorious.org/qtwebkit/performance/blobs/master/host-tools/mirror/main.cpp

Am I on the right track here?

Thanks!

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 8, 2011

ariya.hi...@gmail.com commented:

Yes, proxying both the network access manager and network reply is the right approach.

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 8, 2011

jgorn...@gmail.com commented:

I'm on my way to figuring this out but have run into a snag. Keep in mind I'm not that familiar with C++ but after adding in the reply proxy, I get a compile error of:

Undefined symbols:
"vtable for NetworkReplyProxy", referenced from:
NetworkReplyProxy::NetworkReplyProxy(QObject_, QNetworkReply_)in networkaccessmanager.o

Here is my latest gist of the NetworkAccessManager:
https://gist.github.com/1131280

Any help would be greatly appreciated!

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 14, 2011

jgorn...@gmail.com commented:

Pull request: #124

We'll definitely need to go through a code review on this one :)

@ariya

This comment has been minimized.

Copy link
Owner

commented Aug 21, 2011

ariya.hi...@gmail.com commented:

See 578aa6c.

We need an option to enable and disable (by default) this data gathering (since it introduces some performance penalty).

 
Metadata Updates

  • Milestone updated: Release1.3 (was: ---)
  • Status updated: Accepted
@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 15, 2011

ariya.hi...@gmail.com commented:

Hmm, some ownership issue and/or race condition in the proxy might provoke a crash.

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 15, 2011

jgorn...@gmail.com commented:

Ariya, I'm sorry, but I'm not following. Can you elaborate a little more?

Also, when I get some time, I can add in a patch to disable the use of the proxy.

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 16, 2011

ariya.hi...@gmail.com commented:

I fixed the potential crash in 37404ba.

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 16, 2011

jgorn...@gmail.com commented:

Awesome!

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 16, 2011

ariya.hi...@gmail.com commented:

Closing this one as it is implemented already.

For the API/settings improvement, head to issue 236.

 
Metadata Updates

  • Status updated: Fixed
@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 19, 2011

ariya.hi...@gmail.com commented:

Reopened. Likely needs to be rescheduled soon.

It causes regression, see issue 238.

 
Metadata Updates

  • Status updated: Accepted
@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 19, 2011

ariya.hi...@gmail.com commented:

 

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 19, 2011

ariya.hi...@gmail.com commented:

I can't find a quick way to resolve this. Probably I will revert it for the time being.

@ariya

This comment has been minimized.

Copy link
Owner

commented Sep 21, 2011

ariya.hi...@gmail.com commented:

Unfortunately I have to revert this, see eb25581

We need to have a better working solution for 1.4.

 
Metadata Updates

  • Milestone updated: Release1.4 (was: Release1.3)
@ariya

This comment has been minimized.

Copy link
Owner

commented Oct 4, 2011

jgorn...@gmail.com commented:

Ariya, do you have any other information on what needs to be added to the proxy so we can get this back in? I'd like to be able to help and get this resolved.

@ariya

This comment has been minimized.

Copy link
Owner

commented Oct 5, 2011

ariya.hi...@gmail.com commented:

I have not investigated further. Basically the fix should not regress netlog and netsniff examples (easy to test). See also issue 238.

@ariya

This comment has been minimized.

Copy link
Owner

commented Oct 5, 2011

jgorn...@gmail.com commented:

Sounds good. I will try to look into this and see if I can find a solution. I will also look into adding the flag to disable the proxy for performance reasons.

@ariya

This comment has been minimized.

Copy link
Owner

commented Dec 19, 2011

ariya.hi...@gmail.com commented:

Not enough time to resolve for 1.4. Postpone to 1.5.

 
Metadata Updates

  • Milestone updated: Release1.5 (was: Release1.4)
@ariya

This comment has been minimized.

Copy link
Owner

commented Mar 9, 2012

ariya.hi...@gmail.com commented:

No activity, reschedule.

 
Metadata Updates

  • Milestone updated: FutureRelease (was: Release1.5)
@ariya

This comment has been minimized.

Copy link
Owner

commented Mar 13, 2012

ariya.hi...@gmail.com commented:

Issue 422 has been merged into this issue.

@ariya

This comment has been minimized.

Copy link
Owner

commented Apr 14, 2012

ariya.hi...@gmail.com commented:

Issue 501 has been merged into this issue.

@ariya

This comment has been minimized.

Copy link
Owner

commented May 2, 2012

webmas...@webmaster.ms commented:

I think, the embedded phantomjs server can act as a proxy which intersects and saves the content of the dowloading files and the sending POST.
The only problem is its unability to parse queries like "GET http://www.google.com/ HTTP/1.1".
The server responses with "Error 400: Bad Request, Cannot parse HTTP request: [GET]".
It seems that fixing the server would be an easier and faster solution for the people wanting to access the bodies of requests and responses.

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

response body is only available in onResourceReceived event,
if url matches one of the patterns in 'page.captureContent' property

ariya#10158 (reverted from commit 327f91d)

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Fixed body string initialization code (0 byte for empty strings)
Also fixed same code in filesystem module

ariya#10158 (reverted from commit d0fde08)

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

Vitallium added a commit to Vitallium/phantomjs that referenced this issue Jan 19, 2016

@arrowing

This comment has been minimized.

Copy link

commented Apr 5, 2016

How long I can use reponse body in released version?

I will use proxy for it. :(

@Noiwex

This comment has been minimized.

Copy link

commented Apr 5, 2016

#10158 (comment)

or make changes in cpp and compile

@Noiwex

This comment has been minimized.

Copy link

commented Apr 23, 2016

@MichalBu

This comment has been minimized.

Copy link

commented Aug 3, 2016

This fix was removed from 2.1.1 - why?
How can I get the build with this feature?

@Vitallium

This comment has been minimized.

Copy link
Collaborator

commented Aug 3, 2016

@MichalBu You can find more information here #13922

@jbruggeman

This comment has been minimized.

Copy link

commented Aug 3, 2016

I was able to hack together some code together to re-add this functionality. Basically I edited QNetworkReply to overload read/readAll, intercept the data, and dump it to a QByteArray. I then exposed and Base64 encoded it.

It's not nice (one might even call it terrible), but it works.
https://github.com/jbruggeman/phantomjs
https://github.com/jbruggeman/qtbase

@jksecurity

This comment has been minimized.

Copy link

commented Aug 30, 2016

@jbruggeman As I really would like this feature, could you explain how to set this up using the sources you provided? I am not sure how I would get the build to pick up the recompiled qtbase for example (preferably without overwriting the system wide qtbase).

@jbruggeman

This comment has been minimized.

Copy link

commented Aug 30, 2016

Hi,

Just clone the repos and follow the normal build steps for phantom.

On Tuesday, 30 August 2016, jksecurity notifications@github.com wrote:

@jbruggeman https://github.com/jbruggeman As I really need this feature
urgently, could you explain how to set this up using the sources you
provided? I am not sure how I would get the build to pick up the recompiled
qtbase for example.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10158 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABFMBizZV5fa1ANGsV6NCE6622IVTvp4ks5qk9fdgaJpZM4AgfdU
.

@Frogmouth

This comment has been minimized.

Copy link

commented Sep 15, 2016

@jbruggeman ... Can you explain the process however? I'm not skilled in "build process" but I need the "body response" :( I have already build the Ariya's version with the method that i find i this page: http://phantomjs.org/build.html... but I don't think it's correct also for yours release.

@lifuchao

This comment has been minimized.

Copy link

commented Feb 15, 2017

@jbruggeman would you please supply a compiled phantomjs with onResourceReceived response.body?thanks a lot.my e-mail is :12682082@qq.com

@lifuchao

This comment has been minimized.

Copy link

commented Feb 15, 2017

@MichalBu can you use onResourceReceived response.body now ? I encountered this issue,can you help me ?

@artemeus

This comment has been minimized.

Copy link

commented Feb 27, 2017

@ariya @Paxa Could anyone please help me understand why we have been unable to add this feature to master in 5+ years? Really need it (and less mature frameworks like SlimerJs already have it), yet can't make myself commit to hacking together a build..

@Paxa

This comment has been minimized.

Copy link
Contributor

commented Feb 27, 2017

I don't know, this situation feels weird to me. I already switched to slimerjs, it's also faster and not require few hours to compile

@Vitallium

This comment has been minimized.

Copy link
Collaborator

commented Feb 27, 2017

The problem is that this feature broke AJAX requests. That's why we reverted it.

@artemeus

This comment has been minimized.

Copy link

commented Feb 28, 2017

@Paxa @Vitallium Thank you for quick replies gentlemen!
I guess it is a dead end then.
I will try replacing PhantomJS executable with SlimerJs in Java with Selenium WebDriver code and see whether I can execute all the same unit tests. Hoping I don't have to jump extra hoops to get headless experience on Linux with it.

@developez

This comment has been minimized.

Copy link

commented May 26, 2017

Hello. I also need the body with the content to download some pdf files from a website. I am sorry, I think this is an obvious comment but here on file 'networkaccessmanager.cpp' on line 454 (https://github.com/ariya/phantomjs/blob/master/src/networkaccessmanager.cpp#L454) the body property is assigned to emtpy string so how can we expect that be filled? Why can't we take the content from QNetworkReply or that object hasn't got the content?

@tasoss

This comment has been minimized.

Copy link

commented Jun 9, 2017

Can somebody pretty please update on this issue?

@dmitriyse

This comment has been minimized.

Copy link

commented Jun 22, 2017

I also require this "must have" feature.

@tasoss

This comment has been minimized.

Copy link

commented Jun 22, 2017

I don't understand why don't you answer.Because you believe that the answer is already mentioned above?
Which means that it can't be implemented.It breaks ajax requests so you can't fix it right?Can you please lock the thread please?

@amanda94

This comment has been minimized.

Copy link

commented Nov 16, 2017

I find that I can get postData in requestData for POST requests by listening the event onResourceRequested.But for PUT or PATCH methods, postData do not exist , is there anyway I can get the data for PUT or PATCH?or I should add some extra settings ? Also when I find the method is a "?" when I try to send PATCH or DELETE Ajax requests
something added , I am using the 2.1 version

@artemeus

This comment has been minimized.

Copy link

commented Nov 16, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.