Cookiejar module #11535
Cookiejar module #11535
Conversation
|
I would like to see this supported, and I think it would be particularly useful for web driver implementations like ghost driver. What will it take to get this pull request merged in? |
|
Has someone tried and tested this patch? |
|
Will try to test it soon. |
|
What's the format of the file is supposed to be? |
|
What format are you talking about? The cookie jar? This should be just making multiple cookie jars and using the already existing save/read from disk code. |
|
There are a few tests in the pull request as well. |
|
I would love to help in the testing of this pull request. What can I do? I couldn't find any documentation on how testing is done. |
|
Step 1: run against all the PhantomJS and GhostDriver tests, all pass. |
|
Just to double check: currently API behaviour is fully preserved - the only difference is the Then, if user want to have dedicated CookieJar for their page, they can create and attach one. This solution is neat, let me tell you :) |
| static CookieJar *singleton = NULL; | ||
| if (!singleton) { | ||
| if (cookiesFile.isEmpty()) { | ||
| qDebug() << "CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persisten cookie storage)"; |
There was a problem hiding this comment.
Could you please re-integrate those qDebug() messages? They do help debugging...
|
A side from my comments (that might require some very small re-touches) I think this is good to go. 👍 |
|
Some note. Related issues:
Those issues will all be solved once we merge this. |
|
PING? @jtrollinson I'd really love to merge this soon. Please review my comments :) |
|
Hi, I commented on a few of your code comments asking a couple questions. |
|
Commented :) |
|
I believe those were all the changes that were suggested. Anything missing? |
|
Any reason why CookieJar needs to be passed in the constructor? There's a corresponding setter after all so if we don't need to make the constructor heavier, that's better. For more details on why this is preferable, take a look at Qt API design guidelines. |
|
This doesn't seem to merge cleanly against the last master. Can you take a look at that? |
|
@ariya the problem with not having the
But this will bring us back to the original coding pattern I had devised: the CookieJar was a singleton and so every WebPage could just "use it internally". Instead we want the flexibility of preserving current behaviour (all pages share the default cookiejar) but also being able to get a new one (via the setter). Makes sense? |
|
The conflicts during the merge seem very straight forward. Should I push a merge? |
|
@jtrollinson If they are simple to solve, no need. But @ariya did raise another good point: would be good if you would |
|
That is no problem. That would require a forced git push correct? That's alright with you? |
|
Another question, how detailed would you like the squashed commit message to be? |
|
The About the message I'd make sure you are explaining the changes (what was there before and what you are introducing), the way it looks in terms of Of course, refer to the ISSUE your commit closes. |
|
@detro Ok, I pushed the squashed commit. Thanks for the help. |
|
I don't think WebPage needs to fetch the default cookie jar by itself. Basically what I think is that instead of this: make it something like this (which is like the original code BTW): If the above can work, then it's the preferred style. See The Convenience Trap section at http://doc.qt.digia.com/qq/qq13-apis.html for details. |
|
So one reason why I decided to do it in the first way rather than the second is that the cookie jar is used during the constructor to build the m_customWebPage. |
|
@jtrollinson what @ariya says it does make sense. After re-reading the constructor-code, I think you could easily change from a 1-line to a 2-line contruct+set. |
|
I suppose I do not completely see the benefit. I understand that one could say that this makes the code easier to read by removing arguments from the constructor. However, in this case, I feel like in this case there is no good default value. Thus, for a period of time the webpage would be in some unstable state. It would be quite easy to forget to set the value of the cookie jar, which would not be good. |
|
Well, essentially what will happen is that the But the idea is to stick to Qt API design principle, so it's up to the user of that API to make sure it's used properly. So, it's actually good to preserve those Qt API principles: makes it easier for other Qt devs out there to join us (if we are so lucky to get MORE contributors). |
|
And once (in the near future) we want to have unit tests for our C++ code, it is really weird and annoying to provide a cookie jar instance every time you want to instantiate a WebPage for the tests. |
|
I have push a commit with that change. Once you have read through it, I'll squash it. One thing I noticed was that on a call to Removal of this line leads to segment faults in the code. |
|
Ok, after closer inspection I see that I am pushing a commit with a change to the comments reflecting that |
|
@jtrollinson Crap! That is a bit of a pickle I think. This was because, when deleted, the Now we are introducing a much more (potentially complex) scenario.
This is all driven by the QtObject parental relationship-memory handling thingy (I'm sure this thing has a name by it's 7:30am and I'm still asleep). Now, we want that:
I marked the new behaviour If we keep the relationship like this ("jars belong to phantom") means that those scripts that run MANY pages, and dispose of them, if they decide to use CookieJars (for session isolation), all those JARs will never be deleted, until Phantom is closed. => MEMORY LEAK At this time of the day, I see 2 options for us:
The second option though is bad for a couple of reasons
Sorry if this got too long. |
|
My idea was to use cookieJar.close() to delete cookie jars, because the user is the only one who knows if the cookie jar is going to be used again in the future. I think that Phantom in this situation would still be the best parent. |
|
From what I understand. The way it is set up now works well. If the If the
This is also similar to how web pages are handled. |
|
I agree. This seems to be safest scenario. Ownership of Jars remains to @ariya ? |
|
The lack of reference counting information through QtWebKit JS binding is the problem here. I'm fine with the solution of explicit |
|
Agreed. |
|
To ensure the cross-ref, we prefer to have the issue link in every commit. I can definitely add them as I land your patches, unless you want to do it yourself. |
|
I was planning on squashing the three commits after given the go-ahead. Would you prefer them to be separated? |
|
Squash away :) |
|
Go for it! |
|
All merged! |
|
Looks good! @detro Any further comment before I land this? |
|
@ariya I think we have now the best compromise for usability and stability. The worst case scenario: the user keeps creating Jar's without closing them (memory consumption goes up), but stability is preserved. I'm eager to add support for this in GhostDriver and see if it can stand parallel testing. |
|
Can you rebase it against the current master? Your parent commit was from a bit outdated and the merge causes some conflict (and I'd rather not make a mistake if I solve it myself). |
Previously, there was a single global cookie jar shared between all web pages.
Now, one can have separate cookie jars for different web pages.
Makes CookieJar a normal class, not a singleton.
Moves many public CookieJar methods to public slots.
Adds default cookie jar to Phantom.
Adds the CookieJar module that provides access to cookie jars in javascript.
Adds cookie jar module tests.
Usage:
var jar = require('cookiejar').create();
var webpage = require('webpage').create();
webpage.cookieJar = jar;
...
webpage.close();
jar.close();
JS API changes:
Webpage:
var jar = page.cookieJar; -- assigns 'jar' the given webpage's cookie jar.
page.cookiejar = jar; -- sets 'jar' as the given webpage's cookie jar.
CookieJar:
var jar = require('cookiejar').create(path)
creates a cookie jar with persistent storage at the given file path
(path not mandatory).
var cookies = jar.cookies; -- assign's 'jar' the list of cookies in the
cookie jar.
jar.cookies = [c1, c2]; -- sets the cookie jar's cookies as the ones in the
list.
jar.addCookie(cookie) -- adds cookie 'cookie' to the cookie jar.
fixes issue ariya#11417
|
Ok, it's now rebased to master. |
|
Landed. Thank you very much @jtrollinson! |
Finally an API to support multi-cookie jar is available in PhantomJS: ariya/phantomjs#11535. This allows us to create a completely new CookieJar every time we create a Session. A version of PhantomJS with the new CookieJar API hasn't been released yet: once the commit linked above is part of a stable release, this will work. Otherwise, stick with GhostDriver 1.1.0. Fixes #170.
|
Hi, thank you for providing us with PhantomJS! Awesomely easy to use and very quick. Where is this required cookiejar module located? Whenever I want my crawljax to use the PhantomJS driver, it keeps failing because it cannot find this module: Same for my PhantomJS 1.9.6 installation: Thanks for your answer, |
|
@rpoisel This is not available in 1.9. PLEASE use the mailing-list as a forum support and not this issue tracker. |
|
Thank you for your quick answer! It seems that there is something broken in the 1.9.6 release because in session.js of the included GhostDriver the cookiejar module is already being used. If you want I can also post this message to a suitable mailing list of your project. |
This is an implementation of the cookie jar implementation described in issue #11417
The instance static method of CookieJar was removed and the constructor was made public.
A cookiejar.js module was added that adds a API for creating a cookiejar.
To use:
I would love any suggestions or comments you might have.
Hope this helps.