MajesticSEO now requires login

[ChrisAlvares]

MajesticSEO now requires a login to access its site explorer.

This breaks all Majestic SEO uses.

[ChrisAlvares]

It is possible to login with MajesticSEO using a cookie file and CURL, here is an example (however, this example should not be implemented because it does not use the SEOStats CURL method, nor does it use the seoconfig file for creds.

    $cookiefile = tempnam ("seostats/cookies/", "CURLCOOKIE");
    $params = 'redirect=%2Freports%2Fsite-explorer%2Fsummary%2Fchrisalvares.com';
    $params .= '&EmailAddress=email%40address.com';
    $params .= '&Password=password';
    $params .= '&RememberMe=1';


    /* STEP 2. visit the login page to set the cookie properly */

    $login = curl_init ("https://www.majesticseo.com/account/login");   
    curl_setopt ($login, CURLOPT_COOKIEJAR, $cookiefile); 
    curl_setopt ($login, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($login,CURLOPT_POST,4);
    curl_setopt($login,CURLOPT_POSTFIELDS,$params);
    curl_setopt($login, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($login, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1");
    curl_exec ($login);
    curl_close($login);


    $ch = curl_init('https://www.majesticseo.com/reports/site-explorer/summary/'. str_replace('http://', '', $uri) );
    curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookiefile); 
    curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookiefile); 
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1");

    $tmp = curl_exec ($ch);

//continue with report here

[eyecatchup]

Hi Chris,

thanks for reporting.

I checked all options and there seems no way, to get hands on the majesticseo data without using whatever techniques (like for example the mentioned "with-credentials-and-cookie-curl-option") to get it.

As i moved the project to github, everybody is free to fork it and add whatever he wants. But i will not apply any fix for that (, as long as there's no official API access donated by somebody!? ;o)) Seriously, i don't care about automatically requesting data from open sources. But i will not start cheating on the data providers (on that level ;)), just to get what i want.

So the bottom line: (Unfortunately) The SEOstats_Majesticseo class will be removed from the main SEOstats class as of now.

[eyecatchup]

PS: @TheMajesticSeoTeam: Don't you think the little information, you gave for free before, was worth the benefit you gained from your pushed dns reputation? Feel free to send me your answer to eyecatchup@gmail.com.

Yours Stephan Schmitz

[MajesticSEO]

Hi Stephan,

Stephan, we've already emailed you last Saturday (3 Sep 2011, 7pm GMT) when a bot with "SEOSTATS" user agent was hitting our website from distributed IP addresses at rates of over 200 requests per second. This happened on Saturday morning and resulted in us having to recall our team to work on weekend in order to deal with this problem. To the best of my knowledge we have not received any response to our email which was rather disappointing given problems that we had to deal with.

We do provide a lot of data for free however when someone takes scraping to a level that affects stability of our servers then we have to take action against it.

Regards

Alex

[ChrisAlvares]

What if I send a pizza to the @TheMajesticSeoTeam if they have to work on a Saturday?

[eyecatchup]

Hello Alex!

Thank you, for taking the time to respond. I also want to apologize for missing your email (and thus not answering it). But i did not received that mail!?

Let me say, i can see your point clearly. As you can see from that old note here http://code.google.com/p/seostats/#Google_Pagerank_API_Info i had "the same" issues not too long ago. Definitively it's inacceptable, to affect your business in any negative way.

But hey, i am really surprised! The amount of requests on my server (singular) was almost 4 times of yours and he managed it nevertheless. And you're talking about affecting the stability of your servers (plural)? Anyway.. Eventhough i am the author of this class, i'm not using it more than twice a week. So i don't care about dropping the Majesticseo support for SEOstats. The only thing i feel sorry for, is that you could have solved that problem in a way it would not have affected "the mass" (and not only the bot users).

Respectfully,
Stephan

[eyecatchup]

Hey Alex, i've found your mail. It was catched by a custom regex filter rule in Outlook (that usualy does a good job). Anyway.. Just wanna let you know, that i got your mail now and will come back to you after the weekend. Kind regards, have and a nice weekend !

[MajesticSEO]

Hi,

I am glad you've finally received our email, I will respond to yours in a moment.

I am not particularly interested into comparing server loads because in all probability they do different jobs: querying database that searches in real time multi-terabyte data set with trillions of URLs is a very difficult task especially with high number of concurrent requests. Writing code to query such data is infinitely easier to code and execute in a scalable manner than actually responding to such queries, which is why we only support automation via our API that is optimized for such calls.

We plan to release a better solution for the users of our website next week (I was forced to stop another project in order to do that), however our position on automated calls is unchanged: the only acceptable route is via our API.

Regards,

Alex