Update scraper to work with latest xbox.com update

[jasonclemons]

With the latest xbox.com update, the scraper no longer returns the proper information. Rewrite it so that it can scrape the right information.

[bartekcc]

Just ignore /profile.json?gamertag= and try to get games.json. It should contains all information u need.

I have random errors when trying to get games.json, but usually it works well.
profile.json always returns error.

[djekl]

The latest updates to xbox.com mean that the profile pages REQUIRE javascript. Without using a JS Scraper solution (PhantomJS/CasperJS) you can't get the pages. This is because cURL and file_get_contents() can't mock javascript.

[jasonclemons]

@djekl is correct. I am looking into other libraries that can accomplish what needs to be done. With the 3.0 rewrite I have been slowly working on, it will allow the use of other libraries more easily by making it more Composer-friendly.

[rubenyc]

Is there any update guys?

[reidmain]

It doesn't look like all the profile information is on the redesigned pages. Things like the user's online status and what game they are playing are absent.

[jasonclemons]

I'm currently exploring alternate methods of obtaining this info. The rewrite may take some time.

[djekl]

@reidmain Please remember that this is just a side project for @jasonclemons and I contribute when I can. The rewrite to get around the new issue of pages requiring javascript could take some time

[reidmain]

I totally understand @djekl. I was actually scraping the Xbox.com website just like you guys for a side project I was working on. I was about a week away from launching it when the redesigned happened and so the project has basically been scuttled until I find a workaround. That search lead me to this project.

If you guys need any help investigating alternate data sources just ask. So far the new Xbox.com website seems to be void of the information I need and sniffing traffic from the Xbox SmartGlass app has also proved fruitless.

[djekl]

@reidmain What information are you looking for? Scraping xbox.com will only ever get you the information they want you to have. I have a closed source site that uses the smartglass api (https://xboxapi.com). Its just going through an upgrade today, so may not be 100% stable.

As for this project, I think that the only real option moving forward would be to implement PhantomJS or CasperJS to scrape xbox.com. The only problem with these scripts is that they can be rather slow to run.

[reidmain]

@djekl I was scraping the user's gamerpic, online status and the name of the game they were playing before. Even if you weren't friends with that person their online status used to be displayed if they didn't have the privacy settings turned on. I have been able to make the "profile" request to the SmartGlass API but it seems incredibly stale.

For example while I was testing the SmartGlass API I had the Twitch app running on my Xbox One. I then logged into my dummy account on xbox.com and saw that my main account was using the Twitch app. However according to the SmartGlass app I had "been last seen on Xbox.com 30 minutes ago" because at some point AFTER I started the Twitch app I had visited xbox.com and it was only remember the last thing I did.

[djekl]

@reidmain If you have access to the smartglass code, then its the presence your after. That shit is so real time its scary, you can also see if they have anything snapped etc.

[reidmain]

@djekl just to be sure we're talking bout the same thing I am looking at https://uds-part.xboxlive.com/profile.svc/profile?sectionFlags=97&gamertag=Blah and using the PresenceInfo element.

[djekl]

@reidmain I can't give away the information I have, but thats not what I am looking at. For example, XboxAPI tracks XUID's not gamertags. This is because a gamertag can change, the XUID can't.

An example of a users presence (me right now) https://xboxapi.com/v2/2533274813081462/presence

{
    "xuid": 2533274813081462,
    "state": "Online",
    "devices": [
        {
            "type": "XboxOne",
            "titles": [
                {
                    "id": 714681658,
                    "name": "Home",
                    "placement": "Background",
                    "state": "Active",
                    "lastModified": "2014-07-19T14:27:50.8152271Z"
                },
                {
                    "id": 2137576163,
                    "name": "Party",
                    "placement": "Snapped",
                    "state": "Active",
                    "lastModified": "2014-07-19T14:27:50.8152271Z"
                },
                {
                    "id": 1292135256,
                    "name": "Titanfall",
                    "placement": "Fill",
                    "state": "Active",
                    "lastModified": "2014-07-19T14:27:50.8152271Z"
                }
            ]
        }
    ]
}

and my profile https://xboxapi.com/v2/2533274813081462/profile

{
    "id": 2533274813081462,
    "hostId": null,
    "Gamertag": "djekl",
    "GameDisplayName": "djekl",
    "AppDisplayName": "djekl",
    "Gamerscore": 56751,
    "GameDisplayPicRaw": "http://images-eds.xboxlive.com/image?url=7OTVnZUMVj4OV2zUUGecWvn3U00nQQLfK7_kwpANoghmuNYU71QsOgoG0nx6CNCqtqlZY9IpjKWCzvvk2CkD6qjlO3JBK6bqVYPgnMVLUC..LUR4y4yIeKpA65OH6VqFILokohbfp3amhN.29ni_fA--&format=png",
    "AppDisplayPicRaw": "http://images-eds.xboxlive.com/image?url=7OTVnZUMVj4OV2zUUGecWvn3U00nQQLfK7_kwpANoghmuNYU71QsOgoG0nx6CNCqtqlZY9IpjKWCzvvk2CkD6qjlO3JBK6bqVYPgnMVLUC..LUR4y4yIeKpA65OH6VqFILokohbfp3amhN.29ni_fA--&format=png",
    "AccountTier": "Gold",
    "XboxOneRep": "GoodPlayer",
    "PreferredColor": "http://dlassets.xboxlive.com/public/content/ppl/colors/00015.json",
    "TenureLevel": 6,
    "isSponsoredUser": false
}

and my gamercard https://xboxapi.com/v2/2533274813081462/gamercard

{
    "gamertag": "djekl",
    "name": "Loading...™",
    "location": "Newcastle, England",
    "bio": "Quitting the game since 1988... Fallen friends will never be forgotten! Lets hope I'm not when its my time.",
    "gamerscore": 56751,
    "tier": "Gold",
    "motto": "www.twitter.com/djekl",
    "avatarBodyImagePath": "http://avatar.xboxlive.com/avatar/djekl/avatar-body.png",
    "gamerpicSmallImagePath": "http://avatar.xboxlive.com/avatar/djekl/avatarpic-s.png",
    "gamerpicLargeImagePath": "http://avatar.xboxlive.com/avatar/djekl/avatarpic-l.png",
    "gamerpicSmallSslImagePath": "https://avatar-ssl.xboxlive.com/avatar/djekl/avatarpic-s.png",
    "gamerpicLargeSslImagePath": "https://avatar-ssl.xboxlive.com/avatar/djekl/avatarpic-l.png",
    "avatarManifest": "AAAAAAAAAAA/gAAAABAAAAMdAAPByPEJoZyy4AAIAAADLwADwcjxCaGcsuAAIAAAAzwAA8HI8QmhnLLgAACAAALuAAPByPEJoZyy4AAAAAAAAAAAAAAAAAAAAAAAACAAAowAA8HI8QmhnLLgAAAAAAAAAAAAAAAAAAAAAAAAQAACcQADwcjxCaGcsuAAAAAAAAAAAAAAAAAAAAAAAAEAAALXAAHByPEJoZyy4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAADEQADwcjxCaGcsuAAAAAAAAAAAAAAAAAAAAAA/76HSf83IRb/5YmV/yQ/U/83IRb/qqbJ/zchFv+qHSb/qh0mAAAAAgAAAAHByPEJoZyy4AACAAAAAAAAAAAAAAAAAAAAAAABAAIAA8HI8QmhnLLgAAEAAAAAAAAAAAAAAAAAAAAAEAAjzEIjyAw4dEFWCEQQAAAAAAAAAAAAAAAAAAAAAAAKuBv74hHO/5uQWFgIMAq4AAAAAAAAAAAAAAAAAAAAAAVETziCg8xDnNFYWAggBUQAAP//AAD/AP8A/wAA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIHosEjHC5ploWFgIFgAgAAAAAAAAAAAAAAAAAAAAAAAQAJAAAcHI8QmhnLLgABAAAAAAAAAAAAAAAAAAAAAAAAgAbwABwcjxCaGcsuAACAAAAAAAAAAAAAAAAAAAAAAABAHWAAPByPEJoZyy4AAEAAAAAAAAAAAAAAAAAAAAAAAAAHYnWgEAAAkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=="
}

What your playing with there is not the latest version of smartglass. Your looking at the old Xbox360 SOAP api. This was given to the XCDP members. That doesn't support Xbox One data.

[reidmain]

@djekl Ah OK. Yeah I'm sniffing the traffic of the iOS SmartGlass app which is obviously using the old SOAP API. Thanks for the heads up.

[djekl]

@reidmain The iOS, Android, and Windows8 Smartglass apps all use the new ones, if you are using Xbox One Smartglasss. If your looking at the old Xbox Smartglas (for Xbox360) then thats where your getting SOAP endpoints from.

[reidmain]

@djekl holy crap they released a second app? God damn I had no idea. Again, many thanks.

[djekl]

@reidmain Yeah a few months before XboxOne launch. Android even has a beta of the Smartglass app for XboxOne.

iOS

Xbox 360 Smarglass --> https://itunes.apple.com/gb/app/xbox-360-smartglass/id480914036?mt=8
Xbox One Smarglass --> https://itunes.apple.com/gb/app/xbox-one-smartglass/id736179781?mt=8

Android

Xbox 360 Smarglass --> https://play.google.com/store/apps/details?id=com.microsoft.smartglass&hl=en_GB
Xbox One Smarglass --> https://play.google.com/store/search?q=xbox%20one%20smartglass&c=apps&hl=en_GB

[reidmain]

@djekl wow that is crazy. This new API is 10x better. It actually uses JSON for gods sake.

[djekl]

@reidmain yup, but authentication is a bitch

[reidmain]

@djekl I am seeing that. Still thanks for the heads up man. I may actually be able to decode this.

[djekl]

@reidmail Its certianly possible 😜

Knowing what I do know, I got hung up on something that isn't nessisary.

[prathmeshranaut]

Any help need with this? I am willing to help!

[djekl]

Its an open source project, feel free to fork it and submit a pull request :)

Sent from my iPhone

On 20 Jul 2014, at 03:17 pm, Aayush Ranaut notifications@github.com wrote:

Any help need with this? I am willing to help!

—
Reply to this email directly or view it on GitHub.

[prathmeshranaut]

I though that we could help each other since we all are stuck at the same point i.e. RPS Authentication.

[jbowens]

@aayushranaut did you ever figure out RPS authentication?

[prathmeshranaut]

@jbowens Nah, I forgot about it after a few tries.

[thomasm789]

Is there any more updates on your progress guys?? I'm currently looking at getting gamercard information, online status etc for a crews website that I am developing.

How far did you guys get since Feb?

[dregimbal1]

Hate to kick up an old thread but in case anyone ever stumbles through here looking for a solution to this bug I wrote up my few cents on pastebin (because it is long winded haha). Without a Microsoft Partner ID you will not be able to send party invites, initiate game invites - vice versa for accepting them, or speak into parties (without proper certificates). That said, you can still get a lot of information doing it the proper way. If you are looking to provide a service you won't be able to do much without the provider of that service storing your password and email is almost-plain-text (you can encrypt it and make it hard to read but still). I have tried authorizing users on my own site using OAuth and using that access_token (but that token does not have access to the scope we are looking for so it's worthless)

http://pastebin.com/LitnfcyS

[djekl]

Just to let you know, authorising a user via oAuth isn't impossible. I have done it successfully over on https://xboxapi.com (that way you don't need to store your password with us, or even use it for a one time login via us)

[dregimbal1]

I have done that too but found the Client ID does not have access to grant permission to the scope: service::user.auth.xboxlive.com::MBI_SSL

[djekl]

No no, I have a full Official oAuth package and login process for a user. If you wish, sign up for a free account and test it out for yourself. I FULLY authenticate the user against the Official Xbox API via oAuth

[dregimbal1]

Yeah but you're still technically 'scraping' credentials, right? If it were done through oAuth users would be given something like this:

Where they enter their credentials through Microsoft directly and you are given an access_token. That access_token is what your Client ID (website) would use to generate further tokens to get access to xbox live services.

What endpoint are you using to authenticate?

EDIT:

I see you using scope=Xboxlive.signin+Xboxlive.offline_access
I've tried that too but the access_token I got back wouldn't let me make any requests doing it the same as the manual way

[djekl]

No, I am using the actual oAuth process. If you try it then you are redirected away from XboxAPI and towards the login.live page. And I am sorry, but I am not willing to share this information.

As I have stated, its a fully functional oAuth process, redirects to login.live.com then over to xbx-api.com as xboxapi.com has a banned word (go figure), this then redirects back to xboxapi.com with the access token in the url fragment and finishes off the process there.

[markadrake]

I'm banging my brains out trying to solve this too. I've authenticated with the required credentials, but since I don't know what the endpoints are to request the profile data I can't go any further.

@regimbal93 that writeup looks nice, how far does it get you? I'm looking to pull back the gamertag of the user & achievements. I'd also like to know if they are online, but that's not a necessity.

[dregimbal1]

A lot of the endpoints have been revealed (A few listed in this pastebin)

Which ones you have in mind?

The only authentication I was able to figure out was more of a jimmy rig .. It would be nice to know the way to do it via Xbox OAuth but that way doesn't seem to cough up the right scope for certain things

[markadrake]

Hi @regimbal93 and thanks for replying back so quickly. I'm looking to create an OAuth provider for ASP.NET (Core) to handle all of the work you described via pastebin. Thanks so much for your hard work I'm sure it took a lot of time to figure this out.

You don't do any C# work do you? :) I'm essentially building a class like one found in this GitHub repo (https://github.com/aspnet-contrib/AspNet.Security.OAuth.Providers).

I could really use the help as I've got enough on my plate. After the OAuth takes place, I'm only looking to pull back the gamertag and achievements. The purpose being that I can validate they own (or at least played) a game.

I've tried to speak with people at the ID program at Xbox but between their autoresponders and carelessness to develop relationships with people and business owners I've received no answers, no introductions, nothing. So I'm just going to have to do it the hacky way it seems.

I'm looking to do this with Sony too in the future.

[dregimbal1]

ID@Xbox is looking for studios who have a game to push to their marketplace. The "Xbox API" one day apparently will be public but in the meantime it is a thick glass wall. It quickly turns into a slippery slope. Eventually people will build bots and you will have more spam in your inbox, random friend requests, etc.

Ideally you are a MS Partner with a Client ID that has access to the secure auth.xbox.com scope. Since for us this is not the case we are left with two scopes: Xboxlive.signin Xboxlive.offline_access both of which are open to the public but the endpoints that go with them are to be determined. I've tried reverse engineering it but I have not come up with a solution yet.

I figure the scope we know to be open has no relation to the more complex/cool API used in the smart glass apps. Something more simple like a POST to https://live.xbox.com/en-US/Friends/List with the access_token in the header (similar calls as seen on xbox.com)

I'll take another look this weekend. I am pretty busy too but I would definitely help you where I can to tear down this glass wall. Do you use Skype?