Protocol oddities - what are Niantic up to?

[jonatkins]

Just looking at the network protocol from the stock intel map, and I notice a couple of extra parameters included in each request, that actually have some odd (obsfucated) values.

IITC currently doesn't recreate these correctly (it passes in an empty string), and the intel map still works successfully - but this protocol difference does mean that

1. Niantic can tell IITC from stock intel
2. It could be tested and would stop working

The two extra parameters are called b and c, and are added around the same place that the version code (v) is added to any request parameters. As of the stock intel update of 2014-08-25, this is in the following:

pd.prototype.Gg = function() {
  for (;;) {
    var a = this.Vd.lc();
    if (ea(a)) {
      var b = this.Wa.dequeue();
      if (!ea(b)) {
        b = this.Vd;
        b.oa.remove(a) && b.Yb(a);
        break;
      }
      var c = b, b = c.getMethodName();
      Ub(a, "complete", x(this.Wg, this, c));
      var d = "/r/" + b, e = a, f;
      if (f = !/^(GET|HEAD|OPTIONS|TRACE)$/.test("POST")) {
        f = "//" + document.location.host;
        var g = document.location.protocol + f;
        f = d == g || d.slice(0, g.length + 1) == g + "/" || d == f || d.slice(0, f.length + 1) == f + "/" || !/^(\/\/|http:|https:).*/.test(d);
      }
      f ? e.headers.set("X-CSRFToken", od("csrftoken")) : e.headers.remove("X-CSRFToken");
      c = c.getData();
      c.v = "f9cc465d75c8d13a564e14f2dce726f777760bd7";
      e = sd.g();
      c.b = e.eb;
      c.c = zd(e, b);
      a.send(d, "POST", JSON.stringify(c));
    } else {
      kd(this.Tb);
      break;
    }
  }
};

Note the lines after c.v = ..., where sd.g()is called, then the c.b and c.c values are set in the sent request.

[jonatkins]

The following is also new code that seems to be relevant

function zd(a, b) {
  var c = "mxkd";
  a.za[b] && 0 < a.za[b].length && a.za[b].shift().invoke(function(a) {
    c = a;
  });
  wd(a, b);
  return c;
}

function wd(a, b) {
  if (a.nb[b] && 0 < a.nb[b].length) {
    var c = a.nb[b].shift(), d = c.gf, c = "dkxm" == d ? new Pf : new a.sc[c.eb](d);
    a.za[b] || (a.za[b] = []);
    a.za[b].push(c);
  }
}

function ud(a, b, c) {
  var d = c.eb && a.sc[c.eb];
  if ("dkxm" == c.gf || d) a.nb[b] || (a.nb[b] = []), a.nb[b].push(c);
}

function td(a, b, c) {
  a.eb = b;
  if (!a.sc[b]) {
    var d = !1;
    if (c) try {
      eval(c);
    } catch (e) {
      d = !0;
    }
    d || (a.sc[b] = botguard.bg);
  }
}

function vd(a, b) {
  this.gf = a;
  this.eb = b;
}

function Pf() {}

Pf.prototype.invoke = function(a) {
  a("dkxm");
};

[jonatkins]

... and another piece of the puzzle is a large piece of (encrypted?) javascript in the main html page, included just before gen_dashboard.js itself.

      <script type="text/javascript" language="javascript">
        var B = "gUxMl-AzWMT7Gaa0PPntg58WW5trb6QbxSqpL8w2vCI";
        var CS = {"getPortalDetails": ["TH8pbe1OXfN7hA5Ctt1mRBo3/5hR+RC3YSlgu9lKV...

[ChristophPech]

In the Intel page itself: /* Anti-spam. Want to say hello? Contact (base64) Ym90Z3VhcmQtY29udGFjdEBnb29nbGUuY29tCg== */
Which translates to: botguard-contact@google.com

This is the tool which is used by Google itself to protect against automated account creation.

[jonatkins]

In this case I don't think it's for catching account creation - that's not something that can be done on the intel page itself.

The fact that it's "getPortalDetails" that's in the CS string suggests to me that this is what they're protecting. The good sign is that, at this time at least, IITC still works. Maybe the bot detection code is only triggered after 'excessive' requests, and before that they just don't care?

[iBotPeaches]

I think its a lot more than a honeypot / spam protection of account creation. The response on getPortalDetails has the a, b, c params [https://github.com/iBotPeaches/IntelMapDumps/commit/329e48cccfbfe70b24dc93d5161b27439312614b] while the request adds b and c like mentioned above.

I don't see the extra requests on others, only getPortalDetails, so its something limited to that request. I haven't figured out yet what b (the massive 4k character string) is yet.

[ChristophPech]

I didn't mean it is for account creation. I mean Niantic didn't write their own bot protection instead they use the code which google uses to protect their account creation.
Which means it es hard to break.
If you decode the bot-protection code from the intel html page you can find that it uses the following encryption algorithm: http://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm

[antisane]

In plain english please: does this mean that is safe or not to use IITC? I can't imagine playing ingress without IITC :(

[jonatkins]

For a plain-english response, see my comment on this post in the IITC community
https://plus.google.com/112875802603603993788/posts/bitaQ7iC7zM

[amallah]

Also relevant:

sd.prototype.R = function (a, b) {
  td(this, a);
  for (var c in b) {
    for (var d = 0; d < b[c].length; ++d) ud(this, c, new vd(b[c][d], a));
    wd(this, c)
  }
};

td loads what is global B into the DOM. The iterator is how ud, vd and wd are called to organize the seeded data with the passed parameters.

[ghost]

The botguard.bg function the code calls decodes to

function (a, b, c, d, e, g, h, l) {
    try {
        if (this.c = [], Q(this, this.b, 0), Q(this, this.o, 0), Q(this, this.A, 0), Q(this, this.h, []), Q(this, this.d, []), Q(this, this.Q, "object" == typeof window ? window : u), Q(this, this.R, this), Q(this, this.n, 0), Q(this, this.v, 0), Q(this, this.P, 0), Q(this, this.g, ya(4)), Q(this, this.w, []), Q(this, this.m, {}), Q(this, this.j, 2048), this.B = m, a && "!" == a[0]) this.p = a;
        else {
            for (ja(), b = D, c = [], d = 0; d < a.length;) {
                if (e = b[a.charAt(d++)], g = d < a.length ? b[a.charAt(d)] : 0, ++d, h = d < a.length ? b[a.charAt(d)] : 64, ++d, l = d < a.length ? b[a.charAt(d)] : 64, ++d, e == p || g == p || h == p || l == p) throw Error();
                c.push(e << 2 | g >> 4), 64 != h && (c.push(g << 4 & 240 | h >> 2), 64 != l && c.push(h << 6 & 192 | l))
            }(this.f = c) && this.f.length ? (this.T = [], this.C()) : this.e(this.ca)
        }
    } catch (n) {
        O(this, n)
    }

[kamilbrk]

Just dropping a link here guys as it might be useful for code deobfuscation http://jsnice.org/

[ghost]

Just a little question: why don't you use niantics methods for creating requests (pd.prototype.Gg) instead of building own?

[amallah]

That's literally what I ended up doing. I'll throw up a jsfiddle. You can just use niantic's own code to create the new variables. Based on what I saw, it doesn't look at the guid of the target portal and it does seed with a random number, so every call is different.

[amallah]

http://jsfiddle.net/go0w1foz/1/

[jonatkins]

@radopi The problem with using Niantic's methods is that they change the names of them on every site update. There's been another release this evening, and that method is now called rd.prototype.Gg

While IITC can detect some class/methods from their contents (we do that to extract the version string), reliably doing that for multiple methods becomes tricky to do reliably.

This site update is good though - before this, I didn't know which of the two character names would change as part of their usual site updates, and which, if any, would remain static. For example, the B and CS variables in the main html page remain the same, while the initialisation line in the main script has changed from sd.g().R(B, CS); to ud.g().R(B, CS, nd); (new object name, and a new parameter).

Making this work right does involve some guesswork, and may involve some assumptions on what does/doesn't change between site updates. I want to be as certain as reasonably possible IITC's code in this area is right - as I think it's safer for users to be passing the current 'blank' strings rather than the random looking strings that look right to us, but Niantic can detect are wrong in some way.

I've had a quick glance at the new update and I think I have a fair idea of what needs to be done. I just need to have a closer look again, to ensure I'm not making any daft mistakes.

[ghost]

@jonatkins the new param was added, because they changed the CS array to no longer contain the method names and added nd, which contains them

[jonatkins]

I've had a further look at the code. A possible major pitfall is that, for the relevant requests (getPortalDetails only, at this time), the server can return a string which is then passed to eval to execute it as javascript.

Where is this - follow the response handling code in rd.prototype.Wg

• e is the parsed JSON response
• k is the a value from the server response
• vd(f,l,k) is called - and this can run eval on it's third parameter

Now, I've only seen empty strings so far in the server responses, so if this continues in most cases, IITC can still work OK when this is the case. However, as soon as we get a non-blank string, there's very little we can do with it in a safe/sensible way.

As, for now, Niantic are only protecting the getPortalDetails method, and no others have this behaviour, some parts of IITC will still work. So, my plan is, once I've understood the flow of the code fully, is to

• set the b/c request parameters the same as the stock site
• when there's no a string to eval in the response, process it the same as the stock site
• when we do get an a string, instead of passing it to 'eval'
  • log it
  • block further requests of that type - until reload, or least for a period of time
  • dialog to warn the user that functionality is restricted

Perhaps, once we see some examples of returned strings to eval, it will be possible to handle them in a sensible way - but before then there's nothing we can do sensibly with them.

Of course, each site update could add/change code in this area again, breaking anything we do. Fingers crossed we can get something that works well enough.

[ghost]

@jonatkins maybe you could add an option to send the data to your server, so you have more data

[macrojames]

Now, getPlexts is also affected.
I expect every request which is not sent in parallel will be affected in the near future.

[flamusdiu]

@radopi that sounds like a good idea but there is a privacy problem with doing that as well. Also, should always be an Opt-in option and a way to disable if you wanted to.

[l1bbcsg]

I got myself this parameters.
c in my case has 6044 characters
a contains heavily obfuscated code that starts with this comment: /* Anti-spam. Want to say hello? Contact (base64) Ym90Z3VhcmQtY29udGFjdEBnb29nbGUuY29tCg== */ base64 decodes to botguard-contact@google.com
The code contains a lot of generic stuff: useragent sniffing (yay, Internet Explorer), object type detection, code for listening to mouse/kb events... So it's looks like some generic library. After that there's a lot of cryptic stuff that makes absolutely no sense. The interesting bit is that it calls something labeled as "botguard.bg.prototype.invoke".
Evidently this must be google's botguard. From what I know, It collects data about user behavior on the page and its browser and avaluates it against other know data, this way it can detect anomaly usage and detect bots (kinda like clienBlob in ingress client). My guess would be it's detecting what kind of actions it takes the user to send requests (clicks, map events would be the most sensible)

[jonatkins]

Various recent commits, ending with 92aae95, should implement all is needed for this.