I figured it out.  It's true the key function isn't allowed inside a key
definition, since it may cause a cycle.

I fixed the XSL to workaround this limitation.  Essentially I just had to
change parts like this:
    key('PointsBy_Id', @callerid)
into like this to get the node directly without using key():
    /ProffyResults/PointsEncountered/Point[@id=current()/@callerid]

Now it works great in Firefox and IE.
Chrome still shows blank page; no idea why.

Note there was a stack overflow crash[1] in Firefox in 2009 that
prompted them to disallow key() function in here.  Interestingly,
it seems they fixed it in 2009[2] like a week after the crash was
reported, but the fix didn't really get applied until Sep 2015[3].

1: https://bugzilla.mozilla.org/show_bug.cgi?id=527558
2: mozilla/newtab-dev@6b175b5
3: https://hg.mozilla.org/mozilla-central/rev/d858d1b52b6b