serve facebookexternalhit user agent with spiderable

[ayal]

This change will make Spiderable serve facebookexternalhit user agent.

Using Spiderable after this change would enable one to manipulate the head tag in "client code", which would then actually be served to the Facebook bot when it comes looking for "open graph meta tags" (i.e https://developers.facebook.com/docs/opengraphprotocol/)

The main code-change is adding "facebookexternalhit" to the condition under-which the phantomjs process is initiated. Other than that there are two other complementing code-changes:

1. A rewrite of the code that re-assembles the request url without the "escaped_fragment" query param. This is needed because in the current version of Spiderable the code assumes there is always a query string (which might not be the case in the proposed version)
2. A small bug-fix to the regex test for html validity of the phantomjs response page (html openning tag might contain attributes, especially after client manipulation, which is the case here)

To test this:

1. Create a new meteor app
2. On client startup, add a meta tag to head. Something like:

$("head").append('<meta property="og:title" content="spidy" />');

1. Optionally try having the content arrtibute be something you fetch from the DB in order to simulate how a user might use this. Preferably don't use autopublish, and put the code in the callback of the "subscribe" call to the data you're using. This is to make sure "_allSubscriptionsReady() === false" until you've done your manipulation.
2. Test this with Facebook Debugger: https://developers.facebook.com/tools/debug (notice there's an option at the bottom to actually see what your server served Facebook)

[75lb]

something like this will come in useful for me at some point.. for the time being i'm serving Open Graph objects from the public folder.. Facebook gets the metadata it needs, a user visit gets redirected into the main app..

[ayal]

@75lb, while this is being reviewed - you may be interested in headly:
https://github.com/ayal/headly

[n1mmy]

Thanks for the patch, @ayal! This looks good.

One change I'd request, though: can you pull the 'facebookexternalhit' constant out into a list of user-agents to check for. So the test is something like "is the user-agent in this list" not "is the user-agent facebook". Then it's easier to add more later, and more clear what is going on.

Also, you'll need to sign the Meteor contributor agreement before we can merge your patch. http://contribute.meteor.com/. Thanks!

[ayal]

@n1mmy, see my latest commit with the "agents" list.
Also, I was unable to submit the contribute form. I signed with GitHub successfully, but filling the form always just results in: "You must sign in with GitHub, all fields are required and you must type "I AGREE" in the last box." (which I did)

[ayal]

ok @n1mmy, I managed to sign the contributor agreement.

[n1mmy]

Great, thanks @ayal. Will test and merge.

[n1mmy]

Merged. Thanks!