- This is "Spike" Code. It works but has Zero Unit Tests so is for Learning Purposes Only!
- Was written for the previous version of Meteor and requires some tweaking for Meteor 1.0 (which we are not using)...
People want to find content. Good search is essential.
Full-text search. Using Native MongoDB Commands.
Example app:
By default, Meteor starts up its own instance of MongoDB, this does not have full-text indexing/search, so you need to go native.
If you don't already have a "Real" installation of MongoDB on your system, install it with HomeBrew:
brew update
brew install mongodb
If you're not using Mac ...
- Linux: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-linux/
- Windows? ouch... http://www.geekyprojects.com/tutorials/how-to-use-virtualbox-tutorial/
In a terminal/console window startup up your mongod databse with the following command:
mongod --dbpath ~/code/meteor-search/.meteor/local/db --setParameter textSearchEnabled=true
Notes:
- --dbpath speficies where your data is. Your data directory will be different, my project is at ~/code/meteor-search (replace for what ever yours is)
- --setParameter lets you enable full-text search at run time. (you can do this in a config file if you prefer see: http://docs.mongodb.org/manual/reference/configuration-options/)
Confirm its working by visiting: http://localhost:28017/ (in your browser)
More info on enabling text search: http://docs.mongodb.org/manual/tutorial/enable-text-search/
Because we are using "real" Mongo we need to specifiy it when starting meteor. (This only applies while you are developing in production you would set an ENV
)
MONGO_URL="mongodb://localhost:27017/meteor" meteor
If the app starts up ok, its game on!
(otherwise submit a bug to this repo and I will wil try to assist you!)
When you boot this app it will access the Twitter Streaming API and fetch thousands of tweets for you to search through (locally). (leave it running for a few minutes and you will get 10k posts. Or a few hours if you want hundreds of thousands to stress test search)
If you want LOTS of content very quickly, change the KEYWORD to news.
If you want INSANE amounts of (noisy) data (to symulate high volume), use:
var KEYWORDS = "katie, justin, kim, beyonce, miley, Obama, 1DWorld, OMG, FML, breaking, news";
Once you have some content you need to ensure that MongoDB is indexing it.
Thankfully this is quite easy with MongoDB's ensureIndex method. In our case we are simply going to index the post's body field:
db.posts.ensureIndex( { body: "text" },{ background:true } );
There's a startup script that does this automatically for you at: server/indexes.js
Depending on how much data you have already inserted, this may take some time... I had 92k posts (tweets) in my DB when I ran it and it took less than 10 seconds!
More detail on ensureIndex:
- http://docs.mongodb.org/manual/core/index-text/
- http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex/
db.posts.runCommand( "text", { search: "paypal" } )
While this works fine in RoboMongo:
Meteor does not support the runCommand method:
So ...
I wrote a simple regular expression to turn hashtagged keywords into links. Instead of polluting the raw data with links (and bloating our records) We finde/replace the #keywords at render time (client-side) using a Handlebars template helper method:
// place this code in your main.js or inside an Meteor.isClient block
Handlebars.registerHelper('highlight', function(text) {
var hashtagPattern = /\s*(#\w*)/gi,
link = "/search/",
m, match, matches = [], t, url ='';
// initial check for hashtag in text
if(text.indexOf("#") !== -1) {
// find all #keywords (that have hashtags)
while ( (match = hashtagPattern.exec(text)) ) {
matches.push(match[0]);
}
// replace any #keywords with <a href="/search/keywords">#keywords</a>
for(var j=0; j < matches.length; j++) {
m = matches[j].replace(/\s/g, "");
// console.log('match',m);
url = link+m;
url = url.replace('#',"").toLowerCase(); // remove hashtag for lookup
t = " <a class='hashtag' href='"+url+"'>"+m+"</a> "; // replace with
replace = new RegExp("\\s*("+m+")", 'gi');
text = text.replace(replace, t);
}
}
return text;
});
Note: the link pattern is hard-coded /search/:keywords and method is not
chainable so far from perfect! Send a pull-request if you improve on it. :-)
MongoHQ does not have (full) text indexing enabled by default.
But they were quick to help me when I asked for it: https://twitter.com/nelsonic/statuses/451758108285489152
You will need to set up your indexes manually with a command
(either in your Mongo Client - We use RoboMongo - or the Web Interface)
Once that is set up you are good to go.
In RoboMongo (or what ever MongoDB Client of your choice) use the following command to search through your collection (on the field(s) you specified as searchable)
db.COLLECTION.runCommand( "text", { search: "KEYWORDS" } );
// e.g:
db.posts.runCommand( "text", { search: "learn" } );
Search results returned in 0.15 seconds when I had 327k posts in the collection:
This project uses Iron Router for url routing.
If you are not familiar with it
(you should be if you're serious about using Meteor), read:
- Tutorial: http://www.manuel-schoebel.com/blog/iron-router-tutorial
- Docs: https://github.com/EventedMind/iron-router
- Discover Meteor Uses Iron Router extensively. Read it for a good step-by-step intro.
- Advanced: https://properapp.com/meteor/advanced-routing-in-meteor-navigation-state-w/#.Uz8BIS9dVX4
mrt add iron-router
See the routes.js file for more detail on how I've wired this up to
accept request in the form: http://yoursite.com/search/:keywords
- MeteorPedia Full-text Search: http://www.meteorpedia.com/read/Fulltext_search
- Using Elastic Search with MongoDB: https://github.com/matteodem/meteor-easy-search
- Meteor (MongoDB) Full-text search: http://stackoverflow.com/questions/14567856/full-text-search-with-meteor-js-and-mongodb (non-answer!)
- Lunr JS "Simple" full-text search in your browser: http://lunrjs.com/ (but our data is in MongoDB so not much use...)
- Heroku ENVironment variables (in Node.js Apps): https://devcenter.heroku.com/articles/nodejs-support