Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
213 lines (138 sloc) 7.62 KB
QSearch 0.1.4 README
About QSearch
QSearch is a proof-of-concept application using key-value data stores to quickly search
and look up resources uniquely identifiable within a context.
QSearch uses a persistent key-value store -- a giant hash -- to provide high performance
lookups of uniquely identifiable items as a drop-in component for a web site search.
Downloading QSearch
Download or fork on
QSearch Requirements
Python - a programming language -
Redis - a memory resident key-value data store -
webpy - a python web server library with some templating -
redis-py - a python library for interacting with redis -
JQuery - an awesome and popular Javascript library -
Knowledge of command line, Python, and server administration.
QSearch was developed on a Mac Powerbook Air. The code and libraries should work
well on Mac, Unix, or Linux machines. The core components should also exist for
Windows machines, but it is left as exercise to the user user to make appropriate
Software Licenses
Redis is available under the New BSD License -
webpy is in the public domain.
redis-py is open source without restriction provided copyright notice and terms are included.
QSearch code is in the public domain, initially authored by Greg Elin for the FCC.
QSearch Support
Support is not currently available for QSearch. If you are particularly interested in the
project, I bet you can figure out how to reach the author.
The Idea Behind QSearch
Many items within an organization have a unique identifier or name. For examples, call signs
uniquely identify broadcast television and radio stations.
The emergence of cheap computer RAM and RAM resident NoSQL data stores supporting micro-second
look up across tens of millions of key-value pairs offers an excellent means to simulate
intelligent recognition of user input in search and other interactions.
QSearch loads a variety of uniquely identifiable items into a NoSQL memory-resident datastore
(Redis) in Json notation and a simple web server (webpy) to respond to provide an API for
QSearch Installation
1. You will need to have Python installed. (Developed on Python 2.6.)
2. You will need to download and install Redis from (Developed on version Redis 2.0.4).
It is best if you modified your PATH to include Redis.
3. You will need to download and install webpy from (Developed on version 0.34)
4. You will need to download or remotely use jquery-1.4.4.js
5. You will need to have QSearch downloaded installed from
After you have installed everything, adjust the app/static/searchjsonp.html file to indicate how
you are including the JQuery library.
Running and Using QSearch
1. Launch Redis from command line:
$ cd /path/to/qsearch/
$ redis-server config/redis.conf
2. Use the loader scripts from the utils/dataloaders folder to load data into Redis.
Each script has instructions of how to launch from command line:
$ cd /path/to/qsearch/utils/dataloaders
$ python -v -l "," -k 0 -p acronym -f ../../sampledata/acronyms.csv
$ python -v -l "|" -k 0 -p callsign -f ../../sampledata/tvq-tvstations.txt
3. Adjust static/q_search.js script to call appropriate host. The q_host_port variable needs to be
set to the correct host for jsonp call to work.
// Modify host to call for jsonp data
var q_host_port = "localhost:8080"
4. Launch
$ cd /path/to/qsearch/app/
$ python 8080
Launching as daemon in the background (e.g., continue to run after terminal closed):
nohup python 8080 > ../logs/log.txt 2>&1 &
5. Open
Try a search term, like "SSA"
QSearch Features
• Code uses Jsonp and Jquery demonstrating that all interactions are controlled by Javascript and be added
to a page independently of what that page is being sued for.
• Search is case insenstive.
QSearch API
{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}
callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})
{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}
callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})
myFunction({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})
Alternative Ways of Using and Testing QSearch
A couple other screens from browser:
command line:
$ curl http://localhost:8080/search/json/SSA
$ curl http://localhost:8080/search/json/KOLO-TV
command line piped to -mjson.tool for formatted display :
$ curl http://localhost:8080/search/json/SSA | python -mjson.tool
$ curl http://localhost:8080/search/json/KOLO-TV | python -mjson.tool
command line with callback
$ curl http://localhost:8080/search/jsonp/SSA
$ curl http://localhost:8080/search/jsonp/SSA?callback=myFunction
python client to consumer API
$ python 8081
$ curl http://localhost:8081/ssa
Or open browser and go to:
Redis Tips
Launch redis command line (see for all commands):
$ redis-cli
redis> get acronym:SSA
"{\"definition\": \"Shared Services Agreement\", \"name\": \"SSA\", \"id\": \"SSA\"}"
QSearch FAQ
Q: How do I get my own data into Redis?
A: Fork and modify an appropriate script from the utils/dataloaders/ subdirectory to create a python loader.
You can also write whatever scripts you need in whatever language you want to load data
into Redis. If you add new keys, you will probably need to adjust the script
to search for the additional keys.
Q: How should I format my data to put into Redis?
A: The Redis keys follow a pattern of "<class>:<id>", for example, "callsign:WIAT". The value
associated with a key is formatted to be in a json format. See the scripts in utils/dataloaders
to see how a delimited text file gets read, converted, and loaded into the Redis data store.
Q: Do I have to use Redis? Can I use memcached or another key-value store?
A: You can use another key-value store but you will need to change the code to do so. The code is simple
and there is not much abstraction.
Q: How do I use all this on a Microsoft Windows PC?
A: You will need to find and install the Windows versions of the various tools and libraries and
modify file paths and code appropriately. It would be great if someone wanted to port all this
to Windows and share with me the information.