QSearch uses a persistent key-value data store to provide super fast lookups of uniquely identifiable items as a drop-in component for web site search.
Python JavaScript
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


QSearch 0.1.4 README

About QSearch

QSearch is a proof-of-concept application using key-value data stores to quickly search
and look up resources uniquely identifiable within a context.

QSearch uses a persistent key-value store -- a giant hash -- to provide high performance 
lookups of uniquely identifiable items as a drop-in component for a web site search.

Downloading QSearch

Download or fork on https://github.com/FCC/QSearch

QSearch Requirements

Python  - a programming language - http://www.python.org/
Redis   - a memory resident key-value data store - http://redis.io/
webpy   - a python web server library with some templating - http://webpy.org/
redis-py - a python library for interacting with redis - http://github.com/andymccurdy/redis-py
JQuery  - an awesome and popular Javascript library - http://jquery.org/

Knowledge of command line, Python, and server administration.

QSearch was developed on a Mac Powerbook Air. The code and libraries should work
well on Mac, Unix, or Linux machines. The core components should also exist for
Windows machines, but it is left as exercise to the user user to make appropriate

Software Licenses

Redis is available under the New BSD License - http://www.opensource.org/licenses/bsd-license.php
webpy is in the public domain.
redis-py is open source without restriction provided copyright notice and terms are included.
QSearch code is in the public domain, initially authored by Greg Elin for the FCC.

QSearch Support

Support is not currently available for QSearch. If you are particularly interested in the 
project, I bet you can figure out how to reach the author.

The Idea Behind QSearch

Many items within an organization have a unique identifier or name. For examples, call signs
uniquely identify broadcast television and radio stations.

The emergence of cheap computer RAM and RAM resident NoSQL data stores supporting micro-second
look up across tens of millions of key-value pairs offers an excellent means to simulate
intelligent recognition of user input in search and other interactions.

QSearch loads a variety of uniquely identifiable items into a NoSQL memory-resident datastore
(Redis) in Json notation and a simple web server (webpy) to respond to provide an API for

QSearch Installation

1. You will need to have Python installed. (Developed on Python 2.6.)
2. You will need to download and install Redis from http://redis.io/. (Developed on version Redis 2.0.4).
   It is best if you modified your PATH to include Redis.
3. You will need to download and install webpy from http://webpy.org/. (Developed on version web.py 0.34)
4. You will need to download or remotely use jquery-1.4.4.js
5. You will need to have QSearch downloaded installed from https://github.com/FCC/QSearch.

After you have installed everything, adjust the app/static/searchjsonp.html file to indicate how
you are including the JQuery library.

Running and Using QSearch

1. Launch Redis from command line:

	$ cd /path/to/qsearch/
	$ redis-server config/redis.conf
2. Use the loader scripts from the utils/dataloaders folder to load data into Redis.
   Each script has instructions of how to launch from command line:

	$ cd /path/to/qsearch/utils/dataloaders
	$ python etl_csv_json.py -v -l "," -k 0 -p acronym -f ../../sampledata/acronyms.csv
	$ python etl_callsign_json.py -v -l "|" -k 0 -p callsign -f ../../sampledata/tvq-tvstations.txt
3. Adjust static/q_search.js script to call appropriate host. The q_host_port variable needs to be
set to the correct host for jsonp call to work.

	// Modify host to call for jsonp data
	var q_host_port = "localhost:8080"
4. Launch qsearch_redis.py:
	$ cd /path/to/qsearch/app/
	$ python qsearch_redis.py 8080
	Launching as daemon in the background (e.g., continue to run after terminal closed):
	nohup python qsearch_redis.py 8080 > ../logs/log.txt 2>&1 & 

5. Open http://yourhostname.com:8080/static/q_search.html

	Try a search term, like "SSA"

QSearch Features

• Code uses Jsonp and Jquery demonstrating that all interactions are controlled by Javascript and be added
  to a page independently of what that page is being sued for.
• Search is case insenstive.

QSearch API


{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}


callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})


{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}


callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})


myFunction({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})

Alternative Ways of Using and Testing QSearch

	A couple other screens from browser:

	command line:
	$ curl http://localhost:8080/search/json/SSA 
	$ curl http://localhost:8080/search/json/KOLO-TV

	command line piped to -mjson.tool for formatted display :
	$ curl http://localhost:8080/search/json/SSA | python -mjson.tool
	$ curl http://localhost:8080/search/json/KOLO-TV | python -mjson.tool

	command line with callback
	$ curl http://localhost:8080/search/jsonp/SSA 
	$ curl http://localhost:8080/search/jsonp/SSA?callback=myFunction

	python client to consumer qsearch_redis.py API
	$ python search_client.py 8081
	$ curl http://localhost:8081/ssa
	Or open browser and go to: 

Redis Tips

Launch redis command line (see http://redis.io/commands for all commands):

	$ redis-cli
	redis> get acronym:SSA
	"{\"definition\": \"Shared Services Agreement\", \"name\": \"SSA\", \"id\": \"SSA\"}"

QSearch FAQ

Q: How do I get my own data into Redis?
A: Fork and modify an appropriate script from the utils/dataloaders/ subdirectory to create a python loader.
   You can also write whatever scripts you need in whatever language you want to load data
   into Redis. If you add new keys, you will probably need to adjust the qsearch_redis.py script
   to search for the additional keys. 

Q: How should I format my data to put into Redis?
A: The Redis keys follow a pattern of "<class>:<id>", for example, "callsign:WIAT". The value
   associated with a key is formatted to be in a json format. See the scripts in utils/dataloaders
   to see how a delimited text file gets read, converted, and loaded into the Redis data store.

Q: Do I have to use Redis? Can I use memcached or another key-value store?
A: You can use another key-value store but you will need to change the code to do so. The code is simple
and there is not much abstraction. 

Q: How do I use all this on a Microsoft Windows PC?
A: You will need to find and install the Windows versions of the various tools and libraries and 
   modify file paths and code appropriately. It would be great if someone wanted to port all this
   to Windows and share with me the information.