Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

QSearch uses a persistent key-value data store to provide super fast lookups of uniquely identifiable items as a drop-in component for web site search.

branch: master
README
QSearch 0.1.4 README

About QSearch
=============

QSearch is a proof-of-concept application using key-value data stores to quickly search
and look up resources uniquely identifiable within a context.

QSearch uses a persistent key-value store -- a giant hash -- to provide high performance 
lookups of uniquely identifiable items as a drop-in component for a web site search.

Downloading QSearch
===================

Download or fork on https://github.com/FCC/QSearch

QSearch Requirements
====================

Python  - a programming language - http://www.python.org/
Redis   - a memory resident key-value data store - http://redis.io/
webpy   - a python web server library with some templating - http://webpy.org/
redis-py - a python library for interacting with redis - http://github.com/andymccurdy/redis-py
JQuery  - an awesome and popular Javascript library - http://jquery.org/

Knowledge of command line, Python, and server administration.

QSearch was developed on a Mac Powerbook Air. The code and libraries should work
well on Mac, Unix, or Linux machines. The core components should also exist for
Windows machines, but it is left as exercise to the user user to make appropriate
modifications.

Software Licenses
=================

Redis is available under the New BSD License - http://www.opensource.org/licenses/bsd-license.php
webpy is in the public domain.
redis-py is open source without restriction provided copyright notice and terms are included.
QSearch code is in the public domain, initially authored by Greg Elin for the FCC.


QSearch Support
===============

Support is not currently available for QSearch. If you are particularly interested in the 
project, I bet you can figure out how to reach the author.


The Idea Behind QSearch
=======================

Many items within an organization have a unique identifier or name. For examples, call signs
uniquely identify broadcast television and radio stations.

The emergence of cheap computer RAM and RAM resident NoSQL data stores supporting micro-second
look up across tens of millions of key-value pairs offers an excellent means to simulate
intelligent recognition of user input in search and other interactions.

QSearch loads a variety of uniquely identifiable items into a NoSQL memory-resident datastore
(Redis) in Json notation and a simple web server (webpy) to respond to provide an API for
lookups.


QSearch Installation
====================

1. You will need to have Python installed. (Developed on Python 2.6.)
2. You will need to download and install Redis from http://redis.io/. (Developed on version Redis 2.0.4).
   It is best if you modified your PATH to include Redis.
3. You will need to download and install webpy from http://webpy.org/. (Developed on version web.py 0.34)
4. You will need to download or remotely use jquery-1.4.4.js
5. You will need to have QSearch downloaded installed from https://github.com/FCC/QSearch.

After you have installed everything, adjust the app/static/searchjsonp.html file to indicate how
you are including the JQuery library.


Running and Using QSearch
=========================

1. Launch Redis from command line:

	$ cd /path/to/qsearch/
	$ redis-server config/redis.conf
	
2. Use the loader scripts from the utils/dataloaders folder to load data into Redis.
   Each script has instructions of how to launch from command line:

	$ cd /path/to/qsearch/utils/dataloaders
	$ python etl_csv_json.py -v -l "," -k 0 -p acronym -f ../../sampledata/acronyms.csv
	$ python etl_callsign_json.py -v -l "|" -k 0 -p callsign -f ../../sampledata/tvq-tvstations.txt
	
	
3. Adjust static/q_search.js script to call appropriate host. The q_host_port variable needs to be
set to the correct host for jsonp call to work.

	// Modify host to call for jsonp data
	var q_host_port = "localhost:8080"
	
4. Launch qsearch_redis.py:
	
	$ cd /path/to/qsearch/app/
	$ python qsearch_redis.py 8080
	
	Launching as daemon in the background (e.g., continue to run after terminal closed):
	nohup python qsearch_redis.py 8080 > ../logs/log.txt 2>&1 & 

5. Open http://yourhostname.com:8080/static/q_search.html

	Example:
	http://localhost:8080/static/q_search.html
	
	Try a search term, like "SSA"

QSearch Features
================

• Code uses Jsonp and Jquery demonstrating that all interactions are controlled by Javascript and be added
  to a page independently of what that page is being sued for.
• Search is case insenstive.


QSearch API
===========

http://localhost:8080/search/SSA/>json

{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}

http://localhost:8080/search/SSA/>jsonp

callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})

http://localhost:8080/search/json/SSA

{"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}}

http://localhost:8080/search/jsonp/SSA

callback({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})

http://localhost:8080/search/jsonp/SSA?callback=myFunction

myFunction({"acronym": {"definition": "Shared Services Agreement", "name": "SSA", "id": "SSA"}, "html": "<div id=\"qsearch_result\">SSA: Shared Services Agreement</div>", "type": "acronym"})


Alternative Ways of Using and Testing QSearch
=============================================

	A couple other screens from browser:
	http://localhost:8080/lookup/wiat
	http://localhost:8080/search/wiat
	

	command line:
	$ curl http://localhost:8080/search/json/SSA 
	$ curl http://localhost:8080/search/json/KOLO-TV


	command line piped to -mjson.tool for formatted display :
	$ curl http://localhost:8080/search/json/SSA | python -mjson.tool
	$ curl http://localhost:8080/search/json/KOLO-TV | python -mjson.tool

	command line with callback
	
	$ curl http://localhost:8080/search/jsonp/SSA 
	$ curl http://localhost:8080/search/jsonp/SSA?callback=myFunction

	python client to consumer qsearch_redis.py API
	
	$ python search_client.py 8081
	$ curl http://localhost:8081/ssa
	Or open browser and go to: 
	http://localhost:8081/ssa
	http://localhost:8081/wcau


Redis Tips
==========

Launch redis command line (see http://redis.io/commands for all commands):

	$ redis-cli
	redis> 
	redis> get acronym:SSA
	"{\"definition\": \"Shared Services Agreement\", \"name\": \"SSA\", \"id\": \"SSA\"}"
	redis>


QSearch FAQ
===========

Q: How do I get my own data into Redis?
A: Fork and modify an appropriate script from the utils/dataloaders/ subdirectory to create a python loader.
   You can also write whatever scripts you need in whatever language you want to load data
   into Redis. If you add new keys, you will probably need to adjust the qsearch_redis.py script
   to search for the additional keys. 

Q: How should I format my data to put into Redis?
A: The Redis keys follow a pattern of "<class>:<id>", for example, "callsign:WIAT". The value
   associated with a key is formatted to be in a json format. See the scripts in utils/dataloaders
   to see how a delimited text file gets read, converted, and loaded into the Redis data store.

Q: Do I have to use Redis? Can I use memcached or another key-value store?
A: You can use another key-value store but you will need to change the code to do so. The code is simple
and there is not much abstraction. 

Q: How do I use all this on a Microsoft Windows PC?
A: You will need to find and install the Windows versions of the various tools and libraries and 
   modify file paths and code appropriately. It would be great if someone wanted to port all this
   to Windows and share with me the information.

Something went wrong with that request. Please try again.