This project contains work from Zeleng Zhuang and Huijuan Zou.
This is a final project for the course Search Engine Architecture
If you want to see how our model works, try the following url:
http://linserv2.cims.nyu.edu:34233/?q=This+is+sample+text
Attention: This may take a while for the web page to load, please wait with patience.
You can replace the text after 'q=' and please separate it by '+' ex:
http://linserv2.cims.nyu.edu:34233/?q=Yen+drops+as+European+and+Japanese+stocks+start+week+higher.+*+U.S.+data+fails+to+provide+clarity+on+Fed+rate-hike+timing.+*+Speculators+cut+U.S.+dollar+longs+to+lowest+in+over+a+year+-CFTC
If you want to set it up, please revise the BASE MODEL, and run
python server.py
given you has installed all the required library and an appropriate python version(It works with python3 but it didn't work with python 3.6. Please use python 3.5)
As the training data is very large, it is not possible for us to provide the training data, but you can find it here:
http://research.signalmedia.co/newsir16/signal-dataset.html.
But we do provide our trained model as checkpoints files.
For detail of the implementation, please refer to our paper here:
https://bitbucket.org/ZelengZhuang/search-engine-architecture-final-project/src/340051975247488b5d1dc53420909e749ec7540a/paper.pdf?at=master&fileviewer=file-view-default
We use GloVe which can be found here:
https://nlp.stanford.edu/projects/glove/
We use tornado as our backend server. For training we use Tensorflow and Keras. Also, we use popular data science such like nltk and numpy, etc
For word2vec part: python reformatter_json.py data/train.jsonl --job_path w2v_jobs --num_partitions 5 (or python reformatter_xml.py data/info_ret.xml --job_path w2v_jobs --num_partitions 5) python -m gradient python -m init_model --app apps.word2vec --job_path w2v_jobs python -m update_model --app apps.word2vec --job_path w2v_jobs --iterations 50
run accuracy file: python test_accuracy.py
run model file at w2v_jobs/0.out to use the model