Skip to content

Latest commit

 

History

History
116 lines (93 loc) · 4.2 KB

README.md

File metadata and controls

116 lines (93 loc) · 4.2 KB
Raw

RSSMiner

Features

  • Build for readability
  • Realtime, instant fulltext search
  • Realtime, personalized recommendation based on reading history
  • Google Chrome plugin to add subscription
  • Clean and compact code
  • Super fast (In order to be fast, I write the Web server, Database Connection Pool, Chinese segmentation, Template System, Some JS libs from scratch. I save every bit to make it faster. The server can handle thousands of request per seconds(Including fulltext search) with very low latency.

Technologies

Server-side

  • Java. Clojure.
  • Apache Lucene, use it to do fulltext, realtime, instant search.
  • Ring, compojure, easier http.
  • http-kit, super fast event driven HTTP server and HTTP client. Especially written for Rssminer.
  • dbcp, Simple database connection pool. Especially written for Rssminer
  • mmseg, A java implementation of MMSEG. Especially written for Rssminer
  • MySQL, data store
  • Redis, Message Queue; Per user per feed score Store. Proxy cache.
  • Mustache.clj, Mustache for Clojure

Client-side

How to run it

Install Instructions

  1. Install Dependencies leiningen, JDK7(JDK6 works), Redis, MySQL, sass, rake

  2. clone this repo, install dependency

git clone git://github.com/shenfeng/rssminer.git && cd rssminer && lein deps
  1. Initialize database, create user, import schema
cd rssminer && ./scripts/admin init-db
  1. Run it
# make sure /var/rssminer/index dir exits and is writable. refer command line args for more info
rake run:dev # run server in dev profile, view it: http://127.0.0.1:9090

Run unit test

rake test
rake mysql_dev # replace my.cnf will a dev one, run mysql in /tmp, run it after understand it.

It will create/drop a temp MySQL database for each test. If MySQL's db path is in tmpfs, It will much faster(12s vs 40+s).

Command line args

rake prepare javac && ./scripts/run --help  # rake -T list commands
Usage:

 Switches                 Default                          Desc
 --------                 -------                          ----
 -p, --port               9090                             Port to listen
 --worker                 2                                Http worker thread count
 --fetcher-concurrency    10
 --fetch-size             20                               Bulk fetch size
 --profile                :dev                             dev or prod
 --redis-host             127.0.0.1                        Redis host
 --redis-port             6379                             Redis port
 --static-server          //192.168.1.200                  static server
 --db-url                 jdbc:mysql://localhost/rssminer  MySQL Database url
 --db-user                feng                             MySQL user name
 --db-pass                                                 MySQL password
 --bind-ip                0.0.0.0                          Which ip to bind
 --events-threshold       20                               How many user feed events buffered before recompute again
 --index-path             /var/rssminer/index              Path to store lucene index
 --no-fetcher, --fetcher  false                            Start rss fetcher
 --no-proxy, --proxy      false                            Enable Socks proxy
 --no-help, --help        false                            Print this help

Copyright

Copyright (c) Feng Shenshenedu@gmail.com. All rights reserved.