An online rss reader written in clojure & javascript & java.
JavaScript Java Clojure Shell Ruby Go Other
Permalink
Failed to load latest commit information.
android redirect to mobile if it's; warn before exits Nov 21, 2012
chrome
conf
feedb
public using latest http-kit: support https Apr 26, 2013
scripts fix a fetcher infinite loop: server may return 302 with same location Apr 27, 2013
scss using latest http-kit: support https Apr 26, 2013
src add crc32 check; better test; add transfter code May 12, 2013
templates do not change left nav bar; comment out rcmd logic Mar 28, 2013
test add crc32 check; better test; add transfter code May 12, 2013
.gitignore a custom feed article storage to save disk space May 12, 2013
README.md update readme Mar 16, 2013
Rakefile local prof profile with local favicon server Mar 28, 2013
TODO.org fix many bugs about scrolling Sep 12, 2012
project.clj add crc32 check; better test; add transfter code May 12, 2013

README.md

RSSMiner

Features

  • Build for readability
  • Realtime, instant fulltext search
  • Realtime, personalized recommendation based on reading history
  • Google Chrome plugin to add subscription
  • Clean and compact code
  • Super fast (In order to be fast, I write the Web server, Database Connection Pool, Chinese segmentation, Template System, Some JS libs from scratch. I save every bit to make it faster. The server can handle thousands of request per seconds(Including fulltext search) with very low latency.

Technologies

Server-side

  • Java. Clojure.
  • Apache Lucene, use it to do fulltext, realtime, instant search.
  • Ring, compojure, easier http.
  • http-kit, super fast event driven HTTP server and HTTP client. Especially written for Rssminer.
  • dbcp, Simple database connection pool. Especially written for Rssminer
  • mmseg, A java implementation of MMSEG. Especially written for Rssminer
  • MySQL, data store
  • Redis, Message Queue; Per user per feed score Store. Proxy cache.
  • Mustache.clj, Mustache for Clojure

Client-side

How to run it

Install Instructions

  1. Install Dependencies leiningen, JDK7(JDK6 works), Redis, MySQL, sass, rake

  2. clone this repo, install dependency

git clone git://github.com/shenfeng/rssminer.git && cd rssminer && lein deps
  1. Initialize database, create user, import schema
cd rssminer && ./scripts/admin init-db
  1. Run it
# make sure /var/rssminer/index dir exits and is writable. refer command line args for more info
rake run:dev # run server in dev profile, view it: http://127.0.0.1:9090

Run unit test

rake test
rake mysql_dev # replace my.cnf will a dev one, run mysql in /tmp, run it after understand it.

It will create/drop a temp MySQL database for each test. If MySQL's db path is in tmpfs, It will much faster(12s vs 40+s).

Command line args

rake prepare javac && ./scripts/run --help  # rake -T list commands
Usage:

 Switches                 Default                          Desc
 --------                 -------                          ----
 -p, --port               9090                             Port to listen
 --worker                 2                                Http worker thread count
 --fetcher-concurrency    10
 --fetch-size             20                               Bulk fetch size
 --profile                :dev                             dev or prod
 --redis-host             127.0.0.1                        Redis host
 --redis-port             6379                             Redis port
 --static-server          //192.168.1.200                  static server
 --db-url                 jdbc:mysql://localhost/rssminer  MySQL Database url
 --db-user                feng                             MySQL user name
 --db-pass                                                 MySQL password
 --bind-ip                0.0.0.0                          Which ip to bind
 --events-threshold       20                               How many user feed events buffered before recompute again
 --index-path             /var/rssminer/index              Path to store lucene index
 --no-fetcher, --fetcher  false                            Start rss fetcher
 --no-proxy, --proxy      false                            Enable Socks proxy
 --no-help, --help        false                            Print this help

Copyright

Copyright (c) Feng Shenshenedu@gmail.com. All rights reserved.