Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Copious.com's personalized feed building system
Clojure Ruby Shell Java
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
actions-solr @ 143e40a
bin
client
config/deploy
etc
java-src/risingtide
resources
src/risingtide
test/risingtide
.gitignore
.gitmodules
.rvmrc
Capfile
Gemfile
Gemfile.lock
LICENSE
Procfile
Procfile.test
README.md
design.md
project.clj
testing.md

README.md

Rising Tide

Rising Tide created customized feeds for users of Copious.com, an online social marketplace active from 2011 to 2013.

Note that project setup instructions are unlikely to work as-is: the source is being made available to the community for archaeological and analytic purposes.

Project Setup

git clone git@github.com:utahstreetlabs/risingtide.git
cd risingtide
git submodule init
git submodule update
cd actions-solr && make
gem install bundler
bundle install
foreman start

Tests

bin/run-test-solr
lein midje

Common maintenance tasks

Get a production REPL

ec2-ssh $USERNAME@rt-storm1.copious.com
sudo su utah
cd ~/risingtide/current
RT_ENV=production bin/lein repl

Run a report

ec2-ssh $USERNAME@rt-storm1.copious.com
sudo su utah
cd ~/risingtide/current
RT_ENV=production bin/lein run -m risingtide.reports/report

Get a JMX term to examine metrics

RT_WORKER=2 #
ec2-ssh $USERNAME@rt-storm$RT_WORKER.copious.com
RT_EXECUTOR=0 # currently 0 or 1
java -jar /opt/jmxterm/jmxterm-1.0-alpha-4-uber.jar -l service:jmx:rmi:///jndi/rmi://localhost:1670$RT_EXECUTOR/jmxrmi -n
get -b default:name=seller-follow-interest-score-time,type=default OneMinuteRate

Clear out old actions from solr

ec2-ssh $USERNAME@rt-storm1.copious.com sudo su utah cd ~/risingtide/current RT_ENV=production bin/lein run -m risingtide.utils/flush-old-actions-from-solr

Metrics

A raw list of metrics that we should compile into a more useful format:

(deftimer active-user-fetch-time)
(deftimer active-user-fanout-time)
(deftimer feed-load-time)
(deftimer expiration-time)
(deftimer feed-write-time)
(deftimer like-interest-score-time)
(deftimer follow-interest-score-time)
(deftimer seller-follow-interest-score-time)
(deftimer find-recent-actions-time)
(deftimer feed-build-time)

(defmeter action-saved "actions saved")
(defmeter action-created "actions created")
(defmeter action-processed "actions processed")
(defmeter action-processing-failed "actions processing attempts failed")
(defmeter expiration-run "expiration runs")
(defmeter feed-writes "feeds written")
(defmeter story-scored "stories scored")
(defmeter feed-builds "feeds built")
(defmeter curated-feed-writes "stories written to curated feed")

(gauge "active-user-count" (active-users redii))]
(gauge "feed-set-size" (count @feed-set))
(gauge "feed-set-feed-min-size" (apply max (feed-set-feed-sizes @feed-set)))
(gauge "feed-set-feed-max-size" (apply min (feed-set-feed-sizes @feed-set)))
(gauge "feed-set-feed-mean-size" (apply mean (feed-set-feed-sizes @feed-set)))
(gauge "feed-set-feed-median-size" (apply median (feed-set-feed-sizes @feed-set)))
(gauge "interest-reducer-size" (count @scores))]
(gauge "curated-feed-size" (count (seq @feed-atom)))

(defhistogram recent-actions-found)

Scout monitoring

Scout monitoring is configured with custom plugins. To deploy them to a new machine, run

bin/copy-scout-plugins machine-name

Where machine-name is the subdomain of the host in question. So if you'd like to deploy scout plugins to rt-storm2.copious.com run

bin/copy-scout-plugins rt-storm2

HeapMemoryUsage,NonHeapMemoryUsage@java.lang:type=Memory

License

Copyright © 2014 Utah Street Labs

Something went wrong with that request. Please try again.