You'll need a fair few bits and pieces to make this all work:
- Ruby 2.1 or later;
- PostgreSQL 9.4 or later;
- An AMQP server, such as RabbitMQ;
- A lot of diskspace, if you want to monitor existing logs.
Configu is all done via environment variables (in line with the principles of the 12 Factor App). The following env vars are available:
The URL to an AMQP server (such as RabbitMQ).
The log level of the agent framework (and, as a side-effect, the web application). Set this to, say,
"debug"in order to get more detailed log messages.
A Sequel-compatible URL pointing to a PostgreSQL 9.4 or later database. You include all parameters (username, password, etc) in the URL.
If set, then all sorts of debugging into will be spewed out about the SQL commands being sent to the database. You'll rarely need this, and hopefully only in development.
If set, and RACK_ENV is set to something other than
"development", then all exceptions generated by the webapp will be e-mailed to the address (or addresses, separated by commas) given in this env var.
Specify the Rack-level environment. If left as the default, various debugging-related things are enabled which you almost certainly don't want to have enabled in production.
If set, the agent which calculates Merkle Tree hashes (to compare against the hashes provided by logs in their STHs) will cache intermediate results in the specified Redis datastore. You really, really want this if you're running a real-world monitor. Without it, verifying each tree head of a decent-sized log will take 8-12 hours (and we're retrieving them every minute!). Also, you want to allow at least 4GB of memory, and an
You'll need a few hundred GB of fast disk for the database, a few dozen GB
of fast disk for RabbitMQ, and some way to start and restart the agent
processes. In a pinch, you can just run
foreman, but you'll probably want
to setup something more robust (investigate
Get your env vars configured, then setup the DB schema by running:
Finally, add one or more logs to monitor:
bin/add_log "Log Name" https://ct.example.com base64publickey==
Within a minute or so, an STH should be retrieved from the logs, and entries will start downloading. All the other agents will then spring to life and do their part.
As long as all the agents are running, an STH will be fetched for each log every minute, and any new entries will be retrieved and processed into the database.
Getting information out of the database is up to you. This framework provides no particular facilities for analysing the collected data, as it is assumed that users will have many and varying analysis requirements. You can write additional agents to do further processing on the database contents, or web applications in your choice of language for interactive access.
At present, there is no dedicated mailing list or IRC channel for this software. Try the general certificate-transparency mailing list.
If you need assistance deploying or operating your monitor, please contact me to discuss your requirements.
Bug reports and patches should be sent to the Github issue tracker.
Unless otherwise stated, everything in this repo is covered by the following copyright notice:
Copyright (C) 2015 Tobermory Technology Pty Ltd This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3, as published by the Free Software Foundation. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.