Skip to content
main
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
cmd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

CT Map/Reduce Tooling

Installation

  1. Install the python dependencies: pip install -r python/requirements.txt
  2. Build the CT-to-Disk scraper: go get github.com/jcjones/ct-mapreduce/cmd/ct-fetch

Configuration

  1. Create a configuration file:
cat > ~/.ct-fetch.conf <<EOF
# Available directives:
#
# certPath = Path under which to store full DER-encoded certificates
# issuerCNFilter = Prefixes to match for CNs for permitted issuers, comma delimited
# runForever = Run forever, pausing `pollingDelay` between runs
# pollingDelayMean = Mean polling delay duration
# pollingDelayStdDev = A standard deviation, like 100, or 1000.
# logExpiredEntries = Add expired entries to the database
# numThreads = Use this many threads per CPU
# logList = URLs of the CT Logs, comma delimited
# cacheSize = Size of internal cache in entries, default is probably fine
#
# Examples
#
# Only accept certificates for Let's Encrypt's intermediates or the ISRG roots
issuerCNFilter = Let's Encrypt, ISRG
# Update the CT log list as you like, comma-delimited. Not currently tested with more than one log.
logList = https://ct.googleapis.com/icarus
# Choose if this should complete when it catches up to the CT logs, or be a daemon
runForever = false

# Optionally, a path with plenty of disk space if you want to save PEM files
# certPath = /ct

# Redis cache server
redisHost = 10.10.10.5:6379
redisTimeout = 2s

EOF

Redis

You'll also need to configure your Redis instance with maxmemory_policy:noeviction, which is checked programmatically and a warning will go to the logs if not set correctly.

IAM for Google Cloud

Permissions that seem necessary are:

Cloud Datastore Owner
Cloud Datastore User
Monitoring Metric Writer
Cloud Memorystore Redis Editor
Storage Admin
Storage Object Admin

Populating your storage and Redis with CT certificates

ct-fetch -config ~/.ct-fetch.conf

Note: Consider using --offset X to start from the Xth log entry. Also, --limit Y will stop after processing Y certificates.

Tests

my_ip=$(ipconfig getifaddr en0) # macOS
docker run redis:4
# for some reason `docker run -p 6379:7000 redis:4 --port 7000` is needed for me

RedisHost=${my_ip}:6379 go test -v ./...

About

Map/Reduce functions for processing Certificate Transparency. Used for https://LetsEncrypt.org/stats

Resources

Packages

No packages published

Languages

You can’t perform that action at this time.