Permalink
Fetching contributors…
Cannot retrieve contributors at this time
108 lines (70 sloc) 2.32 KB

Logging

Deflect 'Cruncher' Awstats Log Analysis

This is the method Deflect does its Awstats setup for all edges running Apache Traffic Server.

Directory Structure and Files

  • Installed in /var/deflect/
    • etc/
      • edges
      • id_rsa
    • bin/
      • 0_backup_awdata.sh
      • 1_log_collection.sh
      • 2_uz2live.sh
      • 3_resolve_logs.sh
      • 4_crunch_configs.sh
      • add_aw.sh
      • cron_crunch.sh
    • archives/
      • edge1.domain.com/
      • ... /
      • edge24.domain.com/
    • tlive/
      • 201306/
      • 201307/
      • ... /
      • 201401/
    • backup/
      • var_lib_awstats-20140103010001-341825787/
      • etc_awstats-20131201000138-262232287/
    • lock/

The Scripts ( bin/ )

  • 0_backup_awdata.sh
    • This script just does a copy of the config and awstats data files before running anything. For easier recovery if anything goes wrong.
  • 1_log_collection.sh
    • This script reads a list of edges and rsync the ATS log files '-.gz' to a archive
  • 2_uz2live.sh
    • Gets date and looks for zipped file in archive that match the right date. Unzips to a temp location in a separated YYYYMM directory If file was already unzipped in this directory before it does a 'cat /dev/null > old_file'
  • 3_resolve_logs.sh
    • Uses some of awstats tools to merge the unzipped log files together into one file YYYMM/current.log
  • 4_crunch_configs.sh
    • Read all the Awstats configs setup and forks the to all crunch using YYYYMM/current.log
  • cron_crunch.sh
    • Runs all the files above
  • add_aw.sh
    • A simple config creator script to add a client to Awstats
    • you need (domain, config name, password)
    • Also gives a option to manually add REGEX for advanced REGEX

Configuration ( etc/ )

  • edges
    • List of all edges we are collecting log files from.
  • id_rsa
    • Private ssl key so scripts can access (rsync) logs

Locking ( lock/ )

Scripts write files in here to make sure things finish. If a problem occurs, like a edges could not get collected from, you can later just delete the file inside this directory and do a manual run.