HTTP log explorer
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
lib
web
.gitattributes
.gitignore
LICENSE
README.md
config-example.js
loghowder.js
package.json

README.md

loghowder

Software to explore data from a HTTP log file in a browser.

Copyright © 2014 Rob Schlüter

Licensed under the BSD 3-clause "New" or "Revised" License. For licensing details see the LICENSE file.

The source code of this project is hosted on GitHub: https://github.com/kwebble/loghowder

You can contact me through the contact form on my website: http://kwebble.com/about

Description

Input for loghowder is an access log file produced by a HTTP server. This file is processed to create a data file that can be viewed in a browser.

I made this application to quickly look at how my website was used and find things like requests for non-existent pages and strange user agents. It is not a complete web analytics package.

Run the software on a copy of a HTTP log file. If something goes wrong you still have the original.

Requirements

loghowder requires node.js and npm to be installed.

The lines of the input HTTP log file must be formatted according to the Combined Log Format.

Installation

  • Download the zip file with the software.
  • Extract the zip file.
  • Open a command prompt and change the current directory to the folder with extracted files.
  • Run the command npm install. This will retrieve Node.js dependencies required by loghowder.

The software is now installed. The next step is to configure it.

The configuration must be stored in a file called config.js. The software includes a sample configuration file called config-example.js. Create a copy of this file or rename it to config.js and edit the options as you require.

Configuration options

The configuration options are:

  • logfile: the name of the HTTP log file to process. The default name is http.log.
  • outputFolder: the name of the folder to store the data generated from the HTTP log. The default is the web sub folder.
  • withDomainNames: a logical value (true or false) that indicates if the host names of IP addresses in the HTTP log must be retrieved. The default is false to reduce processing time.
  • skipHosts: an array of IP addresses and/or host names to exclude from the data. Use this to skip requests you are not interested in, for example requests you made yourself.
  • showBubbles: a logical value (true or false) that determines to draw a bubble on the map to indicate the number of requests, or not. If no bubbles are shown the color of the countries indicate the number of requests.
  • title: the title of the report, shown on the page. For example used to indicate for which domain the report is.
  • hostFormat: defines the format of host addresses in the list of requests. If no name if known for the host that made a request, the IP address is shown. Possible values:
    • full: show the host and domain name. This is the default if no format is specified.
    • ip: show the IP address.
    • domain show the domain name. If the name starts with something that looks like a IP address this complete part is removed. For example, the host 192.0.2.1.america.example.com will be shown as america.example.com.

To change the options, open config.js in a text editor and adjust the options as you prefer. Make sure to keep the structure of the file as expected.

Usage

After the software is installed and configured you should be able to use it. First the HTTP log file must be processed and then you can explore the log data in a browser.

To process the HTTP log file open a command prompt, change to the loghowder installation folder and run the command node loghowder.js. This produces the data for the web page.

After the command has finished open the index.html file in the configured outputFolder with a browser. You can directly open the file, it is not necessary to access it through a web server.

Controlling the graph

The web page has 4 parts, a graph, a map, controls and a data table.

The graph

The graph shows the currently selected requests over time. Clicking a label in the legend toggles the display of that series off and on.

On my personal site you can view some samples of graphs loghowder can show.

The map

The map shows the countries where the currently selected requests come from. The color indicates the number of requests or bytes transferred. The lighter the color the higher the number for that country. Hover over a country to see the exact number.

Controls

With the controls you select data and set viewing options. They are:

  • Quick selection list: pick a predefined selection of a field and its value to select requests.
  • Split selection list: determines on which field to split the selected requests.
  • Unit selection list: sets the unit of the graph, either the number of requests or the amount of transferred bytes.
  • Search box: selects requests by a searching for text. Text is searched in the fields host, uri, referrer and client.
  • Selection: displays the current selection. If a selection on a field and value is active then clicking [-] reverses the selection to exclude all requests with that value for the field. Clicking [X] removes the selection.

Request data

The data table shows the first 1000 HTTP requests of the selected data.

Data in a number of columns are clickable. Clicking a value changes the selection to include only requests with the selected value for the field.

The requests can be sorted on a specific field by clicking the table column field name.

Troubleshooting

The software uses a database to determine the country, based on the IP address of each request. If a country is not found a warning will be shown on the web page.

This situation may be solved by running the command npm run-script updatedb in the node_modules/geoip-lite folder. This will update the database.

Acknowledgements

This software uses:

A big thanks to the developers of these products.