Software to explore data from a HTTP log file in a browser.
Copyright © 2014 Rob Schlüter
Licensed under the BSD 3-clause "New" or "Revised" License. For licensing details see the LICENSE file.
The source code of this project is hosted on GitHub: https://github.com/kwebble/loghowder
You can contact me through the contact form on my website: http://kwebble.com/about
Input for loghowder is an access log file produced by a HTTP server. This file is processed to create a data file that can be viewed in a browser.
I made this application to quickly look at how my website was used and find things like requests for non-existent pages and strange user agents. It is not a complete web analytics package.
Run the software on a copy of a HTTP log file. If something goes wrong you still have the original.
The lines of the input HTTP log file must be formatted according to the Combined Log Format.
- Download the zip file with the software.
- Extract the zip file.
- Open a command prompt and change the current directory to the folder with extracted files.
- Run the command
npm install. This will retrieve Node.js dependencies required by loghowder.
The software is now installed. The next step is to configure it.
The configuration must be stored in a file called
config.js. The software includes a sample configuration file called
config-example.js. Create a copy of this file or rename it to
config.js and edit the options as you require.
The configuration options are:
- logfile: the name of the HTTP log file to process. The default name is http.log.
- outputFolder: the name of the folder to store the data generated from the HTTP log. The default is the web sub folder.
- withDomainNames: a logical value (
false) that indicates if the host names of IP addresses in the HTTP log must be retrieved. The default is
falseto reduce processing time.
- skipHosts: an array of IP addresses and/or host names to exclude from the data. Use this to skip requests you are not interested in, for example requests you made yourself.
- showBubbles: a logical value (
false) that determines to draw a bubble on the map to indicate the number of requests, or not. If no bubbles are shown the color of the countries indicate the number of requests.
- title: the title of the report, shown on the page. For example used to indicate for which domain the report is.
- hostFormat: defines the format of host addresses in the list of requests. If no name if known for the host that
made a request, the IP address is shown.
- full: show the host and domain name. This is the default if no format is specified.
- ip: show the IP address.
- domain show the domain name. If the name starts with something that looks like a IP address this complete part is removed. For example, the host 192.0.2.1.america.example.com will be shown as america.example.com.
To change the options, open
config.js in a text editor and adjust the options as you prefer. Make sure to keep the
structure of the file as expected.
After the software is installed and configured you should be able to use it. First the HTTP log file must be processed and then you can explore the log data in a browser.
To process the HTTP log file open a command prompt, change to the loghowder installation folder and run the command
node loghowder.js. This produces the data for the web page.
After the command has finished open the
index.html file in the configured outputFolder with a browser. You can
directly open the file, it is not necessary to access it through a web server.
Controlling the graph
The web page has 4 parts, a graph, a map, controls and a data table.
The graph shows the currently selected requests over time. Clicking a label in the legend toggles the display of that series off and on.
On my personal site you can view some samples of graphs loghowder can show.
The map shows the countries where the currently selected requests come from. The color indicates the number of requests or bytes transferred. The lighter the color the higher the number for that country. Hover over a country to see the exact number.
With the controls you select data and set viewing options. They are:
- Quick selection list: pick a predefined selection of a field and its value to select requests.
- Split selection list: determines on which field to split the selected requests.
- Unit selection list: sets the unit of the graph, either the number of requests or the amount of transferred bytes.
- Search box: selects requests by a searching for text. Text is searched in the fields host, uri, referrer and client.
- Selection: displays the current selection. If a selection on a field and value is active then clicking [-] reverses the selection to exclude all requests with that value for the field. Clicking [X] removes the selection.
The data table shows the first 1000 HTTP requests of the selected data.
Data in a number of columns are clickable. Clicking a value changes the selection to include only requests with the selected value for the field.
The requests can be sorted on a specific field by clicking the table column field name.
The software uses a database to determine the country, based on the IP address of each request. If a country is not found a warning will be shown on the web page.
This situation may be solved by running the command
npm run-script updatedb in the
This will update the database.
This software uses:
- Node.js, npm and jQuery.
- Flot to create the graph.
- datamaps to draw the maps.
- sorttable for sorting the data tables.
- geoip-lite to determine the countries where the requests come from.
- Browser detecting uses data provided by the Browser Capabilities Project.
A big thanks to the developers of these products.