Skip to content

Commit

Permalink
feat: Babel removal, replace Mocha with Jest, break down request field (
Browse files Browse the repository at this point in the history
#28)

* bye babel

* migrated tests to jest and typescript

* improve pr checkers

* revert pr checker changes

* separated the URL field into subsections

* updated docs and types
  • Loading branch information
ozantunca committed Oct 14, 2019
1 parent d5f3232 commit c357fc2
Show file tree
Hide file tree
Showing 17 changed files with 6,074 additions and 5,601 deletions.
13 changes: 0 additions & 13 deletions .babelrc

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/pr-checkers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ jobs:
run: |
npm install
npm run build --if-present
npm test
npm test
10 changes: 7 additions & 3 deletions .prettierrc
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
singleQuote: true
printWidth: 100
trailingComma: "none"
{
"trailingComma": "es5",
"tabWidth": 2,
"semi": false,
"singleQuote": true,
"printWidth": 100
}
61 changes: 59 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Log Analyzer for AWS Elastic Load Balancer

[![npm version](https://badge.fury.io/js/elb-log-analyzer.svg)](https://www.npmjs.org/elb-log-analyzer)
[![Travis](https://travis-ci.org/ozantunca/elb-log-analyzer.svg?branch=master)](https://travis-ci.org/ozantunca/elb-log-analyzer)
[![Snyk](https://snyk.io/test/npm/elb-log-analyzer/badge.svg)](https://snyk.io/test/npm/elb-log-analyzer)
Expand All @@ -8,37 +9,45 @@ ELB log analyzer is a command line tool for parsing Elastic Load Balancer's acce
<a href="https://www.buymeacoffee.com/7U6J1fT" target="_blank"><img src="https://bmc-cdn.nyc3.digitaloceanspaces.com/BMC-button-images/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: auto !important;width: auto !important;" ></a> If you like using the tool, consider buying me a coffee ☕️

## Installation

```sh
npm install -g elb-log-analyzer
```

## Usage

Log analyzer receives input as directories or files. It reads those log files and returns a table-like two column data set.

Assuming we have a directory structure like below...

```
.
└── logs/
├── access-log1.txt
├── access-log2.txt
├── access-log3.txt
```

You can specify a log file to be analyzed like this:

```sh
elb-log-analyzer logs/access-log1.txt
```

or you can specify several of them:

```sh
elb-log-analyzer logs/access-log1.txt logs/access-log2.txt
```

or you can simply specify the **directory**:

```sh
elb-log-analyzer logs/
```

By default log analyzer will count all requests and sort them in descending order so that most requested URLs will be listed. This functionality can be changed but this one was chosen as default behaviour since it appears to be the most common case. Example output:
By default log analyzer will count all requests and sort them in descending order so that most requested URLs will be listed. This functionality can be changed but this one was chosen as default behaviour since it appears to be the most common case. Example output:

```sh
1 - 930: http://example.com:80/img/blabla.jpg
2 - 827: http://example.com:80/images/trans.png
Expand All @@ -51,7 +60,27 @@ By default log analyzer will count all requests and sort them in descending orde
9 - 341: http://example.com:80/fonts/font-awesome-4.0.3/fontawesome-webfont.woff?v=4.0.3
10 - 327: http://www.example.com:80/favicon.ico
```
Values in columns can be set to any of the values in logs files which can be seen here http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/access-log-collection.html#access-log-entry-format. There are total of 3 extra fields added to these which are `count`, `total_time` and `requested_resource`, `client`, `backend`.

Values in columns can be set to any of the values in logs files which can be seen here http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/access-log-collection.html#access-log-entry-format. There are total of 17 extra fields added to these which are:

- `count`
- `total_time`
- `requested_resource`
- `client`
- `backend`
- `requested_resource.pathname`
- `requested_resource.host`
- `requested_resource.protocol`
- `requested_resource.port`
- `requested_resource.hostname`
- `requested_resource.path`
- `requested_resource.origin`
- `requested_resource.search`
- `requested_resource.href`
- `requested_resource.hash`
- `requested_resource.searchParams`
- `requested_resource.username`
- `requested_resource.password`

When `count` is specified, it serves as a **groupBy** method that counts values in the other column and groups them together. Note that column1 is `count` by default.

Expand All @@ -64,10 +93,13 @@ When `count` is specified, it serves as a **groupBy** method that counts values
`backend` is the backend IP. ELB provides IP and ports as pairs. This field added to filter by only IP instead of the pair.

Columns can be changed like this:

```sh
elb-log-analyzer logs/ --col2=client:port
```

This command shows client IPs that make requests the most. Example output:

```
1 - 258: 54.239.167.77:6176
2 - 246: 54.239.167.77:48034
Expand All @@ -82,10 +114,13 @@ This command shows client IPs that make requests the most. Example output:
```

Another example command below gets clients that make requests which take the longest time in total.

```sh
elb-log-analyzer logs/ --col1=total_time --col2=client:port
```

Example output:

```
1 - 3.153657: 188.57.145.98:11668
2 - 2.5415739999999998: 85.103.48.224:59350
Expand All @@ -100,11 +135,15 @@ Example output:
```

#### Adding More Columns

Keep defining columns with `--col*` pattern such as `--col3`, `--col4`, `--col5`.

```sh
elb-log-analyzer logs/ --col1=count --col2=client:port --col3=elb_status_code
```

Example output:

```
1 - 188 - 54.239.167.83:11419 - 200
2 - 180 - 54.239.167.83:3785 - 200
Expand All @@ -119,18 +158,23 @@ Example output:
```

#### SortBy

You can select the column you want the results to be sorted by. Use `--sortBy` argument and specify the column number.

```sh
elb-log-analyzer logs/ --col1=count --col2=client:port --col3=elb_status_code --sortBy=2
```

#### Filtering

##### Filter by prefix

A string can be provided to get values that starts with given string. This can be done using `--prefix1` and/or `--prefix2` options depending the column that needs to be queried. For example this feature can be used to get number of resources requested starting with certain URL. The command that performs this action would be similar to the one below:

```sh
elb-log-analyzer logs/ --col1=count --col2=requested_resource --prefix2=http://example.com:80/article
```

Example output:

```
Expand All @@ -147,12 +191,15 @@ Example output:
```

##### Filter by date

You can specify any valid JavaScript date that `new Date()` successfully processes. Be aware that it does not accept timestamp numbers currently. This is a design choice to enable users to specify only a year, not the whole date string. Example usage is below.

```sh
elb-log-analyzer logs/ --start=2015-11-07T18:45:34.501734Z --end=2015-11-07T18:45:34.768481Z
```

Example output:

```
1 - 4 - http://example.com:80/images/logo/example-o-logo.png
2 - 4 - http://example.com:80/images/logo/google-play.png
Expand All @@ -165,7 +212,9 @@ Example output:
9 - 2 - http://cf-source.example.com:80/img/600/300/2r0/502a456a2ab3d1d03300af9a.jpg
10 - 2 - http://cf-source.example.com:80/img/600/300/2r0/55f94dedf5ef747e16a4a640.jpg
```

The usages below are also acceptable.

```sh
elb-log-analyzer logs/ --start=2016
elb-log-analyzer logs/ --start=2016-05-30
Expand All @@ -175,24 +224,32 @@ elb-log-analyzer logs/ --end=2015-11-07T18:45:34.768481Z
```

#### Limiting

By default analyzer brings first 10 rows but this can be changed using `--limit` option. For instance to be able to get 25 rows `--limit=25` should be specifiied.

#### Ascending Order

Analyzer's default behaviour is to bring results in descending order. If ascending order needed, you simply specify `-a` option.

#### Version

`--version` or `-v` option returns the version of `elb-log-analyzer`.

```sh
elb-log-analyzer -v
```

Example Output:

```
v0.3.0
```

#### Roadmap

- Will be usable as a library in addition to CLI usage
- CLI will run multiple clusters to speed up the process and escape from memory limitations

#### If you like what you see, consider buying me a coffee ☕️

<a href="https://www.buymeacoffee.com/7U6J1fT" target="_blank"><img src="https://bmc-cdn.nyc3.digitaloceanspaces.com/BMC-button-images/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: auto !important;width: auto !important;" ></a>
Loading

0 comments on commit c357fc2

Please sign in to comment.