Simple API Call Counter

The Simple API Call Counter is a program designed to count the occurrences of URLs in text files such as logs. It allows users to determine how many times each URL appears in the provided files. This tool, especially useful for files like nginx logs, provides insights into the frequency of API calls. Use it to quickly analyze and understand how often each API is invoked.

Environment

1. Python 3.11.x

It is recommended to set with pyenv.
If you can configure the Python execution environment in another way, you can skip the pyenv guide below.

pyenv

$ pyenv install 3.11.2
$ pyenv virtualenv 3.11.2 api_counter
$ pyenv local api_counter

Install

$ pip install api-counter

Run

$ counter [FILE]  # Quick start with a single file.

$ counter --help  # Show me the help.

$ counter -d . -r -f  # Counts urls from all (json and log) files under the current folder and 
                      # creates a result file (however, json files starting with result_ are not included!)

Example of Using Options (Refer to the Help)

$ counter 000.json  # Display the count result from 000.json on the screen

$ counter 000.json -f  # Display the count result from 000.json on the screen and create a result_{date_time}.json file

$ counter 000.json 111.json 222.json ...  # Count multiple json files and display the results on the screen

$ counter 000.json 111.json 222.json ...  -f  # Count multiple json files, display the results on the screen, and create a result file

$ counter -d ./0000  # Count all json files in the 0000 folder and display the result (subfolders are not processed)

$ counter -d ./0000  -f  # Count all json files in the 0000 folder, display the result, and create files (subfolders are not included)

$ counter -d -r -f ./0000  # Count all json files in the 0000 folder, display the result, and create files (including subfolders)

Test (unittest with pytest)

$ pytest -svx -rsa tests

Result File Format

Format: result_{date_time(%Y%m%d_%H%M%S)}.json

{
    "targets": [ 
        list of counted files 
    ],
    "result": [
        "Counted URL: 1st place in frequency",
        "Counted URL: Sorted in descending order of frequency"
    ]
}

Sample: result_20231209_201928.json

{
    "targets": [
        "tests/samples/nginx.log"
    ],
    "result": [
        "GET /api/docs: 18",
        "GET /api: 17",
        "POST /api/v1/social/signup: 4",
        "DELETE /api/v1/users/95: 2",
        "PUT /api/v1/users/me: 1",
        "POST /api/v1/social/profile/format: 1",
        "POST /api/v1/social/profile: 1"
    ]
}

One More Thing (About the ignore option)

If you want to specify lines to exclude from the counter. (e.g. request from a specific IP address, etc.)
You can set export ignores=1.1.1.1 or export ignores=1.1.1.1,2.2.2.2 (no space allowed).

$ export ignores=1.1.1.1

or for multiple ignores.
$ export ignores=1.1.1.1,2.2.2.2

And surprisingly, this option can work with any string. (e.g. '/api/token')

$ export ignores='/api/token'

or for multiple ignores.
$ export ignores='/api/token',2.2.2.2