Tools for parsing Centmin Mod Nginx JSON log format based logging as per https://community.centminmod.com/threads/how-to-configure-nginx-for-json-based-access-logging.19641/
The bot-rate.py script will parse Centmin Mod Nginx JSON log fields for specific case-sensitive keyword i.e. botname and calculate the request rate for per second, minute, hour and day.
bot-rate.py by default is set to support Python 2.7 with script using
#!/usr/bin/env python2.7
If you're using Python 3, you can modify script to use
#!/usr/bin/env python3
bot-rate.py supports gzip and zstd compressed JSON logs too.
pip install zstandard
./bot-rate.py /home/nginx/domains/domain.com/log/access_log.json.gz botname
./bot-rate.py /home/nginx/domains/domain.com/log/access_log.json.zst botname
./bot-rate.py
Usage: ./bot-rate.py <path_to_log_file> [keyword] [field_name]
./bot-rate.py /home/nginx/domains/domain.com/log/access_log.json botname
Example filtering for nmap requests in /home/nginx/domains/domain.com/log/access.json Nginx JSON log.
./bot-rate.py /home/nginx/domains/domain.com/log/access.json nmap
Requests with keyword 'nmap':
Requests per second:
2023-09-30T22:16:45 (2) 2023-09-30T22:16:46 (2) 2023-09-30T23:59:59 (2)
2023-10-01T00:00:00 (2) 2023-10-01T06:18:43 (1) 2023-10-01T06:18:45 (1)
2023-10-01T10:43:56 (1) 2023-10-01T10:43:57 (2) 2023-10-01T12:09:02 (2)
2023-10-01T12:09:03 (2) 2023-10-01T16:33:04 (2) 2023-10-01T16:33:05 (2)
2023-10-01T18:12:36 (4) 2023-10-01T23:07:46 (4) 2023-10-01T23:14:57 (4)
2023-10-02T13:15:02 (3) 2023-10-02T13:15:03 (1) 2023-10-02T14:12:44 (2)
2023-10-02T14:12:45 (2) 2023-10-02T19:24:12 (2) 2023-10-02T19:24:13 (2)
2023-10-02T19:24:17 (3) 2023-10-02T19:24:18 (1) 2023-10-03T01:06:57 (1)
2023-10-03T01:06:58 (2) 2023-10-03T01:06:59 (1) 2023-10-03T02:04:14 (1)
2023-10-03T02:04:15 (3) 2023-10-03T12:49:30 (1) 2023-10-03T12:49:31 (3)
2023-10-03T19:52:17 (3) 2023-10-03T19:52:18 (1) 2023-10-03T19:56:34 (4)
2023-10-03T22:00:12 (4) 2023-10-03T22:14:23 (2) 2023-10-03T22:14:24 (1)
2023-10-03T22:14:25 (1) 2023-10-04T05:43:46 (2) 2023-10-04T05:43:47 (2)
2023-10-04T17:20:10 (2) 2023-10-04T17:20:11 (2) 2023-10-04T19:41:31 (4)
2023-10-04T21:18:55 (1) 2023-10-04T21:18:56 (3) 2023-10-04T21:26:16 (4)
2023-10-04T21:31:35 (4) 2023-10-05T06:19:56 (2) 2023-10-05T06:19:57 (2)
2023-10-05T18:10:58 (4) 2023-10-06T00:01:32 (4) 2023-10-06T00:02:28 (1)
2023-10-06T00:02:29 (3) 2023-10-06T00:03:54 (2) 2023-10-06T00:03:55 (2)
2023-10-06T00:09:39 (2) 2023-10-06T00:09:40 (2) 2023-10-06T09:46:45 (4)
2023-10-06T18:26:09 (2) 2023-10-06T18:26:10 (2) 2023-10-06T23:32:02 (3)
2023-10-06T23:32:03 (1) 2023-10-06T23:36:28 (2) 2023-10-06T23:36:29 (2)
2023-10-07T03:46:15 (1) 2023-10-07T03:46:16 (3) 2023-10-07T03:51:05 (1)
2023-10-07T03:51:06 (1) 2023-10-07T03:51:07 (2) 2023-10-07T12:04:00 (4)
2023-10-08T00:33:49 (2) 2023-10-08T00:33:50 (2) 2023-10-08T02:57:46 (3)
2023-10-08T02:57:47 (1) 2023-10-08T03:00:50 (3) 2023-10-08T03:00:51 (1)
2023-10-08T09:06:37 (4) 2023-10-08T09:07:29 (2) 2023-10-08T09:07:30 (2)
2023-10-08T13:25:04 (1) 2023-10-08T13:25:05 (3) 2023-10-09T01:29:29 (4)
2023-10-09T06:58:03 (4) 2023-10-09T08:02:11 (4) 2023-10-09T08:06:37 (2)
2023-10-09T08:06:38 (2) 2023-10-09T08:07:03 (1) 2023-10-09T08:07:04 (3)
2023-10-09T14:36:06 (2) 2023-10-09T14:36:07 (2) 2023-10-09T20:36:25 (1)
2023-10-10T00:30:12 (4) 2023-10-10T04:08:13 (4) 2023-10-10T09:28:15 (1)
2023-10-10T09:28:16 (3) 2023-10-10T10:42:52 (4) 2023-10-10T10:47:06 (3)
2023-10-10T10:47:07 (1) 2023-10-10T13:18:19 (2) 2023-10-10T13:18:20 (2)
2023-10-11T00:15:12 (2) 2023-10-11T00:15:13 (2) 2023-10-11T03:49:26 (1)
2023-10-11T03:49:27 (3) 2023-10-11T10:03:11 (4) 2023-10-11T12:59:05 (1)
2023-10-11T19:53:34 (1) 2023-10-11T19:53:35 (3) 2023-10-11T19:58:21 (4)
2023-10-11T20:11:13 (4) 2023-10-11T20:16:58 (4)
Requests per minute:
2023-09-30T22:16 (4) 2023-09-30T23:59 (2) 2023-10-01T00:00 (2)
2023-10-01T06:18 (2) 2023-10-01T10:43 (3) 2023-10-01T12:09 (4)
2023-10-01T16:33 (4) 2023-10-01T18:12 (4) 2023-10-01T23:07 (4)
2023-10-01T23:14 (4) 2023-10-02T13:15 (4) 2023-10-02T14:12 (4)
2023-10-02T19:24 (8) 2023-10-03T01:06 (4) 2023-10-03T02:04 (4)
2023-10-03T12:49 (4) 2023-10-03T19:52 (4) 2023-10-03T19:56 (4)
2023-10-03T22:00 (4) 2023-10-03T22:14 (4) 2023-10-04T05:43 (4)
2023-10-04T17:20 (4) 2023-10-04T19:41 (4) 2023-10-04T21:18 (4)
2023-10-04T21:26 (4) 2023-10-04T21:31 (4) 2023-10-05T06:19 (4)
2023-10-05T18:10 (4) 2023-10-06T00:01 (4) 2023-10-06T00:02 (4)
2023-10-06T00:03 (4) 2023-10-06T00:09 (4) 2023-10-06T09:46 (4)
2023-10-06T18:26 (4) 2023-10-06T23:32 (4) 2023-10-06T23:36 (4)
2023-10-07T03:46 (4) 2023-10-07T03:51 (4) 2023-10-07T12:04 (4)
2023-10-08T00:33 (4) 2023-10-08T02:57 (4) 2023-10-08T03:00 (4)
2023-10-08T09:06 (4) 2023-10-08T09:07 (4) 2023-10-08T13:25 (4)
2023-10-09T01:29 (4) 2023-10-09T06:58 (4) 2023-10-09T08:02 (4)
2023-10-09T08:06 (4) 2023-10-09T08:07 (4) 2023-10-09T14:36 (4)
2023-10-09T20:36 (1) 2023-10-10T00:30 (4) 2023-10-10T04:08 (4)
2023-10-10T09:28 (4) 2023-10-10T10:42 (4) 2023-10-10T10:47 (4)
2023-10-10T13:18 (4) 2023-10-11T00:15 (4) 2023-10-11T03:49 (4)
2023-10-11T10:03 (4) 2023-10-11T12:59 (1) 2023-10-11T19:53 (4)
2023-10-11T19:58 (4) 2023-10-11T20:11 (4) 2023-10-11T20:16 (4)
Requests per hour:
2023-09-30T22 (4) 2023-09-30T23 (2) 2023-10-01T00 (2)
2023-10-01T06 (2) 2023-10-01T10 (3) 2023-10-01T12 (4)
2023-10-01T16 (4) 2023-10-01T18 (4) 2023-10-01T23 (8)
2023-10-02T13 (4) 2023-10-02T14 (4) 2023-10-02T19 (8)
2023-10-03T01 (4) 2023-10-03T02 (4) 2023-10-03T12 (4)
2023-10-03T19 (8) 2023-10-03T22 (8) 2023-10-04T05 (4)
2023-10-04T17 (4) 2023-10-04T19 (4) 2023-10-04T21 (12)
2023-10-05T06 (4) 2023-10-05T18 (4) 2023-10-06T00 (16)
2023-10-06T09 (4) 2023-10-06T18 (4) 2023-10-06T23 (8)
2023-10-07T03 (8) 2023-10-07T12 (4) 2023-10-08T00 (4)
2023-10-08T02 (4) 2023-10-08T03 (4) 2023-10-08T09 (8)
2023-10-08T13 (4) 2023-10-09T01 (4) 2023-10-09T06 (4)
2023-10-09T08 (12) 2023-10-09T14 (4) 2023-10-09T20 (1)
2023-10-10T00 (4) 2023-10-10T04 (4) 2023-10-10T09 (4)
2023-10-10T10 (8) 2023-10-10T13 (4) 2023-10-11T00 (4)
2023-10-11T03 (4) 2023-10-11T10 (4) 2023-10-11T12 (1)
2023-10-11T19 (8) 2023-10-11T20 (8)
Requests per day:
2023-09-30 (6) 2023-10-01 (27) 2023-10-02 (16)
2023-10-03 (28) 2023-10-04 (24) 2023-10-05 (8)
2023-10-06 (32) 2023-10-07 (12) 2023-10-08 (24)
2023-10-09 (25) 2023-10-10 (24) 2023-10-11 (29)
Example filtering for Mozilla requests in /home/nginx/domains/domain.com/log/access.json Nginx JSON log.
./bot-rate.py /home/nginx/domains/domain.com/log/access.json Mozilla
Requests with keyword 'Mozilla':
Requests per second:
2023-09-30T21:54:53 (1) 2023-09-30T22:02:05 (1) 2023-09-30T22:16:45 (2)
2023-09-30T22:16:46 (2) 2023-09-30T23:29:31 (2) 2023-09-30T23:55:04 (2)
2023-09-30T23:59:59 (2) 2023-10-01T00:00:00 (2) 2023-10-01T00:53:41 (1)
2023-10-01T01:09:39 (1) 2023-10-01T04:12:17 (1) 2023-10-01T04:18:18 (1)
2023-10-01T06:18:43 (6) 2023-10-01T06:18:44 (6) 2023-10-01T06:18:45 (4)
2023-10-01T06:18:46 (4) 2023-10-01T06:18:47 (2) 2023-10-01T06:18:48 (2)
2023-10-01T06:18:49 (2) 2023-10-01T06:18:50 (1) 2023-10-01T06:18:51 (1)
2023-10-01T06:18:52 (1) 2023-10-01T06:20:15 (2) 2023-10-01T06:20:16 (1)
2023-10-01T09:59:31 (1) 2023-10-01T10:15:26 (1) 2023-10-01T10:43:56 (1)
2023-10-01T10:43:57 (2) 2023-10-01T11:27:37 (1) 2023-10-01T12:09:02 (2)
2023-10-01T12:09:03 (2) 2023-10-01T12:14:59 (1) 2023-10-01T12:18:19 (1)
2023-10-01T15:03:55 (1) 2023-10-01T15:15:15 (1) 2023-10-01T16:11:28 (1)
2023-10-01T16:33:04 (2) 2023-10-01T16:33:05 (2) 2023-10-01T17:17:30 (2)
2023-10-01T18:12:36 (4) 2023-10-01T22:52:19 (1) 2023-10-01T23:07:46 (4)
2023-10-01T23:08:15 (1) 2023-10-01T23:14:57 (4) 2023-10-01T23:31:04 (2)
2023-10-01T23:31:05 (1) 2023-10-01T23:35:01 (1) 2023-10-01T23:39:17 (1)
2023-10-01T23:40:24 (1) 2023-10-01T23:45:31 (1) 2023-10-01T23:45:33 (1)
2023-10-01T23:45:35 (1) 2023-10-01T23:49:40 (1) 2023-10-01T23:50:48 (1)
2023-10-01T23:53:53 (1) 2023-10-01T23:55:26 (1) 2023-10-01T23:59:51 (1)
2023-10-02T00:05:15 (1) 2023-10-02T00:08:29 (1) 2023-10-02T00:17:27 (1)
2023-10-02T03:00:36 (1) 2023-10-02T03:01:06 (1) 2023-10-02T03:01:12 (1)
2023-10-02T03:08:31 (1) 2023-10-02T04:23:01 (1) 2023-10-02T04:41:58 (1)
2023-10-02T05:17:26 (1) 2023-10-02T08:18:48 (1) 2023-10-02T08:18:52 (1)
2023-10-02T11:49:00 (1) 2023-10-02T11:53:58 (1) 2023-10-02T13:15:02 (3)
2023-10-02T13:15:03 (1) 2023-10-02T14:12:44 (2) 2023-10-02T14:12:45 (2)
2023-10-02T17:18:51 (1) 2023-10-02T18:29:16 (1) 2023-10-02T18:33:28 (1)
2023-10-02T18:45:11 (1) 2023-10-02T18:45:12 (1) 2023-10-02T19:14:53 (1)
2023-10-02T19:14:55 (1) 2023-10-02T19:24:12 (2) 2023-10-02T19:24:13 (2)
2023-10-02T19:24:17 (3) 2023-10-02T19:24:18 (1) 2023-10-02T20:17:01 (2)
2023-10-02T20:17:02 (1) 2023-10-02T20:43:01 (1) 2023-10-02T20:44:26 (1)
2023-10-02T20:44:28 (1) 2023-10-02T21:34:38 (1) 2023-10-02T21:35:20 (1)
2023-10-02T21:35:56 (1) 2023-10-02T22:08:32 (1) 2023-10-02T22:09:23 (1)
2023-10-02T22:09:24 (1) 2023-10-02T22:40:55 (1) 2023-10-02T23:23:45 (1)
2023-10-02T23:33:21 (1) 2023-10-02T23:33:22 (3) 2023-10-02T23:36:36 (1)
2023-10-03T00:11:08 (1) 2023-10-03T00:39:48 (1) 2023-10-03T00:40:50 (1)
2023-10-03T00:57:51 (1) 2023-10-03T01:03:49 (1) 2023-10-03T01:03:51 (1)
2023-10-03T01:06:57 (1) 2023-10-03T01:06:58 (2) 2023-10-03T01:06:59 (1)
2023-10-03T01:20:29 (1) 2023-10-03T01:32:24 (1) 2023-10-03T01:37:52 (1)
2023-10-03T02:04:14 (1) 2023-10-03T02:04:15 (3) 2023-10-03T02:40:47 (1)
2023-10-03T02:40:48 (2) 2023-10-03T03:42:21 (1) 2023-10-03T04:40:21 (1)
2023-10-03T07:40:27 (1) 2023-10-03T12:49:30 (1) 2023-10-03T12:49:31 (3)
2023-10-03T14:13:46 (1) 2023-10-03T15:47:40 (1) 2023-10-03T17:08:40 (1)
2023-10-03T18:24:27 (1) 2023-10-03T18:28:49 (2) 2023-10-03T19:09:30 (1)
2023-10-03T19:52:17 (3) 2023-10-03T19:52:18 (1) 2023-10-03T19:56:34 (4)
2023-10-03T21:29:44 (1) 2023-10-03T21:29:45 (1) 2023-10-03T22:00:12 (4)
2023-10-03T22:05:15 (1) 2023-10-03T22:14:23 (2) 2023-10-03T22:14:24 (1)
2023-10-03T22:14:25 (1) 2023-10-04T03:04:13 (1) 2023-10-04T05:43:46 (2)
2023-10-04T05:43:47 (2) 2023-10-04T06:08:21 (1) 2023-10-04T06:45:01 (1)
2023-10-04T06:50:48 (1) 2023-10-04T09:58:30 (1) 2023-10-04T11:27:13 (1)
2023-10-04T11:45:24 (1) 2023-10-04T11:48:14 (2) 2023-10-04T12:38:37 (1)
2023-10-04T12:38:38 (1) 2023-10-04T16:46:57 (1) 2023-10-04T16:47:00 (1)
2023-10-04T16:47:02 (1) 2023-10-04T17:20:10 (2) 2023-10-04T17:20:11 (2)
2023-10-04T18:50:23 (1) 2023-10-04T19:41:31 (4) 2023-10-04T20:07:36 (1)
2023-10-04T21:18:55 (1) 2023-10-04T21:18:56 (3) 2023-10-04T21:26:16 (4)
2023-10-04T21:31:35 (4) 2023-10-04T22:03:51 (1) 2023-10-05T01:09:03 (1)
2023-10-05T02:02:52 (1) 2023-10-05T06:19:56 (2) 2023-10-05T06:19:57 (2)
2023-10-05T06:43:34 (1) 2023-10-05T06:50:13 (1) 2023-10-05T08:38:54 (1)
2023-10-05T09:16:30 (1) 2023-10-05T10:05:11 (1) 2023-10-05T11:04:24 (1)
2023-10-05T11:20:29 (4) 2023-10-05T13:43:16 (1) 2023-10-05T15:06:08 (3)
2023-10-05T15:15:14 (2) 2023-10-05T15:15:53 (1) 2023-10-05T18:10:58 (4)
2023-10-05T18:38:59 (1) 2023-10-05T19:43:24 (1) 2023-10-05T19:50:16 (1)
2023-10-05T20:38:51 (1) 2023-10-06T00:01:32 (4) 2023-10-06T00:02:28 (1)
2023-10-06T00:02:29 (3) 2023-10-06T00:03:54 (2) 2023-10-06T00:03:55 (2)
2023-10-06T00:09:39 (2) 2023-10-06T00:09:40 (2) 2023-10-06T05:47:46 (1)
2023-10-06T06:43:53 (1) 2023-10-06T06:49:34 (1) 2023-10-06T07:35:55 (1)
2023-10-06T07:51:34 (1) 2023-10-06T07:57:52 (1) 2023-10-06T07:57:54 (2)
2023-10-06T09:46:45 (4) 2023-10-06T10:51:12 (1) 2023-10-06T11:59:49 (1)
2023-10-06T11:59:51 (1) 2023-10-06T12:47:42 (1) 2023-10-06T13:33:29 (1)
2023-10-06T13:42:36 (1) 2023-10-06T18:26:09 (2) 2023-10-06T18:26:10 (2)
2023-10-06T19:55:36 (1) 2023-10-06T21:19:54 (1) 2023-10-06T22:34:49 (1)
2023-10-06T22:51:47 (1) 2023-10-06T23:32:02 (3) 2023-10-06T23:32:03 (1)
2023-10-06T23:36:28 (2) 2023-10-06T23:36:29 (2) 2023-10-07T00:15:41 (1)
2023-10-07T00:40:19 (1) 2023-10-07T00:40:52 (1) 2023-10-07T01:16:43 (1)
2023-10-07T03:46:15 (1) 2023-10-07T03:46:16 (3) 2023-10-07T03:51:05 (1)
2023-10-07T03:51:06 (1) 2023-10-07T03:51:07 (2) 2023-10-07T04:13:39 (1)
2023-10-07T04:13:41 (1) 2023-10-07T06:44:05 (1) 2023-10-07T06:49:47 (1)
2023-10-07T09:03:39 (1) 2023-10-07T12:04:00 (4) 2023-10-07T12:42:04 (1)
2023-10-07T12:42:08 (1) 2023-10-07T12:42:15 (1) 2023-10-07T21:30:16 (1)
2023-10-07T21:30:17 (1) 2023-10-07T21:30:49 (1) 2023-10-07T21:46:47 (1)
2023-10-07T23:02:46 (1) 2023-10-08T00:27:27 (1) 2023-10-08T00:33:49 (2)
2023-10-08T00:33:50 (2) 2023-10-08T02:57:46 (3) 2023-10-08T02:57:47 (1)
2023-10-08T03:00:50 (3) 2023-10-08T03:00:51 (1) 2023-10-08T06:33:17 (1)
2023-10-08T06:51:37 (1) 2023-10-08T06:58:17 (1) 2023-10-08T09:06:37 (4)
2023-10-08T09:07:29 (2) 2023-10-08T09:07:30 (2) 2023-10-08T10:19:38 (1)
2023-10-08T10:19:39 (1) 2023-10-08T10:57:40 (1) 2023-10-08T13:25:04 (1)
2023-10-08T13:25:05 (3) 2023-10-08T13:36:02 (1) 2023-10-08T13:36:03 (2)
2023-10-08T13:36:05 (2) 2023-10-08T14:46:27 (9) 2023-10-08T14:46:28 (7)
2023-10-08T14:46:30 (1) 2023-10-08T14:47:26 (1) 2023-10-08T14:48:40 (1)
2023-10-08T15:15:10 (1) 2023-10-08T18:12:36 (1) 2023-10-08T20:58:11 (1)
2023-10-08T20:58:13 (1) 2023-10-09T01:29:29 (4) 2023-10-09T01:37:58 (1)
2023-10-09T01:54:08 (1) 2023-10-09T02:35:23 (1) 2023-10-09T04:26:06 (1)
2023-10-09T06:58:03 (4) 2023-10-09T08:02:11 (4) 2023-10-09T08:06:37 (2)
2023-10-09T08:06:38 (2) 2023-10-09T08:07:03 (1) 2023-10-09T08:07:04 (3)
2023-10-09T11:46:33 (1) 2023-10-09T11:53:25 (1) 2023-10-09T14:36:06 (2)
2023-10-09T14:36:07 (2) 2023-10-09T15:18:15 (2) 2023-10-09T15:35:28 (1)
2023-10-09T23:15:50 (2) 2023-10-10T00:30:12 (4) 2023-10-10T01:35:38 (1)
2023-10-10T03:16:34 (1) 2023-10-10T04:08:13 (4) 2023-10-10T04:08:26 (1)
2023-10-10T04:15:38 (1) 2023-10-10T06:27:32 (1) 2023-10-10T09:28:15 (1)
2023-10-10T09:28:16 (3) 2023-10-10T10:32:41 (1) 2023-10-10T10:32:46 (1)
2023-10-10T10:36:26 (1) 2023-10-10T10:36:27 (1) 2023-10-10T10:42:52 (4)
2023-10-10T10:47:06 (3) 2023-10-10T10:47:07 (1) 2023-10-10T13:18:19 (2)
2023-10-10T13:18:20 (2) 2023-10-10T14:09:20 (1) 2023-10-10T14:09:26 (1)
2023-10-10T14:09:33 (1) 2023-10-10T17:28:27 (1) 2023-10-10T18:06:01 (1)
2023-10-11T00:15:12 (2) 2023-10-11T00:15:13 (2) 2023-10-11T02:22:45 (1)
2023-10-11T02:22:51 (2) 2023-10-11T03:49:26 (1) 2023-10-11T03:49:27 (3)
2023-10-11T03:55:01 (1) 2023-10-11T04:48:10 (1) 2023-10-11T05:03:50 (1)
2023-10-11T05:03:52 (1) 2023-10-11T05:50:12 (1) 2023-10-11T06:46:22 (1)
2023-10-11T06:46:25 (1) 2023-10-11T09:25:05 (1) 2023-10-11T09:32:11 (1)
2023-10-11T09:59:03 (1) 2023-10-11T10:03:11 (4) 2023-10-11T11:01:07 (1)
2023-10-11T11:44:52 (1) 2023-10-11T12:55:39 (1) 2023-10-11T12:59:05 (8)
2023-10-11T12:59:06 (10) 2023-10-11T12:59:07 (4) 2023-10-11T12:59:08 (7)
2023-10-11T12:59:09 (4) 2023-10-11T12:59:10 (3) 2023-10-11T12:59:11 (3)
2023-10-11T12:59:13 (3) 2023-10-11T12:59:14 (2) 2023-10-11T12:59:15 (2)
2023-10-11T12:59:16 (2) 2023-10-11T12:59:17 (3) 2023-10-11T12:59:18 (2)
2023-10-11T12:59:19 (3) 2023-10-11T12:59:21 (3) 2023-10-11T12:59:22 (1)
2023-10-11T12:59:23 (2) 2023-10-11T12:59:24 (1) 2023-10-11T12:59:25 (3)
2023-10-11T12:59:27 (2) 2023-10-11T12:59:28 (3) 2023-10-11T12:59:29 (1)
2023-10-11T12:59:31 (1) 2023-10-11T12:59:32 (3) 2023-10-11T12:59:33 (2)
2023-10-11T12:59:34 (1) 2023-10-11T12:59:35 (2) 2023-10-11T12:59:36 (1)
2023-10-11T12:59:40 (1) 2023-10-11T12:59:41 (1) 2023-10-11T12:59:42 (2)
2023-10-11T12:59:43 (2) 2023-10-11T12:59:44 (3) 2023-10-11T12:59:45 (2)
2023-10-11T12:59:46 (3) 2023-10-11T12:59:48 (1) 2023-10-11T12:59:49 (2)
2023-10-11T12:59:50 (2) 2023-10-11T12:59:51 (2) 2023-10-11T12:59:52 (1)
2023-10-11T12:59:53 (1) 2023-10-11T12:59:56 (1) 2023-10-11T12:59:58 (1)
2023-10-11T12:59:59 (1) 2023-10-11T13:00:03 (1) 2023-10-11T13:00:04 (1)
2023-10-11T13:00:09 (2) 2023-10-11T13:00:10 (1) 2023-10-11T13:00:11 (2)
2023-10-11T13:00:12 (1) 2023-10-11T13:00:13 (1) 2023-10-11T13:00:14 (1)
2023-10-11T13:00:15 (2) 2023-10-11T13:10:53 (1) 2023-10-11T13:11:38 (1)
2023-10-11T13:14:13 (1) 2023-10-11T13:14:14 (1) 2023-10-11T16:09:49 (1)
2023-10-11T17:57:39 (1) 2023-10-11T17:57:43 (1) 2023-10-11T19:53:34 (1)
2023-10-11T19:53:35 (3) 2023-10-11T19:58:21 (4) 2023-10-11T20:11:13 (4)
2023-10-11T20:16:58 (4)
Requests per minute:
2023-09-30T21:54 (1) 2023-09-30T22:02 (1) 2023-09-30T22:16 (4)
2023-09-30T23:29 (2) 2023-09-30T23:55 (2) 2023-09-30T23:59 (2)
2023-10-01T00:00 (2) 2023-10-01T00:53 (1) 2023-10-01T01:09 (1)
2023-10-01T04:12 (1) 2023-10-01T04:18 (1) 2023-10-01T06:18 (29)
2023-10-01T06:20 (3) 2023-10-01T09:59 (1) 2023-10-01T10:15 (1)
2023-10-01T10:43 (3) 2023-10-01T11:27 (1) 2023-10-01T12:09 (4)
2023-10-01T12:14 (1) 2023-10-01T12:18 (1) 2023-10-01T15:03 (1)
2023-10-01T15:15 (1) 2023-10-01T16:11 (1) 2023-10-01T16:33 (4)
2023-10-01T17:17 (2) 2023-10-01T18:12 (4) 2023-10-01T22:52 (1)
2023-10-01T23:07 (4) 2023-10-01T23:08 (1) 2023-10-01T23:14 (4)
2023-10-01T23:31 (3) 2023-10-01T23:35 (1) 2023-10-01T23:39 (1)
2023-10-01T23:40 (1) 2023-10-01T23:45 (3) 2023-10-01T23:49 (1)
2023-10-01T23:50 (1) 2023-10-01T23:53 (1) 2023-10-01T23:55 (1)
2023-10-01T23:59 (1) 2023-10-02T00:05 (1) 2023-10-02T00:08 (1)
2023-10-02T00:17 (1) 2023-10-02T03:00 (1) 2023-10-02T03:01 (2)
2023-10-02T03:08 (1) 2023-10-02T04:23 (1) 2023-10-02T04:41 (1)
2023-10-02T05:17 (1) 2023-10-02T08:18 (2) 2023-10-02T11:49 (1)
2023-10-02T11:53 (1) 2023-10-02T13:15 (4) 2023-10-02T14:12 (4)
2023-10-02T17:18 (1) 2023-10-02T18:29 (1) 2023-10-02T18:33 (1)
2023-10-02T18:45 (2) 2023-10-02T19:14 (2) 2023-10-02T19:24 (8)
2023-10-02T20:17 (3) 2023-10-02T20:43 (1) 2023-10-02T20:44 (2)
2023-10-02T21:34 (1) 2023-10-02T21:35 (2) 2023-10-02T22:08 (1)
2023-10-02T22:09 (2) 2023-10-02T22:40 (1) 2023-10-02T23:23 (1)
2023-10-02T23:33 (4) 2023-10-02T23:36 (1) 2023-10-03T00:11 (1)
2023-10-03T00:39 (1) 2023-10-03T00:40 (1) 2023-10-03T00:57 (1)
2023-10-03T01:03 (2) 2023-10-03T01:06 (4) 2023-10-03T01:20 (1)
2023-10-03T01:32 (1) 2023-10-03T01:37 (1) 2023-10-03T02:04 (4)
2023-10-03T02:40 (3) 2023-10-03T03:42 (1) 2023-10-03T04:40 (1)
2023-10-03T07:40 (1) 2023-10-03T12:49 (4) 2023-10-03T14:13 (1)
2023-10-03T15:47 (1) 2023-10-03T17:08 (1) 2023-10-03T18:24 (1)
2023-10-03T18:28 (2) 2023-10-03T19:09 (1) 2023-10-03T19:52 (4)
2023-10-03T19:56 (4) 2023-10-03T21:29 (2) 2023-10-03T22:00 (4)
2023-10-03T22:05 (1) 2023-10-03T22:14 (4) 2023-10-04T03:04 (1)
2023-10-04T05:43 (4) 2023-10-04T06:08 (1) 2023-10-04T06:45 (1)
2023-10-04T06:50 (1) 2023-10-04T09:58 (1) 2023-10-04T11:27 (1)
2023-10-04T11:45 (1) 2023-10-04T11:48 (2) 2023-10-04T12:38 (2)
2023-10-04T16:46 (1) 2023-10-04T16:47 (2) 2023-10-04T17:20 (4)
2023-10-04T18:50 (1) 2023-10-04T19:41 (4) 2023-10-04T20:07 (1)
2023-10-04T21:18 (4) 2023-10-04T21:26 (4) 2023-10-04T21:31 (4)
2023-10-04T22:03 (1) 2023-10-05T01:09 (1) 2023-10-05T02:02 (1)
2023-10-05T06:19 (4) 2023-10-05T06:43 (1) 2023-10-05T06:50 (1)
2023-10-05T08:38 (1) 2023-10-05T09:16 (1) 2023-10-05T10:05 (1)
2023-10-05T11:04 (1) 2023-10-05T11:20 (4) 2023-10-05T13:43 (1)
2023-10-05T15:06 (3) 2023-10-05T15:15 (3) 2023-10-05T18:10 (4)
2023-10-05T18:38 (1) 2023-10-05T19:43 (1) 2023-10-05T19:50 (1)
2023-10-05T20:38 (1) 2023-10-06T00:01 (4) 2023-10-06T00:02 (4)
2023-10-06T00:03 (4) 2023-10-06T00:09 (4) 2023-10-06T05:47 (1)
2023-10-06T06:43 (1) 2023-10-06T06:49 (1) 2023-10-06T07:35 (1)
2023-10-06T07:51 (1) 2023-10-06T07:57 (3) 2023-10-06T09:46 (4)
2023-10-06T10:51 (1) 2023-10-06T11:59 (2) 2023-10-06T12:47 (1)
2023-10-06T13:33 (1) 2023-10-06T13:42 (1) 2023-10-06T18:26 (4)
2023-10-06T19:55 (1) 2023-10-06T21:19 (1) 2023-10-06T22:34 (1)
2023-10-06T22:51 (1) 2023-10-06T23:32 (4) 2023-10-06T23:36 (4)
2023-10-07T00:15 (1) 2023-10-07T00:40 (2) 2023-10-07T01:16 (1)
2023-10-07T03:46 (4) 2023-10-07T03:51 (4) 2023-10-07T04:13 (2)
2023-10-07T06:44 (1) 2023-10-07T06:49 (1) 2023-10-07T09:03 (1)
2023-10-07T12:04 (4) 2023-10-07T12:42 (3) 2023-10-07T21:30 (3)
2023-10-07T21:46 (1) 2023-10-07T23:02 (1) 2023-10-08T00:27 (1)
2023-10-08T00:33 (4) 2023-10-08T02:57 (4) 2023-10-08T03:00 (4)
2023-10-08T06:33 (1) 2023-10-08T06:51 (1) 2023-10-08T06:58 (1)
2023-10-08T09:06 (4) 2023-10-08T09:07 (4) 2023-10-08T10:19 (2)
2023-10-08T10:57 (1) 2023-10-08T13:25 (4) 2023-10-08T13:36 (5)
2023-10-08T14:46 (17) 2023-10-08T14:47 (1) 2023-10-08T14:48 (1)
2023-10-08T15:15 (1) 2023-10-08T18:12 (1) 2023-10-08T20:58 (2)
2023-10-09T01:29 (4) 2023-10-09T01:37 (1) 2023-10-09T01:54 (1)
2023-10-09T02:35 (1) 2023-10-09T04:26 (1) 2023-10-09T06:58 (4)
2023-10-09T08:02 (4) 2023-10-09T08:06 (4) 2023-10-09T08:07 (4)
2023-10-09T11:46 (1) 2023-10-09T11:53 (1) 2023-10-09T14:36 (4)
2023-10-09T15:18 (2) 2023-10-09T15:35 (1) 2023-10-09T23:15 (2)
2023-10-10T00:30 (4) 2023-10-10T01:35 (1) 2023-10-10T03:16 (1)
2023-10-10T04:08 (5) 2023-10-10T04:15 (1) 2023-10-10T06:27 (1)
2023-10-10T09:28 (4) 2023-10-10T10:32 (2) 2023-10-10T10:36 (2)
2023-10-10T10:42 (4) 2023-10-10T10:47 (4) 2023-10-10T13:18 (4)
2023-10-10T14:09 (3) 2023-10-10T17:28 (1) 2023-10-10T18:06 (1)
2023-10-11T00:15 (4) 2023-10-11T02:22 (3) 2023-10-11T03:49 (4)
2023-10-11T03:55 (1) 2023-10-11T04:48 (1) 2023-10-11T05:03 (2)
2023-10-11T05:50 (1) 2023-10-11T06:46 (2) 2023-10-11T09:25 (1)
2023-10-11T09:32 (1) 2023-10-11T09:59 (1) 2023-10-11T10:03 (4)
2023-10-11T11:01 (1) 2023-10-11T11:44 (1) 2023-10-11T12:55 (1)
2023-10-11T12:59 (108) 2023-10-11T13:00 (12) 2023-10-11T13:10 (1)
2023-10-11T13:11 (1) 2023-10-11T13:14 (2) 2023-10-11T16:09 (1)
2023-10-11T17:57 (2) 2023-10-11T19:53 (4) 2023-10-11T19:58 (4)
2023-10-11T20:11 (4) 2023-10-11T20:16 (4)
Requests per hour:
2023-09-30T21 (1) 2023-09-30T22 (5) 2023-09-30T23 (6)
2023-10-01T00 (3) 2023-10-01T01 (1) 2023-10-01T04 (2)
2023-10-01T06 (32) 2023-10-01T09 (1) 2023-10-01T10 (4)
2023-10-01T11 (1) 2023-10-01T12 (6) 2023-10-01T15 (2)
2023-10-01T16 (5) 2023-10-01T17 (2) 2023-10-01T18 (4)
2023-10-01T22 (1) 2023-10-01T23 (23) 2023-10-02T00 (3)
2023-10-02T03 (4) 2023-10-02T04 (2) 2023-10-02T05 (1)
2023-10-02T08 (2) 2023-10-02T11 (2) 2023-10-02T13 (4)
2023-10-02T14 (4) 2023-10-02T17 (1) 2023-10-02T18 (4)
2023-10-02T19 (10) 2023-10-02T20 (6) 2023-10-02T21 (3)
2023-10-02T22 (4) 2023-10-02T23 (6) 2023-10-03T00 (4)
2023-10-03T01 (9) 2023-10-03T02 (7) 2023-10-03T03 (1)
2023-10-03T04 (1) 2023-10-03T07 (1) 2023-10-03T12 (4)
2023-10-03T14 (1) 2023-10-03T15 (1) 2023-10-03T17 (1)
2023-10-03T18 (3) 2023-10-03T19 (9) 2023-10-03T21 (2)
2023-10-03T22 (9) 2023-10-04T03 (1) 2023-10-04T05 (4)
2023-10-04T06 (3) 2023-10-04T09 (1) 2023-10-04T11 (4)
2023-10-04T12 (2) 2023-10-04T16 (3) 2023-10-04T17 (4)
2023-10-04T18 (1) 2023-10-04T19 (4) 2023-10-04T20 (1)
2023-10-04T21 (12) 2023-10-04T22 (1) 2023-10-05T01 (1)
2023-10-05T02 (1) 2023-10-05T06 (6) 2023-10-05T08 (1)
2023-10-05T09 (1) 2023-10-05T10 (1) 2023-10-05T11 (5)
2023-10-05T13 (1) 2023-10-05T15 (6) 2023-10-05T18 (5)
2023-10-05T19 (2) 2023-10-05T20 (1) 2023-10-06T00 (16)
2023-10-06T05 (1) 2023-10-06T06 (2) 2023-10-06T07 (5)
2023-10-06T09 (4) 2023-10-06T10 (1) 2023-10-06T11 (2)
2023-10-06T12 (1) 2023-10-06T13 (2) 2023-10-06T18 (4)
2023-10-06T19 (1) 2023-10-06T21 (1) 2023-10-06T22 (2)
2023-10-06T23 (8) 2023-10-07T00 (3) 2023-10-07T01 (1)
2023-10-07T03 (8) 2023-10-07T04 (2) 2023-10-07T06 (2)
2023-10-07T09 (1) 2023-10-07T12 (7) 2023-10-07T21 (4)
2023-10-07T23 (1) 2023-10-08T00 (5) 2023-10-08T02 (4)
2023-10-08T03 (4) 2023-10-08T06 (3) 2023-10-08T09 (8)
2023-10-08T10 (3) 2023-10-08T13 (9) 2023-10-08T14 (19)
2023-10-08T15 (1) 2023-10-08T18 (1) 2023-10-08T20 (2)
2023-10-09T01 (6) 2023-10-09T02 (1) 2023-10-09T04 (1)
2023-10-09T06 (4) 2023-10-09T08 (12) 2023-10-09T11 (2)
2023-10-09T14 (4) 2023-10-09T15 (3) 2023-10-09T23 (2)
2023-10-10T00 (4) 2023-10-10T01 (1) 2023-10-10T03 (1)
2023-10-10T04 (6) 2023-10-10T06 (1) 2023-10-10T09 (4)
2023-10-10T10 (12) 2023-10-10T13 (4) 2023-10-10T14 (3)
2023-10-10T17 (1) 2023-10-10T18 (1) 2023-10-11T00 (4)
2023-10-11T02 (3) 2023-10-11T03 (5) 2023-10-11T04 (1)
2023-10-11T05 (3) 2023-10-11T06 (2) 2023-10-11T09 (3)
2023-10-11T10 (4) 2023-10-11T11 (2) 2023-10-11T12 (109)
2023-10-11T13 (16) 2023-10-11T16 (1) 2023-10-11T17 (2)
2023-10-11T19 (8) 2023-10-11T20 (8)
Requests per day:
2023-09-30 (12) 2023-10-01 (87) 2023-10-02 (56)
2023-10-03 (53) 2023-10-04 (41) 2023-10-05 (31)
2023-10-06 (50) 2023-10-07 (29) 2023-10-08 (59)
2023-10-09 (35) 2023-10-10 (38) 2023-10-11 (171)
Example filtering for 185.170.144.3 requests in /home/nginx/domains/domain.com/log/access.json Nginx JSON log.
./bot-rate.py /home/nginx/domains/domain.com/log/access.json 185.170.144.3
Requests with keyword '185.170.144.3':
Requests per second:
2023-10-02T15:21:04 (1) 2023-10-02T15:21:05 (1) 2023-10-02T15:21:06 (1)
2023-10-06T01:26:26 (1) 2023-10-06T01:26:28 (1) 2023-10-07T02:33:57 (1)
2023-10-07T02:33:58 (1) 2023-10-09T11:10:45 (1) 2023-10-09T11:10:47 (1)
2023-10-11T20:59:35 (1) 2023-10-11T20:59:37 (1) 2023-10-11T20:59:39 (1)
Requests per minute:
2023-10-02T15:21 (3) 2023-10-06T01:26 (2) 2023-10-07T02:33 (2)
2023-10-09T11:10 (2) 2023-10-11T20:59 (3)
Requests per hour:
2023-10-02T15 (3) 2023-10-06T01 (2) 2023-10-07T02 (2)
2023-10-09T11 (2) 2023-10-11T20 (3)
Requests per day:
2023-10-02 (3) 2023-10-06 (2) 2023-10-07 (2)
2023-10-09 (2) 2023-10-11 (3)
To setup Nginx JSON access logging, you will need to manually edit your main /usr/local/nginx/conf/nginx.conf config file and add a 4th log_format to existing listing which is listed directly below where log_format named = main_json:
log_format main_json escape=json '{'
'"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
'"connection": "$connection", ' # connection serial number
'"connection_requests": "$connection_requests", ' # number of requests made in connection
'"pid": "$pid", ' # process pid
'"request_id": "$request_id", ' # the unique request id
'"request_length": "$request_length", ' # request length (including headers and body)
'"remote_addr": "$remote_addr", ' # client IP
'"remote_user": "$remote_user", ' # client HTTP username
'"remote_port": "$remote_port", ' # client port
'"time_local": "$time_local", '
'"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
'"request": "$request", ' # full path no arguments if the request
'"request_uri": "$request_uri", ' # full path and arguments if the request
'"args": "$args", ' # args
'"status": "$status", ' # response status code
'"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client
'"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
'"http_referer": "$http_referer", ' # HTTP referer
'"http_user_agent": "$http_user_agent", ' # user agent
'"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for
'"http_host": "$http_host", ' # the request Host: header
'"server_name": "$server_name", ' # the name of the vhost serving the request
'"request_time": "$request_time", ' # request processing time in seconds with msec resolution
'"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
'"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
'"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
'"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body
'"upstream_response_length": "$upstream_response_length", ' # upstream response length
'"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable
'"ssl_protocol": "$ssl_protocol", ' # TLS protocol
'"ssl_session_reused": "$ssl_session_reused", ' # TLS session reused
'"ssl_cipher": "$ssl_cipher", ' # TLS cipher only OpenSSL 3.0 supported
'"ssl_curve": "$ssl_curve", ' # TLS curve OpenSSL 1.0.2+ supported
'"ssl_curves": "$ssl_curves", ' # TLS curves
'"scheme": "$scheme", ' # http or https
'"request_method": "$request_method", ' # request method
'"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
'"pipe": "$pipe", ' # “p” if request was pipelined, “.” otherwise
'"gzip_ratio": "$gzip_ratio", '
'"http_cf_ray": "$http_cf_ray"'
'}';
Next in your Nginx site vhost config file i.e. HTTPS vhost would be at /usr/local/nginx/conf/conf.d/domain.com.ssl.conf, you will need to add a new access_log line referencing the log_format named = main_json. The buffer and flush directives tell Nginx to use a memory buffer to write to access log every 256KB size or after 5 minutes for better performance. So you will need to do Nginx reload/restart to flush any memory buffered access logs to disk if you want to inspect latest log info.
access_log /home/nginx/domains/domain.com/log/access_log.json main_json buffer=256k flush=5m;
Normally, you would want to place it grouped with your existing access_log in your Nginx site vhost config file like.
access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
access_log /home/nginx/domains/domain.com/log/access_log.json main_json buffer=256k flush=5m;
error_log /home/nginx/domains/domain.com/log/error.log;
Then restart Nginx server for changes to take affect
service nginx restart
or cmd shortcut
ngxrestart
You can use jq to filter just the Nginx metrics you want. Example
cat access_log.json | tail -1 | jq -r '"\(.time_local) \(.remote_addr) \(.http_x_forwarded_for) \(.request_uri) \(.status) \(.request_method) \(.http_host) \(.scheme) \(.server_protocol) \(.ssl_protocol) \(.ssl_cipher) \(.http_cf_ray) \(.http_user_agent)"'
03/May/2020:08:59:16 +0000 66.249.73.203 66.249.73.203 /threads/ovh-new-infrastructure-line-severs-intel-xeon-e-22xx-xeon-silver-amd-epyc-7371.18392/ 200 GET community.centminmod.com https HTTP/1.1 TLSv1.3 TLS_AES_256_GCM_SHA384 58d8c8f393bbe049-DFW Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Or find out the proportion of HTTPS SSL ciphers served to visitors
cat access_log.json | jq -r '.ssl_cipher' | sort | uniq -c | sort -rn
3099 TLS_AES_256_GCM_SHA384
171 ECDHE-ECDSA-AES256-GCM-SHA384
22 ECDHE-ECDSA-CHACHA20-POLY1305
2 ECDHE-ECDSA-AES128-SHA
HTTPS SSL protocols and ciphers served to visitors
cat access_log.json | jq -r '"\(.ssl_protocol) \(.ssl_cipher)"' | sort | uniq -c | sort -rn
3099 TLSv1.3 TLS_AES_256_GCM_SHA384
171 TLSv1.2 ECDHE-ECDSA-AES256-GCM-SHA384
22 TLSv1.2 ECDHE-ECDSA-CHACHA20-POLY1305
1 TLSv1 ECDHE-ECDSA-AES128-SHA
1 TLSv1.1 ECDHE-ECDSA-AES128-SHA
HTTPS SSL protocols and HTTP status codes served to visitors
cat access_log.json | jq -r '"\(.ssl_protocol) \(.status)"' | sort | uniq -c | sort -rn
2115 TLSv1.3 200
478 TLSv1.3 301
226 TLSv1.3 307
150 TLSv1.2 200
135 TLSv1.3 304
76 TLSv1.3 303
62 TLSv1.3 403
25 TLSv1.2 304
6 TLSv1.2 301
5 TLSv1.3 404
4 TLSv1.2 307
3 TLSv1.2 403
3 TLSv1.2 303
2 TLSv1.2 400
1 TLSv1 400
1 TLSv1.3 405
1 TLSv1.3 400
1 TLSv1.1 400
HTTPS protocol and HTTP status codes for only urls = /login/login
cat access_log.json | jq -r 'select(.request_uri == "/login/login")| "\(.ssl_protocol) \(.status)"' | sort | uniq -c | sort -rn
25 TLSv1.3 200
If you have Cloudflare proxy in front of Nginx, you can alter the Nginx json log format added to /usr/local/nginx/conf/nginx.conf to below version for additional logged fields
log_format named = cf_json
log_format cf_json escape=json '{'
'"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution
'"connection": "$connection", ' # connection serial number
'"connection_requests": "$connection_requests", ' # number of requests made in connection
'"pid": "$pid", ' # process pid
'"request_id": "$request_id", ' # the unique request id
'"request_length": "$request_length", ' # request length (including headers and body)
'"remote_addr": "$remote_addr", ' # client IP
'"remote_user": "$remote_user", ' # client HTTP username
'"remote_port": "$remote_port", ' # client port
'"time_local": "$time_local", '
'"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format
'"request": "$request", ' # full path no arguments if the request
'"request_uri": "$request_uri", ' # full path and arguments if the request
'"args": "$args", ' # args
'"status": "$status", ' # response status code
'"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client
'"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client
'"http_referer": "$http_referer", ' # HTTP referer
'"http_user_agent": "$http_user_agent", ' # user agent
'"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for
'"http_host": "$http_host", ' # the request Host: header
'"server_name": "$server_name", ' # the name of the vhost serving the request
'"request_time": "$request_time", ' # request processing time in seconds with msec resolution
'"upstream": "$upstream_addr", ' # upstream backend server for proxied requests
'"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS
'"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers
'"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body
'"upstream_response_length": "$upstream_response_length", ' # upstream response length
'"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable
'"ssl_protocol": "$ssl_protocol", ' # TLS protocol
'"ssl_session_reused": "$ssl_session_reused", ' # TLS session reused
'"ssl_cipher": "$ssl_cipher", ' # TLS cipher only OpenSSL 3.0 supported
'"ssl_curve": "$ssl_curve", ' # TLS curve OpenSSL 1.0.2+ supported
'"ssl_curves": "$ssl_curves", ' # TLS curves
'"scheme": "$scheme", ' # http or https
'"request_method": "$request_method", ' # request method
'"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0
'"pipe": "$pipe", ' # “p” if request was pipelined, “.” otherwise
'"gzip_ratio": "$gzip_ratio", '
'"http_cf_ray": "$http_cf_ray", '
'"http_cf_worker": "$http_cf_worker", '
'"http_cf_request_id": "$http_cf_request_id", '
'"http_cf_railgun": "$http_cf_railgun", '
'"http_accept": "$http_accept"'
'}';
and use this in nginx vhost adding a 2nd log for
/home/nginx/domains/domain.com/log/access_log.json using nginx log format = cf_json
access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
access_log /home/nginx/domains/domain.com/log/access_log.json cf_json buffer=256k flush=5m;
error_log /home/nginx/domains/domain.com/log/error.log;