/
awstats_faq.html
1745 lines (1644 loc) · 109 KB
/
awstats_faq.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<meta name="description" content="AWStats Documentation - FAQ page">
<meta name="keywords" content="awstats, awstat, faq, error, errors, frequently, asked, questions, support, help, problem, solution, troubleshooting, rotate log">
<meta name="robots" content="index,follow">
<meta name="title" content="AWStats Documentation - FAQs">
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>AWStats Documentation - FAQs</title>
<link rel="stylesheet" href="styles.css" type="text/css">
<link href="https://plus.google.com/+LaurentDestailleur-Open-Source-Expert" rel="publisher" />
</head>
<body topmargin=10 leftmargin=5>
<table style="font: 10pt arial,helvetica,verdana" cellpadding=0 cellspacing=0 border=0 bgcolor=#FFFFFF width=100%>
<!-- Large -->
<tr style="font: 10pt arial,helvetica,verdana">
<td bgcolor=#9999cc align=center><a href="/"><img src="images/awstats_logo6.png" border=0></a></td>
<td bgcolor=#9999cc align=center>
<br>
<font style="font: 16pt arial,helvetica,sans-serif" color=#EEEEFF><b>AWStats logfile analyzer 7.6 Documentation</b></font><br>
<br>
</td>
<td bgcolor=#9999cc align=center>
</td>
</tr>
</table>
<br><br><H1 style="font: 26px arial,helvetica,sans-serif">Frequently Asked Questions + Troubleshooting</H1>
<br>
<u>ABOUT QUESTIONS:</u><br>
<ul>
FAQ-ABO100 <a href="#SERVERSOS">Which server log files or operating systems are supported ?</a><br>
FAQ-ABO150 <a href="#LOGFORMAT">Which log format can AWStats analyze ?</a><br>
FAQ-ABO200 <a href="#LANG">Which languages are available ? How to add my own language ?</a><br>
FAQ-ABO250 <a href="#PHPNUKE">Can AWStats be integrated with PHP Nuke ?</a><br>
FAQ-ABO300 <a href="#ABOUTHISTORY">About AWStats history</a><br>
</ul>
<br>
<u>COMMON SETUP/USAGE QUESTIONS:</u><br>
Here, you can find the most common questions and answers about AWStats setup/usage process.<br>
<ul>
FAQ-COM025 <a href="#NOLOG">How to use AWStats with no server log</a><br>
FAQ-COM050 <a href="#LIMITLOG">What is the log size limit AWStats can analyze ?</a><br>
FAQ-COM090 <a href="#FTP">Setup for FTP server log files (proftpd, vsftpd, ...).</a><br>
FAQ-COM100 <a href="#MAIL">Setup for MAIL log files (Postfix, Sendmail, QMail, MDaemon, Exchange).</a><br>
FAQ-COM110 <a href="#MEDIASERVER">Setup for MEDIA SERVER log files (Realmedia, Windows media, Darwin streaming server).</a><br>
FAQ-COM115 <a href="#PERSONALIZEDLOG">Setup/Examples for LogFormat parameter.</a><br>
FAQ-COM120 <a href="#ROTATE">How to rotate my logs without losing data.</a><br>
FAQ-COM130 <a href="#CRONTAB">How to run AWStats frequently ?</a><br>
FAQ-COM140 <a href="#EXCLUDEHOSTS">How to exclude my IP address (or whole subnet mask) from stats ?</a><br>
FAQ-COM142 <a href="#SCREENSIZE">How to get the screen size and browser capabilities report working ?</a><br>
FAQ-COM145 <a href="#EXTRA">How to use the Extra Sections features ?</a><br>
FAQ-COM150 <a href="#BENCHMARK">Benchmark question.</a><br>
FAQ-COM200 <a href="#DNS">How reverse DNS Lookup works, unresolved IP Addresses ?</a><br>
FAQ-COM250 <a href="#DIFFERENT_RESULTS">Different results than other log analyzers (Analog, Webalizer, WUsage, wwwStats...).</a><br>
FAQ-COM300 <a href="#DIFFERENCE_HOURS">Difference between local hour and AWStats reported hour.</a><br>
FAQ-COM320 <a href="#GEOIPEU">What does mean "eu (European country)" in GeoIP country reports.</a><br>
FAQ-COM350 <a href="#OLDLOG">How can I process old log file ?</a><br>
FAQ-COM360 <a href="#MULTILOG">How can I process several log files in one run ?</a><br>
FAQ-COM400 <a href="#LOADLOG">How can I update my statistics when I use a load balancing system that splits my logs ?</a><br>
FAQ-COM500 <a href="#RESET">How can I reset all my statistics ?</a><br>
FAQ-COM600 <a href="#DAILY">How can I compile and build statistics on a daily basis only ?</a><br>
FAQ-COM700 <a href="#EDITHISTORY">Can I safely remove a line in AWStats history files (awstatsMMYYYY*.txt) ?</a><br>
</ul>
<br>
<u>ERRORS/TROUBLESHOOTING QUESTIONS:</u><br>
Here, you can find the most common questions and answers about errors or problems using AWStats.<br>
<ul>
FAQ-SET050 <a href="#MISSINGDOLLAR">Error "Missing $ on loop variable ..."</a><br>
FAQ-SET100 <a href="#CGISOURCE">I see Perl script's source instead of its execution in my browser.</a><br>
FAQ-SET150 <a href="#SPAWNERROR">Error "...couldn't create/spawn child process..." with Apache for windows.</a><br>
FAQ-SET200 <a href="#INTERNAL">"Internal Error" or "Error 500" in a browser connecting to Apache.</a><br>
FAQ-SET210 <a href="#SPEED">"Internal Error" after a long time in my browser (See FAQ-COM100 "AWStats speed/timeout problems").</a><br>
FAQ-SET220 <a href="#CRASH">Crash while running awstats.pl or page content only partialy loaded</a><br>
FAQ-SET270 <a href="#CORRUPTEDDROPPED">Only corrupted/dropped records</a><br>
FAQ-SET280 <a href="#NOTSAMENUMBER">Error "Not same number of records of...".</a><br>
FAQ-SET300 <a href="#COULDNOTOPEN">Error "Couldn't open file ..."</a><br>
FAQ-SET320 <a href="#MALFORMEDUTF8">Error "Malformed UTF-8 character (unexpected..."</a><br>
FAQ-SET350 <a href="#EMPTY_STATS">Empty or null statistics reported.</a><br>
FAQ-SET360 <a href="#PARTIAL_STATS">Statistics reported except for os, browsers, robots and keywords/keyphrases.</a><br>
FAQ-SET400 <a href="#REDIRECT">Pipe redirection to a file give me an empty file.</a><br>
FAQ-SET450 <a href="#NO_ICON">No pictures/graphics shown.</a><br>
FAQ-SET700 <a href="#MIGRATEDOUBLED">My visits are doubled for old month I migrated from 3.2 to 5.x</a><br>
FAQ-SET750 <a href="#OUTOFMEMORYCYGWIN">AWStats run out of memory during update process with cygwin Perl.</a><br>
FAQ-SET800 <a href="#SPEED">AWStats speed/timeout problems.</a><br>
</ul>
<br>
<u>SECURITY QUESTIONS:</u><br>
Here, you can find the common questions about security problems when setting or using AWStats.<br>
<ul>
FAQ-SEC100 <a href="#CSSATTACK">Can AWStats be used to make Cross Site Scripting Attacks ?</a><br>
FAQ-SEC150 <a href="#SECUSER">How can I prevent some users to see statistics of other users ?</a><br>
FAQ-SEC200 <a href="#WORMS">How to manage log files (and statistics) corrupted by worms attacks like 'Code Red Virus like'.</a><br>
</ul>
<br>
<hr>
<br><br>
<a name="SERVERSOS"></a><br>
<b><u>FAQ-ABO100 : WHICH SERVER LOG FILES OR OS ARE SUPPORTED ?</u></b><br>
AWStats can works with :<br>
<li> All web server able to write log file with a <u>combined log format (XLF/ELF)</u> like Apache,
a <u>common log format (CLF)</u> like Apache or Squid, a <u>W3C log format</u> like IIS 5.0 or higher,
or any other log format that contains all information AWStats expect to find.<br>
<li> Most of all others Web/Wap/Proxy/Streaming servers.<br>
<li> Some FTP, Syslog or Mail log files.<br>
Because AWStats is in Perl, it can works on all Operating Systems.<br>
<br>
Examples of used platforms (bold means 'tested by author', others were reported by AWStats users to work correctly) :<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
<u>OS:</u><br>
<b>Windows 2000</b>, <b>Windows NT 4.0</b>, Windows Me, <b>Linux (RedHat, Mandrake, Debian, Suse...)</b>, Macintosh, <b>Solaris</b>, <b>Aix</b>, BeOS, FreeBSD, ...<br>
<u>Web/Wap/Proxy/Streaming servers</u><br>
<b>Apache 1.3.x and 2.x</b>, <b>IIS 5.0 or higher</b>, ISA, WebStar, WebLogic, WebSite, <b>Windows Media Server</b>, Tomcat, <b>Squid</b>,
Sambar, Roxen, Resin, RealMedia server, Oracle9iAS, <b>Lotus Notes/Domino</b>, Darwin, IPlanet, IceCast, ZeroBrand, Zeus, Zope, Abyss...<br>
<u>FTP servers</u><br>
<b>ProFTPd</b>, vsFTPd...<br>
<u>Mails servers</u><br>
<b>Postfix</b>, <b>Sendmail</b>, QMail, <b>Mdaemon</b>, www4mail, ...<br>
<u>Perl interpreters (all Perl >= 5.005):</u><br>
<b>ActivePerl 5.6</b>, <b>ActivePerl 5.8</b>, <b>Perl 5.8</b>, <b>Perl 5.6</b>, <b>Perl 5.005</b>, <b>mod_perl</b> and mod_perl2 for Apache, ...<br>
</td></tr></table>
<br>
<a name="LOGFORMAT"></a><br>
<b><u>FAQ-ABO150 : WHICH LOG FORMATS CAN AWSTATS ANALYZE ?</u></b><br>
AWStats setup knows predefined log formats you can use to make AWStats config easier. However,
you can define your own log format, that's the reason why AWStats can analyze nearly all web, wap
and proxy server log files. Some FTP servers log files, Syslog or mail logs can also be analyzed.<br>
The only requirement is "Your log file must contain required information".<br>
<br>
This is very short examples of possible log format:<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
Apache common log format (see Note*),<br>
Apache combined log format (known as NCSA combined log format or XLF or ELF format),<br>
Any other personalized Apache log format,<br>
Any IIS log format (known as W3C format),<br>
Webstar native log format,<br>
Realmedia server, Windows Media Server, Darwin streaming server,<br>
ProFTPd server, vsFTPd server,<br>
Postfix, Sendmail, QMail, Mdaemon<br>
A lot of web/wap/proxy/streaming servers log format<br>
</td></tr></table>
<br>
*Note: Apache common log format (AWStats can now analyze such log files but such log files does not
contain all information AWStats is looking for. The problem is in the content, not in the
format). I think analyzing common log files is not interesting because there is a lot of
missing information: no way to filter robots, find search engines, keywords, os, browser.
But a lot of users asked me for it, so AWStats support it.
However, a lot of interesting advanced features can't work: browsers, os's, keywords, robot detection...).
<br>
See also <a href="#PERSONALIZEDLOG">F.A.Q.: LOG FORMAT SETUP OR ERRORS </a>.<br>
<br>
<a name="LANG"></a><br>
<b><u>FAQ-ABO200 : WHICH LANGUAGES ARE AVAILABLE ?</u></b><br>
AWStats can make reports in 43 languages. This is a list of all of them, for last version, in
alphabetical order (The code you can use for <a href="awstats_config.html#Lang">Lang</a>
and <a href="awstats_config.html#ShowFlagLinks">ShowFlagLinks</a> parameter are
the ISO-639-1 language codes):<br>
<i>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
Albanian=al, Bosnian=ba, Brezhoneg=bzg, Bulgarian=bg, Catalan=ca,
Chinese (Taiwan)=tw, Chinese (Simpliefied)=cn, Croatian=hr, Czech=cz,
Danish=dk, Dutch=nl, English=en, Estonian=et, Euskara=eu, Finnish=fi,
French=fr, Galician=gl, German=de, Greek=gr, Hebrew=he, Hungarian=hu,
Icelandic=is, Indonesian=id, Italian=it, Japanese=jp, Korean=ko,
Latvian=lv, Norwegian (Nynorsk)=nn, Norwegian (Bokmal)=nb, Polish=pl,
Portuguese=pt, Portuguese (Brazilian)=br, Romanian=ro, Russian=ru,
Serbian=sr, Slovak=sk, Solvenian=si, Spanish=es, Swedish=se, Thai=th,
Turkish=tr, Ukrainian=ua, Welsh=cy.
</td></tr></table>
</i>
However, AWStats documentation is only provided in English.<br>
But, you may find small documentation for other languages made by contributors on
<a href="awstats_contrib.html#DOC">Documentation Contrib page</a>.<br>
<br>
If your language is not in this list, you can translate it yourself. For this, find what is your
2 letter language code: <a href="http://www.oasis-open.org/cover/iso639a.html">here</a>.<br>
Once, you get it, for example "gl" for Galician, copy the file awstats-en.txt into awstats-<i>gl</i>.txt,
in langs directory and translate every sentence inside. You can do same for files inside
tooltips_f, tooltips_m and tooltips_w sub-directories. Then send your translated file(s) to eldy@users.sourceforge.net.<br>
<br>
<a name="PHPNUKE"></a><br>
<b><u>FAQ-ABO250 : CAN AWSTATS BE INTEGRATED WITH PHP NUKE ?</u></b><br>
The only plugin I know to integrate AWStats inside PHPNuke is here: <a href="http://phpnuke.org/modules.php?name=News&file=article&sid=7041">PhpNuke addon for AWStats</a><br>
<br>
<a name="ABOUTHISTORY"></a><br>
<b><u>FAQ-ABO300 : ABOUT AWSTATS HISTORY</u></b><br>
AWStats was initialy designed for my own use to track visitors on my personal web sites or other projects i worked on
(<a href="http://www.chiensderace.com" alt="Chiens De Race .com">www.chiensderace.com</a>,
<a href="http://www.chatsderace.com" alt="Chats De Race .com">www.chatsderace.com</a>,
<a href="https://www.dolibarr.org" alt="Dolibarr.org, the simple ERP and CRM to manage your business">www.dolibarr.org</a>,
<a href="http://www.nltechno.com" alt="NLTechno">www.nltechno.com</a>,
and <a href="http://www.destailleur.fr" alt="Site personnel Laurent Destailleur">www.destailleur.fr</a>)<br>
Then I decided to put it on sourceforge in year 2000. Then a lot of new versions were
developed to add enhancements until today. See changelog for full history of changes.<br>
<br>
<hr><br>
<a name="NOLOG"></a><br>
<b><u>FAQ-COM025 : HOW TO USE AWSTATS WITH NO SERVER LOG FILE</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I want to have AWStats statistics but i have no access to my server log file.<br>
<font class=CSolution>SOLUTION:</font><br>
Because AWStats is a log analyzer, if you don't have any way to read your server log file,
you have nothing to analyze and you should not be able to use AWStats.
However, this is a trick that you can use to have a log file be built. You must add
a tag to call a CGI script like pslogger into each of your web pages. This will
allow you to have an artificial log file that can be analyzed by AWStats.<br>
You can find a Perl version of CGI pslogger enhanced by AWStats author <a href="/files/pslogger.pl">here</a>
or a php version of CGI pslogger made by Florent CHANTRET <a href="/files/pslogger.phps">here</a>.<br>
<br>
<a name="LIMITLOG"></a><br>
<b><u>FAQ-COM050 : WHAT IS THE LOG SIZE LIMIT AWSTATS CAN ANALYZE</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I know I must run AWStats update process frequently on new log files, this means thoose
files have a regular size, but for my first update, I want/need to run update process
on old log files that are very large. Is there a limit on log file size AWStats can analyze ?<br>
<font class=CSolution>SOLUTION:</font><br>
No. There is no limit in AWStats. This means you can use it on large log files (test were
made on 10GB log files).<br>
However your system (Operating System or Perl version) might have a limit. For example, you can
experience size limit errors on files larger than 2 or 4 GB.
If limit is Perl only, try to use a Perl version compiled with "large file" option.<br>
If you can't find it nor build it, you can try to use a LogFile parameter that looks like this
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="cat /yourlogfilepath/yourlogfile |"
</td></tr></table>
instead of
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="/yourlogfilepath/yourlogfile"
</td></tr></table>
<br>
<a name="FTP"></a><br>
<b><u>FAQ-COM090 : SETUP FOR FTP SERVER LOG FILES (proftpd, vsftpd, ...)</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
What do I have to do to use AWStats to analyze some FTP server log files ?<br>
<font class=CSolution>SOLUTION:</font><br>
AWStats can be used with some FTP server log files.<br>
<br>
<u>With ProFTPd</u>:<br>
<br>
1- Setup your server log file format:<br>
<br>
Modify the proftpd.conf file to add the following two lines :
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
<!--<br>With ProFTPd 1.2.2:-->
LogFormat awstats "%t %h %u %m %f %s %b" # WARNING: You must use a tab char between % tags and not a space !
<br>ExtendedLog /var/log/xferlog read,write awstats # WARNING: ExtendedLog directive might need to be placed inside a virtual host context if you use them.
<!--<br>With ProFTPd 1.2.6:
<i><br>LogFormat awstats ""${%F %H-%M-%S}t %h %u %m %F %s %b"</i> # WARNING: You must use a tab char between % tags and not a space !
<i><br>ExtendedLog /var/log/xferlog read,write awstats</i> # WARNING: ExtendedLog directive might need to be placed inside a virtual host context if you use them.
-->
</td></tr></table>
<br>
Then turn off old format Transfer log:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
TransferLog none # WARNING: TransferLog directive might need to be placed inside a virtual host context if you use them.
</td></tr></table>
<br>
To have the change effective, stop your server, remove old log file /var/log/xferlog and restart the server.<br>
Download a file by FTP and check that your new log file looks like this:<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
[01/Jan/2001:21:49:57 +0200] ftp.server.com user RETR /home/fileiget.txt 226 1499
</td></tr></table>
<br>
2- Then setup AWStats to analyze the FTP log file:<br>
<br>
Copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
Modify this new config file:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="/var/log/xferlog"
<br>LogType=F
<br>LogFormat="%time1 %host %logname %method %url %code %bytesd"
<br>LogSeparator="\t"
<br>NotPageList=""
<br>LevelForBrowsersDetection=0
<br>LevelForOSDetection=0
<br>LevelForRefererAnalyze=0
<br>LevelForRobotsDetection=0
<br>LevelForWormsDetection=0
<br>LevelForSearchEnginesDetection=0
<br>ShowLinksOnUrl=0
<br>ShowMenu=1
<br>ShowSummary=UVHB
<br>ShowMonthStats=UVHB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=HB
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=HBL
<br>ShowRobotsStats=0
<br>ShowEMailSenders=0
<br>ShowEMailReceivers=0
<br>ShowSessionsStats=1
<br>ShowPagesStats=PBEX
<br>ShowFileTypesStats=HB
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=0
<br>ShowOSStats=0
<br>ShowOriginStats=0
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=0
<br>ShowSMTPErrorsStats=0
</td></tr></table>
<br>
Now you can use AWStats as usual (run the update process and read statistics).<br>
<br>
<br>
<u>With vsFTPd, or any FTP server that log with xferlog format</u>:<br>
<br>
1- Check your server log file format:<br>
<br>
Take a look at your FTP server log file. You must have a format that match the following example to
use this FAQ :<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
Wed Jan 01 19:29:35 2001 1 192.168.1.1 102 /home/file1.txt b _ o r username ftp 0 * c
</td></tr></table>
<br>
2- Then setup AWStats to analyze the FTP log file:<br>
<br>
If your FTP log file format looks good, copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
Modify this new config file:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="/var/log/xferlog"
<br>LogType=F
<br>LogFormat="%time3 %other %host %bytesd %url %other %other %method %other %logname %other %code %other %other"
<br>LogSeparator="\s"
<br>NotPageList=""
<br>LevelForBrowsersDetection=0
<br>LevelForOSDetection=0
<br>LevelForRefererAnalyze=0
<br>LevelForRobotsDetection=0
<br>LevelForWormsDetection=0
<br>LevelForSearchEnginesDetection=0
<br>ShowLinksOnUrl=0
<br>ShowMenu=1
<br>ShowSummary=UVHB
<br>ShowMonthStats=UVHB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=HB
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=HBL
<br>ShowRobotsStats=0
<br>ShowEMailSenders=0
<br>ShowEMailReceivers=0
<br>ShowSessionsStats=1
<br>ShowPagesStats=PBEX
<br>ShowFileTypesStats=HB
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=0
<br>ShowOSStats=0
<br>ShowOriginStats=0
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=0
<br>ShowSMTPErrorsStats=0
</td></tr></table>
<br>
Now you can use AWStats as usual (run the update process and read statistics).<br>
<br>
<a name="MAIL"></a><br>
<b><u>FAQ-COM100 : SETUP FOR MAIL LOG FILES (Postfix, Sendmail, Qmail, MDaemon, Exchange...)</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
What do I have to do to use AWStats to analyze my mail log files ?<br>
<font class=CSolution>SOLUTION:</font><br>
<br>
This tip works with AWStats 5.5 or higher.<br>
<br>
<u>For Postfix, Sendmail, QMail or MDaemon log files</u><br>
<br>
You must setup AWStats to use a mail log file preprocessor (<i>maillogconvert.pl</i> is provided
into AWStats <i>tools</i> directory, but you can use the one of your choice):<br>
For this, copy config <i>"awstats.model.conf"</i> file to <i>"awstats.mail.conf"</i>.<br>
Modify this new config file:
For standard Postfix, Sendmail, MDaemon and standard QMail logfiles, set<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="perl /path/to/maillogconvert.pl standard < /pathtomaillog/maillog |"
</td></tr></table>
If the logfiles are compressed, they can be processed this way<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="gzip -cd /var/log/maillog.0.gz | /path/to/maillogconvert.pl standard |"<br>
</td></tr></table>
And for VAdmin QMail logfiles (multi-host/virtualhost mail servers running vadmin software), set<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="perl /path/to/maillogconvert.pl vadmin < /pathtomaillog/maillog |"<br>
</td></tr></table>
<br>
Then, whatever is you mail server, you must also change:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogType=M
<br>LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
<br>LevelForBrowsersDetection=0
<br>LevelForOSDetection=0
<br>LevelForRefererAnalyze=0
<br>LevelForRobotsDetection=0
<br>LevelForWormsDetection=0
<br>LevelForSearchEnginesDetection=0
<br>LevelForFileTypesDetection=0
<br>ShowMenu=1
<br>ShowSummary=HB
<br>ShowMonthStats=HB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=0
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=0
<br>ShowRobotsStats=0
<br>ShowEMailSenders=HBML
<br>ShowEMailReceivers=HBML
<br>ShowSessionsStats=0
<br>ShowPagesStats=0
<br>ShowFileTypesStats=0
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=0
<br>ShowOSStats=0
<br>ShowOriginStats=0
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=0
<br>ShowSMTPErrorsStats=1
</td></tr></table>
Warning: For MDaemon mail server, you must use the new MDaemon log file that ends
with "-Statistics.log".<br>
<br>
Now you can use AWStats as usual (run the update process and read statistics).<br>
<br>
<u>For Exchange log files</u><br>
<br>
Despite the high number of possible log format provided with Exchange,
none of them is built enough seriously to offer an interseting analyze (missing informations,
messy data, no id to join multiple records for same mail, etc...).
For this reason, an "exact" log analysis is a joke with Exchange log files.
However a little support is provided. In order to analyze Exchange traffic, you have to
enable "Message Tracking" (see article http://support.microsoft.com/default.aspx?scid=kb;EN-US;246856).<br>
Then copy config awstats.model.conf file to "awstats.mail.conf".<br>
Modify this new config file:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogType=M
<br>LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd"
<br>LevelForBrowsersDetection=0
<br>LevelForOSDetection=0
<br>LevelForRefererAnalyze=0
<br>LevelForRobotsDetection=0
<br>LevelForWormsDetection=0
<br>LevelForSearchEnginesDetection=0
<br>LevelForFileTypesDetection=0
<br>ShowMenu=1
<br>ShowSummary=HB
<br>ShowMonthStats=HB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=0
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=0
<br>ShowRobotsStats=0
<br>ShowEMailSenders=HBML
<br>ShowEMailReceivers=HBML
<br>ShowSessionsStats=0
<br>ShowPagesStats=0
<br>ShowFileTypesStats=0
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=0
<br>ShowOSStats=0
<br>ShowOriginStats=0
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=0
<br>ShowSMTPErrorsStats=1
</td></tr></table>
<br>
Also don't forget that with Exchange, informations in a log analyses can't be exact.
Do not send any questions or requests for using AWStats with Exchange, this is not a problem in
AWStats and we have no time to support non opened products.<br>
If you want to have complete and accurate information with Exchange, forget using AWStats or use a
more serious mail serveur (Postfix, Sendmail, QMail...)<br>
<br>
<a name="MEDIASERVER"></a><br>
<b><u>FAQ-COM110 : SETUP FOR A MEDIA SERVER (REALMEDIA, WINDOWS MEDIA SERVER, DARWIN STREAMING SERVER)</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
What do I have to do to use AWStats to analyze my Media Server log files.<br>
<font class=CSolution>SOLUTION:</font><br>
<br>
<u>For Realmedia</u><br>
<br>
Your log file will probably looks like this:<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
216.125.146.50 - - [16/Sep/2002:14:57:21 -0500] "GET cme/rhythmcity/rcitycaddy.rm?cloakport=8080,554,7070 RTSP/1.0" 200 6672 [Win95_4.0_6.0.9.374_play32_NS80_en-US_586] [80d280e1-c9ae-11d6-fa53-d52aaed98681] [UNKNOWN] 281712 141 3 0 0 494<br>
</td></tr></table>
<br>
Copy config awstats.model.conf file to "awstats.mediaserver.conf".
Modify this new config file:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="/pathtomediaserverlog/mediaserverlog"
<br>LogType=S
<br>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %uabracket %other %other %other %other %other %other %other %other"
<br>LogSeparator="\s+"
<br>ShowMenu=1
<br>ShowSummary=UHB
<br>ShowMonthStats=UHB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=HB
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=0
<br>ShowRobotsStats=0
<br>ShowEMailSenders=0
<br>ShowEMailReceivers=0
<br>ShowSessionsStats=0
<br>ShowPagesStats=PB
<br>ShowFileTypesStats=HB
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=1
<br>ShowOSStats=1
<br>ShowOriginStats=H
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=1
<br>ShowSMTPErrorsStats=0
</td></tr></table>
<br>
Now you can use AWStats as usual (run the update process and read statistics).<br>
<br>
<br>
<u>For Windows Media Server / Darwin Streaming Server</u><br>
<br>
1- If your Windows Media / Darwin streaming Server version allows it, setup your log format to write the following fields:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
c-ip
<br>date
<br>time
<br>cs-uri-stem
<br>c-starttime
<br>x-duration
<br>c-rate
<br>c-status
<br>c-playerid
<br>c-playerversion
<br>c-playerlanguage
<br>cs(User-Agent)
<br>cs(Referer)
<br>c-hostexe
<br>c-hostexever
<br>c-os
<br>c-osversion
<br>c-cpu
<br>filelength
<br>filesize
<br>avgbandwidth
<br>protocol
<br>transport
<br>audiocodec
<br>videocodec
<br>channelURL
<br>sc-bytes
</td></tr></table>
<br>
To make the change effective, stop your server, remove old log files and restart the server.<br>
Listen to streaming files and check that your new log file looks like this:<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
80.223.91.37 2002-10-08 14:18:58 mmst://mydomain.com/mystream 0 106 1 200 {F4A826EE-FA46-480F-A49B-76786320FC6B} 8.0.0.4477 fi-FI - - wmplayer.exe 8.0.0.4477 Windows_2000 5.1.0.2600 Pentium 0 0 20702 mms TCP Windows_Media_Audio_9 - - 277721
</td></tr></table>
<br>
<br>
If your Windows Media/Darwin Streaming Server version does not allow to define your log format:<br>
Just follow instructions in step 2 directly but use the log format string found in first lines
of your log files (Just after the "<i>#Fields:</i>" string) as value for AWStats LogFormat
parameter. For example, you could have a LogFormat defined like this:<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFormat="c-ip date time c-dns cs-uri-stem c-starttime x-duration c-rate
c-status c-playerid c-playerversion c-playerlanguage cs(User-Agent)
cs(Referer) c-hostexe c-hostexever c-os c-osversion c-cpu filelength
filesize avgbandwidth protocol transport audiocodec videocodec channelURL
sc-bytes c-bytes s-pkts-sent c-pkts-received c-pkts-lost-client
c-pkts-lost-net c-pkts-lost-cont-net c-resendreqs c-pkts-recovered-ECC
c-pkts-recovered-resent c-buffercount c-totalbuffertime c-quality s-ip s-dns
s-totalclients s-cpu-util"
</td></tr></table>
<br>This means you don't use the AWStats tags but AWStats can often also understand all the IIS and/or
Windows Media Server tags.<br>
<br>
2- Then setup AWStats to analyze your Media Server log:<br>
Copy config awstats.model.conf file to "awstats.mediaserver.conf".<br>
Modify this new config file:
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
LogFile="/pathtomediaserver/mediaserverlog"
<br>LogType=S
<br>LogFormat="c-ip date time cs-uri-stem c-starttime x-duration c-rate c-status c-playerid c-playerversion c-playerlanguage cs(User-Agent) cs(Referer) c-hostexe c-hostexever c-os c-osversion c-cpu filelength filesize avgbandwidth protocol transport audiocodec videocodec channelURL sc-bytes"
<br>DecodeUA=1
<br>ShowMenu=1
<br>ShowSummary=UHB
<br>ShowMonthStats=UHB
<br>ShowDaysOfMonthStats=HB
<br>ShowDaysOfWeekStats=HB
<br>ShowHoursStats=HB
<br>ShowDomainsStats=HB
<br>ShowHostsStats=HBL
<br>ShowAuthenticatedUsers=0
<br>ShowRobotsStats=0
<br>ShowEMailSenders=0
<br>ShowEMailReceivers=0
<br>ShowSessionsStats=0
<br>ShowPagesStats=PB
<br>ShowFileTypesStats=HB
<br>ShowFileSizesStats=0
<br>ShowBrowsersStats=1
<br>ShowOSStats=1
<br>ShowOriginStats=H
<br>ShowKeyphrasesStats=0
<br>ShowKeywordsStats=0
<br>ShowMiscStats=0
<br>ShowHTTPErrorsStats=1
<br>ShowSMTPErrorsStats=0
</td></tr></table>
<br>
Now you can use AWStats as usual (run the update process and read statistics).<br>
<br>
<a name="PERSONALIZEDLOG"></a><br>
<b><u>FAQ-COM115 : SETUP/EXAMPLES FOR LOGFORMAT PARAMETER</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
Which value do I have to put in the LogFormat parameter to make AWStats working with my log
file format ?<br>
<font class=CSolution>SOLUTION:</font><br>
The AWStats config file gives you all the possible values for LogFormat parameter.
To help you, this is some common cases of log file format, and
the corresponding value of LogFormat you must use in your AWStats config file:<br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined/XLF/ELF</b> log format):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
You must use : <i>LogFormat=1</i><br>
This is same than: <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined with several virtualhostname</b> sharing same log file).</u><br>
<i>virtualserver1 62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
You must use : <i>LogFormat="%virtualname %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined and mod_gzip format 1</b> with <b>Apache 1.x</b>):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" mod_gzip: 66pct.</i><br>
You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other %gzipratio"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined and mod_gzip format 2</b> with <b>Apache 1.x</b>):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" mod_gzip: DECHUNK:OK In:11393 Out:3904:66pct.</i><br>
You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other %other %gzipin %gzipout"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined and mod_deflate</b> with <b>Apache 2</b>):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" (45)</i><br>
You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %deflateratio"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA combined with 2 spaces between some fields</b> with <b>Zope</b>):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 3904 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" (45)</i><br>
You must use : <i><br>
LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"<br>
LogSeparator=" *"<br>
</i>
<hr>
<u>If your log records are EXACTLY like this (<b>NCSA common CLF</b> log format):</u><br>
<i>62.161.78.73 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234</i><br>
You must use : <i>LogFormat=4</i><br>
Note: Browsers, OS's, Keywords and Referers features are not available with a such format.<br>
<hr>
<u>If your log records are EXACTLY like this (With some <b>Squid</b> versions, after setting <i>emulate_http_log</i> to on):</u><br>
<i>200.135.30.181 - - [dd/mmm/yyyy:hh:mm:ss +0x00] "GET http://www.mydomain.com/page.html HTTP/1.0" 200 456 TCP_CLIENT_REFRESH_MISS:DIRECT</i><br>
You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %code %bytesd %other"</i><br>
<hr>
<u>If your log records are EXACTLY like this (Some old <b>IIS</b> W3C log format):</u><br>
<i>yyyy-mm-dd hh:mm:ss 62.161.78.73 - GET /page.html 200 1234 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html</i><br>
You must use : <i>LogFormat=2</i><br>
<hr>
<u>If your log records are EXACTLY like this (Some <b>IIS</b> W3C log format with some <b>.net</b> servers):</u><br>
<i>yyyy-mm-dd hh:mm:ss GET /page.html - 62.161.78.73 - Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html 200 1234 HTTP/1.1</i><br>
You must use : <i>LogFormat=2 (or LogFormat="%time2 %method %url %logname %host %other %ua %referer %code %bytesd %other")</i><br>
<hr>
<u>If your log records are EXACTLY like this (Some <b>IIS 6+</b> W3C log format):</u><br>
<i>yyyy-mm-dd hh:mm:ss GET /page.html - 62.161.78.73 - Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) http://www.from.com/from.html 200 1234</i><br>
You must use : <i>LogFormat=2 (or LogFormat="date time cs-method cs-uri-stem cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes")</i><br>
<hr>
<u>If your log records are EXACTLY like this (Some <b>ISA</b> W3C log format):</u><br>
<i>62.161.78.73, anonymous, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1), N, 1/1/2001, 0:00:16, W3ReverseProxy, HCSERV2, -, www.host.be, 192.168.141.101, 80, 266, 406, 10042, http, TCP, GET, http://192.168.141.101/, text/html, Inet, 200, 0x42330010, -, -</i><br>
You must use :<br>
<i>LogFile="sed -e 's/, /\t/g' "/yourlogpath/yourlogfile.log" |"</i><br>
<i>LogFormat=2</i><br>
<i>LogSeparator=" "</i><br>
<hr>
<u>If your log records are EXACTLY like this (With some <b>WebSite</b> versions):</u><br>
<i>yyyy-mm-dd hh:mm:ss 62.161.78.73 - 192.168.1.1 80 GET /page.html - 200 11205 0 0 HTTP/1.1 mydomain.com Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+98) - http://www.from.com/from.html</i><br>
You must use : <i>LogFormat="%time2 %host %logname %other %other %method %url %other %code %bytesd %other %other %other %other %ua %other %referer"</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>Webstar</b> native log format):</u><br>
<i>05/21/00 00:17:31 OK 200 212.242.30.6 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) http://www.cover.dk/ "www.cover.dk" :Documentation:graphics:starninelogo.white.gif 1133</i><br>
You must use : <i>LogFormat=3</i><br>
<hr>
<u>If your log records are EXACTLY like this (With some <b>Lotus Notes/Domino</b> versions):</u><br>
<i>62.161.78.73 - Name Surname Service [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
You must use : <i>LogFormat=6</i><br>
<hr>
<u>If your log records are EXACTLY like this (<b>Lotus Notes/Domino 6.x</b> log format):</u><br>
<i>62.161.78.73 - "Name Surname" Service [dd/mmm/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" 200 1234 "http://www.from.com/from.htm" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
You must use : <i>LogFormat="%host %other %lognamequot %time1 %methodurl %code %bytesd %refererquot %uaquot"</i><br>
<hr>
<u>If your log records are EXACTLY like this (With <b>Oracle9iAS</b>):</u><br>
<i>62.161.78.73 - [dd/mmm/yyyy:hh:mm:ss +0x00] GET /page.html HTTP/1.1 200 1234 - "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"</i><br>
Where separators are "tab" characters or several "spaces",
You must use : <i>LogFormat="%host %logname %time1 %method %url %other %code %bytesd %referer %uaquot"</i>
and <i>LogSeparator="\s+"</i><br>
<hr>
<u>If you use a FTP server like <b>ProFTPd</b>:</u><br>
See <a href="#FTP">FAQ-COM090</a>.<br>
<hr>
<u>If you want to analyze a mail log file (<b>Postfix</b>, <b>Sendmail</b>, <b>QMail</b>, <b>MDaemon</b>, <b>Exchange</b>):</u><br>
See <a href="#MAIL">FAQ-COM100</a>.<br>
<hr>
<u>If you use a Media Server (<b>Realmedia</b>, <b>Windows Media Server</b>):</u><br>
See <a href="#MEDIASERVER">FAQ-COM110</a>.<br>
<hr>
<u>If your log records are EXACTLY like this (With some providers):</u><br>
<i>62.161.78.73 - - [dd/Month/yyyy:hh:mm:ss +0x00] "GET /page.html HTTP/1.1" "-" 200 1234</i><br>
You must use : <i>LogFormat="%host %other %logname %time1 %methodurl %other %code %bytesd"</i><br>
Note: Browsers, OS's, Keywords and Referers features are not available with a such format.<br>
<hr>
<u>There are a lot of other possible log formats.</u><br>
You must use a personalized log format LogFormat ="..." as described in config file to
support other various log formats.<br>
<br><br>
<a name="ROTATE"></a><br>
<b><u>FAQ-COM120 : HOW TO ROTATE MY LOGS WITHOUT LOSING DATA</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I want to archive/rotate my logs using my server system (for example logrotate) options or a third
software (rotatelog, cronolog) but I don't want to lose any visits information during the rotate
process.<br>
<font class=CSolution>SOLUTION:</font><br>
<li> If your config file is setup with a <a href="awstats_config.html#LogFile">LogFile</a> parameter
that point to your current running log file (required if you want to use the
<a href="awstats_config.html#AllowToUpdateStatsFromBrowser">AllowToUpdateStatsFromBrowser</a>
option to have "real-time" statistics), to avoid losing too much records during the rotate
process, you must run the AWStats update JUST BEFORE the rotate process is done.<br>
The best way to do that on 'Linux like' OS is to use the linux built-in logrotate feature. You must
edit the logrotate config file used for your web server log file (usually stored in /etc/logrotate.d
directory) by adding the AWStats update process as a preprocessor command, like this example (bold
lines are lines to add for having a prerotate process):<br>
<table width="95%" border=1 cellpadding=0 cellspacing=0 bgcolor=#F4F4F4 class=CFAQ><tr class=CFAQ><td class=CFAQ>
/usr/local/apache/logs/*log<br>
{<br>
notifempty<br>
daily<br>
rotate 7<br>
compress<br>
<b>
sharedscripts<br>
prerotate<br>
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=mydomainconfig<br>
endscript<br>
</b>
postrotate<br>
/usr/bin/killall -HUP httpd<br>
endscript<br>
}<br>
</td></tr></table>
<br>
If using a such solution, this is sequential steps that happens:<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ>
<tr class=CFAQ><td class=CFAQ><b>Step</b></td><td class=CFAQ><b>Description</b></td><td class=CFAQ><b>Step name</b></td><td class=CFAQ><b>Date/Time example</b></td></tr>
<tr class=CFAQ valign=top><td class=CFAQ>A</td><td class=CFAQ>logrotate is started (by cron)</td><td class=CFAQ>Start of logrotate</td><td class=CFAQ>04:02:00</td></tr>
<tr valign=top><td class=CFAQ>B</td><td class=CFAQ><ul>awstats -update is launched by logrotate</td><td class=CFAQ>Start of awstats</td><td class=CFAQ>04:02:01</td><tr>
<tr valign=top><td class=CFAQ>C</td><td class=CFAQ><ul><ul>awstats start to read the log file <i>file.log</i></td><td class=CFAQ> </td><td class=CFAQ>04:02:02</td><tr>
<tr valign=top><td class=CFAQ>D</td><td class=CFAQ><ul>awstats has reached the end of log file so now it starts to save its database on disk.</td><td class=CFAQ> </td><td class=CFAQ>04:05:00</td><tr>
<tr valign=top><td class=CFAQ>E</td><td class=CFAQ><ul>awstats has finished to save its new database, so it stops</td><td class=CFAQ>End of awstats</td><td class=CFAQ>04:06:00</td><tr>
<tr valign=top><td class=CFAQ>F</td><td class=CFAQ>logrotate moves old log file <i>file.log</i> to a new name <i>file.log.sav</i>. Apache now logs in this file <i>file.log.sav</i> since log file handle has not been changed (only log file name has been renamed).</td><td class=CFAQ>Log move</td><td class=CFAQ>04:06:01</td><tr>
<tr valign=top><td class=CFAQ>G</td><td class=CFAQ>logrotate sends the -HUP or -USR1 signal to Apache.<br>With -HUP, Apache immediatly kills all its child process/thread, close log file <i>file.log.sav</i>, and reopen file <i>file.log</i>. So now, ALL hits are written to new file.<br>With -USR1, Apache only ask its child process/thread to stop only when HTTP request will be completely served. However it closes immediatly log file <i>file.log.sav</i>, and reopen file <i>file.log</i>. So only NEW hits are written to new log file. HTTP requests that are still running will write in old one.</td><td class=CFAQ>Apache restart</td><td class=CFAQ>04:06:02</td><tr>
<tr valign=top><td class=CFAQ>H</td><td class=CFAQ>logrotate starts compress the old log file <i>file.log.sav</i> into <i>file.log.gz</i></td><td class=CFAQ>Start compress</td><td class=CFAQ>04:06:03</td><tr>
<tr valign=top><td class=CFAQ>I</td><td class=CFAQ><ul>If some apache threads/processes are still running (because the kill sent was -USR1, so child process are waiting end of request before to stop), then those threads/processes are still writing to <i>file.log.sav</i>.<br>If kill -HUP was used, all process are already restarted so all writes in new <i>file.log</i>.</td><td class=CFAQ> </td><td class=CFAQ> </td><tr>
<tr valign=top><td class=CFAQ>J</td><td class=CFAQ>logrotate has finished to compress log file into <i>file.log.gz</i>. File <i>file.log.sav</i> is deleted.</td><td class=CFAQ>End of compress<br>End of logrotate</td><td class=CFAQ>04:07:03</td><tr>
<tr valign=top><td class=CFAQ>K</td><td class=CFAQ><ul>If signal was -USR1, some old childs can still run (when serving a very long request for example). So the log writing, still done in same file handle are going to a file that has been removed. So log writing are lost nowhere (this is only if -USR1 was used and if request was very long).</td><td class=CFAQ> </td><td class=CFAQ> </td><tr>
</table>
<br>
The advantage of this solution is that it is a very common way of working, used by a lot of
products, and easy to setup. You will notice that you can "lose" some hits:<br>
If you use the -HUP signal, you will only lose all hits that were written during D and E.
Note that you will also break all requests still running at G. In the example, it's a
1 minute lost (for small or medium web sites, it will be less than few seconds), so this
give you an error lower than 0.07% (less for small web sites). This is not significant,
above all for a "statistics" progam.<br>
If you use the -USR1 signal, you will not kill any request. But you will lose all hits that
were wrote during D and E (like with -HUP) but also all hits that are still running after H
(all very long request that requires several minutes to be served). If hit ends during I, it is
wrote in a log file already analyzed, if hit ends at K, it is wrote nowhere. In the example,
it's also a 0.07% error plus error for other not visible hits that were finished during I or K,
but number of such hits should be very low since only hits that started before G and not
finished after H are concerned. In most cases a hits needs only few milliseconds to be served
so lost hits could be ignored.<br>
<br>
Note also that if you have x logrotate config files, with each of them a postrotate with a
kill -HUP, you send a kill x times to your server process. So try to include several log files
in same logrotate config file. You can have several awstats update command in the same
prerotate section and you will send the -HUP only once, after all updates are finished.
However, doing this, you will have a lap time between D and F (were some hits are lost) that will
be higher.<br>
<br>
<li> Another common way of working is to choose to run the AWStats update process only once the log file has been
archived.<br>
This is required for example if you use the <a href="http://cronolog.org" target=awstatsbis>cronolog</a>
or rotatelog tools to rotate your log files. For example, Apache users can setup their Apache
httpd config file to write log file through a pipe to cronolog or rotatelog using
Apache <i>CustomLog</i> directive:<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
CustomLog "|/usr/sbin/cronolog [cronolog_options] /var/logs/access.%Y%m%d.log" combined
</td></tr></table>
If you use a such feature, you can't trigger AWStats update process to be ran just BEFORE the
rotate is done, so you must run it AFTER the rotate process, so on the archived log file.<br>
To setup awstats to always point to last archive log file, you can use the 'tags' available for
<a href="awstats_config.html#LogFile">LogFile</a>.<br>
The problem with that is that your data are refreshed only after a rotate has done. However,
you will miss absolutely nothing (no hits) and your server processes are never killed.<br>
<br>
<li> So, if you really want to not lose absolutely no hit and want to have updates more
frequently than the rotate frequency, the best way is still an hybrid solution (i am not sure
that it worth the pain, and remember that statistics are only statistics):<br>
You run the awstats update process from you crontab frequently, every hour for example, and half and hour
before the rotate has done. See next FAQ to know how to setup a scheduled job.<br>
Then, once the rotate has been done (by the logrotate or by a piped cronolog log file), and
before the next scheduled awstats update process start,
you run another update process on the archived log file using the -logfile option to force
update on the archived log file and not the current log file defined in awstats
config file. This will allow you to update the half hour missing,
until the log rotate (AWStats will find the new lines). However don't forget that this
particular update MUST be finished before the next croned update.<br>
<br>
<a name="CRONTAB"></a><br>
<b><u>FAQ-COM130 : HOW TO RUN AWSTATS UPDATE PROCESS FREQUENTLY</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
AWStats must be ran frequently to update statistics. How can I do this ?<br>
<font class=CSolution>SOLUTION:</font><br>
A good way of working is to run the AWStats update process as a preprocessor of your log rotate
process. See previous FAQ (<a href="#ROTATE">FAQ-COM120</a>) for this.<br>
But you can also run AWStats update process regularly by a scheduler:<br>
<br>
<u>With Windows</u>, you can use the internal task scheduler.
The use of this tool is not an AWStats related problem, so please take a look at your Windows manual.
Warning, if you use <i>"awstats.pl -config=mysite -update"</i> in your scheduled task, you might
experience problem of failing task. Try this instead<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
"C:\WINNT\system32\CMD.EXE /C C:\[awstats_path]\awstats.pl -config=mysite -update"
</td></tr></table>
or<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
"C:\[perl_path]\perl.exe C:\[awstats_path]\awstats.pl -config=mysite -update"
</td></tr></table>
A lot of other open source schedulers are often better (otherwise there is also good sharewares or freewares).<br>
<br>
<u>With unix-like operating systems</u>, you can use the "<b>crontab</b>".<br>
This is examples of lines you can add in the cron file (see your unix reference manual for cron) :<br>
To run update every day at 03:50, use :<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
50 3 * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update >/dev/null<br>
</td></tr></table>
To run update every hour, use :<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
0 * * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -config=mysite -update >/dev/null<br>
</td></tr></table>
<br>
<a name="EXCLUDEHOSTS"></a><br>
<b><u>FAQ-COM140 : HOW CAN I EXCLUDE MY IP ADDRESS (OR WHOLE SUBNET MASK) FROM STATS ?</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I don't want to see my own IP address in the stats or I want to exclude counting visits from a whole subnet.<br>
<font class=CSolution>SOLUTION:</font><br>
You must edit the config file to change the <a href="awstats_config.html#SkipHosts">SkipHosts</a> parameter.<br>
For example, to exclude:<br>
<li> your own IP address 123.123.123.123, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="123.123.123.123"<br>
<li> the whole subnet 123.123.123.xxx, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="REGEX[^123\.123\.123\.]"<br>
<li> all sub hosts xxx.myintranet.com, use <a href="awstats_config.html#SkipHosts">SkipHosts</a>="REGEX[\.myintranet\.com$]" (This one works only if DNS lookup is already done in your
log file).<br>
<br>
<a name="SCREENSIZE"></a><br>
<b><u>FAQ-COM142 : HOW TO GET THE SCREEN SIZE AND BROWSER CAPABILITIES REPORT WORKING ?</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I see in the AWStats features list that it can report the screen size used by visitors and other browsers' informations,
(like if browser support Flash, Java, Javascript, PDF, MAcromedia, Audio plugins, etc...). How can I do that ?<br>
<font class=CSolution>SOLUTION:</font><br>
AWStats is a log analyzer, so to report the screen size of your visitor, we need to have information inside the log file itself.
For this, the only way to do that, is to add some HTML tags inside some of your pages (the home page is enough to
get use ratios). This tag will add call to a javascript that ask your browser to get a virtual URL that
includes, in its parameters, the screen size resolution and all other informations about browser capabilities (Flash, Java, Javascript, PDF, Macromedia, Audio plgins...).<br>
<br>
This is the code you must add (at bottom of your home page for example) :<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
<script language="javascript" type="text/javascript" src="/js/awstats_misc_tracker.js" ></script><br>
<noscript><img src="/js/awstats_misc_tracker.js?nojs=y" height="0" width="0" border="0" style="display: none"></noscript><br>
</td></tr></table>
<br>
Note that you must also place the <i>awstats_misc_tracker.js</i> script (provided in /js directory with AWStats) inside
a js directory stored in your web root.<br>
Once this is done, load your home page with your browser and go to check that inside your log file
if you can see a line that looks like that:<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
123.123.123.123 - - [24/Apr/2005:16:09:38 +0200] "GET /js/awstats_misc_tracker.js?screen=800x600&win=724x517&...&sid=awssession_id123 HTTP/1.1" 200 6237 "http://therefererwebsite.com/index.php" "Mozilla/5.0 (Linux) Gecko/20050414 Firefox/1.0.3"
</td></tr></table>
<br>
If yes, you can then run the AWStats update process. Screen sizes information will be analyzed. All you have to do
now is to edit your config file to tell AWStats to add the report on html output. For this, change
the <a href="awstats_config.html#Show">ShowMiscStats</a> parameter.<br>
<table border=1 cellpadding=1 cellspacing=0 bgcolor=#F4F4F4 width="95%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
ShowMiscStats=anjdfrqwp
</td></tr></table>
<br>
Note: If you change directory where <i>awstats_misc_tracker.js</i> is stored (somewhere else than the /js directory),
you must modify, according to your change:<br>
- the html tags added<br>
- the line: <i>var awstatsmisctrackerurl="/js/awstats_misc_tracker.js";</i> inside the awstats_misc_tracker.js script<br>
- the parameter <a href="awstats_config.html#MiscTrackerUrl">MiscTrackerUrl</a> inside AWStats configuration file.<br>
<br>
<a name="EXTRA"></a><br>
<b><u>FAQ-COM145 : HOW TO USE THE EXTRA SECTIONS FEATURES ?</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
I want to build personalized reports not provided in default AWStats reports. How can I setup
the Extra Sections parameters in my AWStats config file to do so ?<br>
<font class=CSolution>SOLUTION:</font><br>
Take a look at the <a href="awstats_extra.html">Using AWStats Extra Sections features</a><br>
<br>
<a name="BENCHMARK"></a><br>
<b><u>FAQ-COM150 : BENCHMARK / FREQUENCY TO LAUNCH AWSTATS TO UPDATE STATISTICS</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
What is AWStats speed ?<br>
What is the frequency to launch AWStats process to update my statistics ?<br>
<font class=CSolution>SOLUTION:</font><br>
All benchmarks information and advice on frequency for update process are related into
the <a href="awstats_benchmark.html">Benchmark page</a>.<br>
<br>
<a name="DNS"></a><br>
<b><u>FAQ-COM200 : HOW REVERSE DNS LOOKUP WORKS, UNRESOLVED IP ADDRESSES</u></b><br>
<font class=CProblem>PROBLEM:</font><br>
The reported page AWStats shows me has no hostnames, only IP addresses, countries reported are all "unknown".<br>
<font class=CSolution>SOLUTION:</font><br>
When AWStats find an IP address in your log file, it tries a reverse DNS lookup to find
the hostname and domain if the <a href="awstats_config.html#DNSLookup">DNSLookup</a> parameter, in your AWStats config file, is <a href="awstats_config.html#DNSLookup">DNSLookup</a>=1
(Default value). So, first, check if you have the good value. The <a href="awstats_config.html#DNSLookup">DNSLookup</a>=0
must be used only if your log file contains already resolved IP address. For example,
when you set up Apache with the <i>HostNameLookups=on</i> directive. When you
ask your web server to make itself the reverse DNS lookup to log hostname instead
of IP address, you will still find some IP addresses in your log file because
the reverse DNS lookup is not always possible. But if your web server fails in
it, AWStats will also fails (All reverse DNS lookups use the same system API).