forked from apache/chukwa
/
CHANGES.txt
951 lines (486 loc) · 38.1 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
Chukwa Change Log
Trunk (unreleased changes)
NEW FEATURES
CHUKWA-635. Collect swap usage. (Eric Yang)
CHUKWA-664. Added network compression between agent and collector. (Sourygna Luangsay via Eric Yang)
IMPROVEMENTS
CHUKWA-648. Make Chukwa Reduce Type to support hierarchy format. (Jie Huang via asrabkin)
CHUKWA-650. Re-configure Demux ReduceNumber without re-starting the DemuxManager service. (Jie Huang via asrabkin)
CHUKWA-649. Handle pid-file missing case while stoping chukwa processes. (Jie Huang via asrabkin)
CHUKWA-645. Updated Hadoop and HBase dependency. (Eric Yang)
CHUKWA-630. Added chukwa user setup to rpm package. (Noel Duffy via Eric Yang)
CHUKWA-639. Updated to HBase 0.92.1. (Eric Yang)
CHUKWA-647. Spread out intermediate data with the same ReduceType into different Reduce Tasks. (Jie Huang via asrabkin)
BUGS
CHUKWA-668. Fix infinite loop in FileTailingAdaptor. (Sourygna Luangsay via Eric Yang)
CHUKWA-653. Added pre-compile JSP code profile for build process. (Jie Huang via Eric Yang)
CHUKWA-642. Added regular expression validation. (Eric Spishak via Eric Yang)
CHUKWA-632. Added getter method for startOffset in AbstractProcessor. (Abhijit Dhar via Eric Yang)
CHUKWA-637. Fixed default cluster selection in hicc. (Eric Yang)
CHUKWA-654. Added additional check for getting row key. (Saisai Shao via Eric Yang)
CHUKWA-652. Fixed JSON error in utilization.descriptor. (Saisai Shao via Eric Yang)
CHUKWA-646. Fix file rotation correctly for 0 byte rotated files. (Ivy Tang via Eric Yang)
CHUKWA-643. Updated Jersey dependency. (Prakhar Srivastava via Eric Yang)
CHUKWA-641. Fix stack trace for dumpChunk command when invalid regular expression is specified. (Eric Spishak via Eric Yang)
CHUKWA-636. Updated dependency to HBase 0.92.0. (Eric Yang)
CHUKWA-638. Fixed timezone offset in test case. (Eric Yang)
CHUKWA-300. Updated adaptor list in document. (Eric Yang)
Release 0.5
NEW FEATURES
CHUKWA-590. Migrate from ant to maven. (Eric Yang)
CHUKWA-599. Added Metrics 2 support for Hadoop 0.20.203+. (Eric Yang)
CHUKWA-575. Added cluster summary pig script. (Eric Yang)
CHUKWA-571. Add digested system metrics to SystemMetrics demux parser. (Eric Yang)
CHUKWA-563. Added configuration to skip HBase schema validation for HBaseWriter. (Eric Yang)
CHUKWA-558. Update default Chukwa configuration files to use SocketAdaptor for Hadoop metrics streaming. (Eric Yang)
CHUKWA-527 Configuration spellcheck. (Ari Rabkin)
CHUKWA-520. Implemented graphing rest api and graph explorer for plotting HBase data to graph. (Eric Yang)
CHUKWA-530. Implemented Syslog Adaptor for mappig Syslog facility name to Chukwa data type. (Eric Yang)
CHUKWA-419. Replaced system metrics collection with Sigar. (Eric Yang)
CHUKWA-444. Added HBaseWriter for storing time series data in HBase for faster random read/write. (Eric Yang)
CHUKWA-473. Make default processor configurable. (Bill Graham via asrabkin)
CHUKWA-479. Support HTTP trigger actions (Bill Graham via asrabkin)
CHUKWA-469. Add JMSAdaptor. (Bill Graham via asrabkin)
CHUKWA-477. Support post-demux trigger. (Bill Graham via Eric Yang)
IMPROVEMENTS
CHUKWA-629. Updated Hadoop dependency to Hadoop 1.0.0 (Eric Yang)
CHUKWA-627. Updated DISCLAIMER.txt and LICENSE.txt (Eric Yang)
CHUKWA-626. Updated NOTICE.txt to include dependency licenses. (Eric Yang)
CHUKWA-624. Added md5 signature signing to maven build for release tarball. (Eric Yang)
CHUKWA-623. Added build profile src for building release source tarball. (Eric Yang)
CHUKWA-622. Updated HDFS instruction for collector configuration. (Eric Yang)
CHUKWA-620. Updated code analysis and license check in build structure. (Eric Yang)
CHUKWA-618. Added HICC programming guide. (Eric Yang)
CHUKWA-522. Added HICC User guide. (Eric Yang)
CHUKWA-615. Changed HICC default view to monitor Hadoop. (Eric Yang)
CHUKWA-612. Convert Chukwa document from forrest format to apt format. (Eric Yang)
CHUKWA-608. Added Quick start guide. (Ahmed Fathalla via Eric Yang)
CHUKWA-605. Update directory structure to be aligned with Hadoop. (Eric Yang)
CHUKWA-601. Removed legacy mysql upgrade script. (Eric Yang)
CHUKWA-596. Updated schema retrieve REST API to use end time for scanning columns. (Eric Yang)
CHUKWA-586. Updated HBase dependency to HBase 0.90.3. (Eric Yang)
CHUKWA-569. Added temperory fix for load meta data faster for graph_explorer.jsp. (Eric Yang)
CHUKWA-573. Setup ChukwaMetrics to send to port 9097 and use ChukwaMetricsProcessor for parsing. (Eric Yang)
CHUKWA-572. Changed disk and network stats to gauge instead of counter. (Eric Yang)
CHUKWA-554. Remove jdbc.conf dependency from Chukwa. (Eric Yang)
CHUKWA-562. Added RegExDirTailingAdaptor. (Deepak Deshpande via Eric Yang)
CHUKWA-474. Replaced JSON library with JSON-Simple. (Eric Yang)
CHUKWA-560. Improve HBaseWriter performance. (Jerome Boulon via Eric Yang)
CHUKWA-555. Update descriptor files to use HBase REST API. (Eric Yang)
CHUKWA-557. Improve metrics differentation between recordName and contextName and clean up chukwa metrics. (Eric Yang)
CHUKWA-547. Adjustable Axes on HICC graphs. (Michael Chang via asrabkin)
CHUKWA-548. Gray out options with no data. (Michael Chang via asrabkin)
CHUKWA-546. SocketAdaptor. (Eric Yang via asrabkin)
CHUKWA-493. Change release artifact naming. (Eric Yang via asrabkin)
CHUKWA-525. Extensible DirTailingAdaptor. (Jaydeep Ayachit via asrabkin).
CHUKWA-531. Bundle HBase 0.20.6 in library path, and comment out test case for HBaseWriter for now. (Eric Yang)
CHUKWA-524. Use TODO-HBASE-HOME and TODO-HBASE-CONF-DIR to control hbase location. (Eric Yang)
CHUKWA-521. Changed default HICC port to 4080. (Eric Yang)
CHUKWA-506. Added retry logic for MetricDataLoader (William Bajzek via Eric Yang)
CHUKWA-507. Imrpoved url validation for JMSAdaptor. (Bill Graham via Eric Yang)
CHUKWA-501. Newer version of Ivy, pull in newer deps. (Eric Yang via asrabkin)
CHUKWA-504. Ability to delete file in FileAdaptor after successfully uploaded. (Kirk True via asrabkin)
CHUKWA-495. Support for pig 0.7. (Bill Graham via asrabkin)
CHUKWA-490. Enabling mdl.xml to be split into multiple files for ease of maintenance. (Kirk True via Eric Yang)
CHUKWA-480. Allow CreateRecordFile to read job confs. (Bill Graham via asrabkin)
CHUKWA-471. Expose JobConf to Demux Processors. (Jerome Boulon via asrabkin)
CHUKWA-472. Added ability to configure timestamp format for the default Time Series Processor. (Bill Graham via Eric Yang)
BUG FIXES
CHUKWA-628. Updated LICENSE.txt and NOTICE.txt to meet IPMC request. (Eric Yang)
CHUKWA-621. Implemented findbugs and pmd code analysis check. (Eric Yang)
CHUKWA-619. Disabled trace method on Chukwa servlets. (Julio Conca via Eric Yang)
CHUKWA-617. Fixed test case casting and race conditions. (Eric Yang)
CHUKWA-616. Updated release note and change log generation. (Eric Yang)
CHUKWA-614. Updated parent iframe resize code. (Eric Yang)
CHUKWA-613. Removed Jersey 1.1 from source code library path. (Eric Yang)
CHUKWA-611. Updated Chukwa tarball to include chukwa-0.5.0-client.jar. (Eric Yang)
CHUKWA-606. Updated HICC to interface with data in HBase. (Eric Yang)
CHUKWA-588. Updated hbase.schema to match Hadoop 0.20.205.0 metrics and job summary. (Eric Yang)
CHUKWA-609. Map / to /hicc for HICC web server. (Eric Yang)
CHUKWA-593. Fixed infinite loop archiving at midnight. (Sourygna Luangsay via Eric Yang)
CHUKWA-602. Set svn ignore properties to ignore eclipse and maven files. (Eric Yang)
CHUKWA-597. Close stderr and stdout after parent process detached. (Eric Yang)
CHUKWA-38. Download Hadoop jar files from ivy. (Eric Yang)
CHUKWA-595. Fixed buffering time calculation for Agent to send data to Collector. (Eric Yang)
CHUKWA-594. Fix row key matching in HICC web interface. (Preetam Patil via Eric Yang)
CHUKWA-592. Enable test case for HBaseWriter. (Eric Yang)
CHUKWA-591. Check for empty result from HBase Scanner. (Eric Yang)
CHUKWA-585. Enable HICC server to bind to all IP addresses. (Eric Yang)
CHUKWA-582. Change MoveToRepository to ignore anything starting with _. (Ari Rabkin)
CHUKWA-587. Improved default cluster selection code. (Eric Yang)
CHUKWA-579. Package initial_adaptors config file for tarball. (Eric Yang)
CHUKWA-577. Convert HadoopMetricsProcessor to use Simple JSON library. (Eric Yang)
CHUKWA-574. Added locking for recording metrics through log4j. (Eric Yang)
CHUKWA-568. Moved from json.jar library to JSON Simple library for SystemMetrics demux parser and add tags column family. (Eric Yang)
CHUKWA-231. Update default view to show graph explorer and remove deprecated widgets. (Eric Yang)
CHUKWA-7. Handle pid in /tmp in case CHUKWA_HOME is not defined. (Ari Rabkin via Eric Yang)
CHUKWA-553. Retain file filter argument for DirTailingAdaptor. (Deepak Deshpande via Eric Yang)
CHUKWA-559. Switched unit test HICC port to 4080. (Eric Yang)
CHUKWA-543. Parse only .view files for processing user view files. (Eric Yang)
CHUKWA-552. Handle % signs in series names. (Michael Chang via asrabkin)
CHUKWA-544. HICC graph should handle single series or multiple series more gracefully. (Eric Yang via asrabkin)
CHUKWA-447. Correct time units in ExecAdaptor. (Eric Yang via asrabkin)
CHUKWA-542. Disable hbase jar download from maven, and include hicc rest api in HICC web.xml (Eric Yang)
CHUKWA-516. Downgrade to HBase 0.20.6 until 0.90 arrives. (Eric Yang)
CHUKWA-496. Fixed config file reference in design.html (Bernd Fondermann via Eric Yang)
CHUKWA-510. JMSMessagePropertyTransformer inserts "null" string for null JMS message property values (Bill Graham via asrabkin)
CHUKWA-505. Don't include duplicate classes in chukwa-core.jar. (Bill Graham via asrabkin)
CHUKWA-487. Collector left in a bad state after temprorary NN outage. (Bill Graham via asrabkin)
CHUKWA-499. JMSAdaptor misses last argument in some cases when parsing. (Bill Graham via asrabkin)
CHUKWA-498. Agent logs incorrect number of chunks sent. (Bill Graham via asrabkin)
CHHUKWA-494. NPE when no time range is specified. (Bernd Fondermann via asrabkin)
CHUKWA-492. Correction to database setup instruction. (Bernd Fondermann via Eric Yang)
CHUKWA-489. Fixed malformed SQL in MetricDataLoader when values contain single quote character. (Kirk True via Eric Yang)
CHUKWA-488. Filter user customized jar file path URL. (Kirk True via Eric Yang)
CHUKWA-483. Use SIGTERM to stop chukwa, not signal 1. (Bill Graham via asrabkin)
CHUKWA-4. Collectors don't finish writing .dones on close. (Ahmed Fathalla via asrabkin)
CHUKWA-485. Fix TSProcessor handling of some dates. (Bill Graham via asrabkin)
CHUKWA-478. Fixed TestSocketTee intermittent failure. (Chris Douglas via Eric Yang)
Release 0.4
NEW FEATURES
CHUKWA-445. Realtime display at collector. (asrabkin)
CHUKWA-454. DirTailingAdaptor can filter files. (Gerrit Jansen van Vuuren via asrabkin)
CHUKWA-449. Utility to generate sequence file from log file. (Bill Graham via asrabkin)
CHUKWA-440. Enable addon jar file for Demux from Distributed Cache. (Eric Yang)
CHUKWA-448. Write-ahead buffering for arbitrary adaptors. (asrabkin)
CHUKWA-441. Added real time Hadoop activity monitor. (Eric Yang)
CHUKWA-433. File-per-post writer for benchmark purposes. (asrabkin)
CHUKWA-431. UDP Adaptor. (asrabkin)
CHUKWA-46. Ability to allow only local connections to agent control port. (asrabkin)
CHUKWA-427. Automate FSM state calculation and load to database. (Eric Yang)
CHUKWA-424. FSMBuilder driver script. (Jiaqi Tan via asrabkin)
CHUKWA-395. Support for generalized buffering of adaptor data. (asrabkin)
CHUKWA-405 Add a "stop all" command. (asrabkin)
IMPROVEMENTS
CHUKWA-468. Use SIGHUP instead of SIGTERM for shutdown. (Eric Yang)
CHUKWA-470. License headers. (asrabkin)
CHUKWA-458. Documentation fixes for 0.4 (asrabkin)
CHUKWA-446. Refactor start/stop scripts. (Eric Yang)
CHUKWA-450. Ability to turn off sort in dumpchunks. (asrabkin)
CHUKWA-359. Report detailed error on adaptor start failure. (asrabkin)
CHUKWA-420. Clean up adaptor stop methods. (asrabkin)
CHUKWA-437. Clean up widget category labels. (Eric Yang)
CHUKWA-421. Use modification time to detect rotation. (asrabkin)
CHUKWA-432. PipelineableWriter becomes an abstract class. (asrabkin)
CHUKWA-429. Update HDFS heatmap color with rainbow colors. (Eric Yang)
CHUKWA-418. Standardize logging. Mark adaptor IDs. (asrabkin)
CHUKWA-207. Store HICC dashboard data in HDFS. (Eric Yang)
CHUKWA-415. Read initial adaptors at start every time. (asrabkin)
CHUKWA-413. Improve admin guide. (asrabkin)
CHUKWA-345. Remove redundant 'application' field from Chunk API. (asrabkin)
CHUKWA-409. Make SocketTeeWriter work in single-stage pipeline. (Thushara Wijeratna via asrabkin)
CHUKWA-408. Add optional metadata to real-time feed. (asrabkin)
BUG FIXES
CHUKWA-523. Fix reload collectors. (Eric Yang via asrabkin)
CHUKWA-508. Include contrib directory build files. (Bill Graham via Eric Yang)
CHUKWA-465. Setup default value for Post Demux Data Loader. (Ahmed Fathalla via Eric Yang)
CHUKWA-467. Disabled war file extraction to /tmp. (Eric Yang)
CHUKWA-456. Remove /tmp/chukwa/hicc during test phase to prevent bad data building up and prevent test case to run properly. (Eric Yang)
CHUKWA-460. Removed dependency of writer.hdfs.filesystem for demux phase. (Eric Yang)
CHUKWA-466. Fix failing tests. (asrabkin)
CHUKWA-457. Fix TestUserResource. (Eric Yang via asrabkin)
CHUKWA-463. Fix pattern match for tags. (Guillermo Perez via asrabkin)
CHUKWA-464. Fix spelling of processClusterDirectory. (Guillermo Perez via asrabkin)
CHUKWA-456. Fixed failing unit test. (Eric Yang)
CHUKWA-455. Updated slaves.sh to detect collectors list. (Guillermo Pérez)
CHUKWA-453. Specify bash instead of sh in script headers. (Gerrit Jansen van Vuuren via asrabkin)
CHUKWA-451. Shouldn't use ssh to start processes locally. (Bill Graham via asrabkin)
CHUKWA-452. Fix support for using pig on final archives. (Gerrit Jansen van Vuuren via asrabkin)
CHUKWA-443. Updated HDFS metrics logging filename. (Eric Yang)
CHUKWA-442. Updated Apache license for HICC Rest API. (Eric Yang)
CHUKWA-438. Fixed column resizing issue in HICC (Eric Yang)
CHUKWA-439. Added blank image for HICC. (Eric Yang)
CHUKWA-436. Include time widget related javascript in the dashboard. (Eric Yang)
CHUKWA-430. Narrow down the list of demux output for FSM to improve processing time. (Eric Yang)
CHUKWA-428. Revised location for Chukwa HDFS repository. (Eric Yang)
CHUKWA-241. Revise chukwa-config.sh so that chukwa runs after building from source. (Eric Yang via asrabkin)
CHUKWA-2. Removed the default /var/log/messages streaming. (Eric Yang)
CHUKWA-414. Better error messaging for pid file errors. (Bill Graham via asrabkin)
CHUKWA-411. Hosts for DataNode ClientTrace messages parsed incorrectly. (Jiaqi Tan via asrabkin)
CHUKWA-229. Fix file descriptor leak in ExecAdaptor. (asrabkin)
Release 0.3
NEW FEATURES
CHUKWA-344. State-Machine Generation for input to SALSA visualizations. (Jiaqi Tan via asrabkin)
CHUKWA-384. Added REST API charting capability. (Eric Yang)
CHUKWA-369. Tolerance of collector failures. (asrabkin)
CHUKWA-368. New data integrity validation tool. (asrabkin)
CHUKWA-383. Added embed mode for HICC. (Eric Yang)
CHUKWA-382. Added export button to export HICC graph as static image. (Eric Yang)
CHUKWA-366. Custom tags. (asrabkin)
CHUKWA-358. Real-time monitoring at collector. (asrabkin)
CHUKWA-352. Xtrace in contrib. (asrabkin)
CHUKWA-346. Simplified sink archiver. (asrabkin)
CHUKWA-343. Static HDFS Heatmap visualization. (Jiaqi Tan via asrabkin)
CHUKWA-342. Static swimlanes visualization widget. (Jiaqi Tan via asrabkin)
CHUKWA-299. Added HDFS Spatial heatmaps visualization. (Jiaqi Tan via Eric Yang)
CHUKWA-185. Ability to tail a whole directory. (asrabkin)
CHUKWA-280. Added end to end test to detect iostat overflow. (Eric Yang)
CHUKWA-194. Backfilling tools. (Jerome Boulon via asrabkin)
CHUKWA-253. Added aggregations by user. (Cheng Zhang via Eric Yang)
CHUKWA-95. Added Web Service API to export data from database. (Terence Kwan via Eric Yang)
CHUKWA-281. Created pig scripts to perform down sampling. (Cheng Zhang via Eric Yang)
CHUKWA-20. Added pig support for ChukwaRecords. (Jerome Boulon via Eric Yang)
CHUKWA-279. Added swim lane visualization for Hadoop job progress. (Jiaqi Tan via Eric Yang)
CHUKWA-267. Added TaskLogAppender for collecting Task Tracker Log file. (Eric Yang)
CHUKWA-71. Status/version command for chukwa agent. (asrabkin)
CHUKWA-271. Added SQL client to HICC for SQL data exploration. (Eric Yang)
CHUKWA-210. Add new job conf metrics as database columns. (Cheng Zhang via Eric Yang)
CHUKWA-91. Additional test case for acks. (asrabkin)
IMPROVEMENTS
CHUKWA-390. Improvements to asynchronous acknowledgement mechanism. (asrabkin)
CHUKWA-397. Allow "all" as search pattern. (asrabkin)
CHUKWA-393. Support using pig on Chunks. (asrabkin)
CHUKWA-392. FIFO queueing of threads in collector. (asrabkin)
CHUKWA-388. Clean up user interface color. (Eric Yang)
CHUKWA-387. Summarize mode for dumpChunks should count bytes. (asrabkin)
CHUKWA-379. Refactor sender code. (asrabkin)
CHUKWA-374. Adaptor.getStatus() shouldn't throw exceptions. (asrabkin)
CHUKWA-373. Test code for backpressure. (asrabkin)
CHUKWA-370. Exec adaptor should commit immediately. (asrabkin)
CHUKWA-367. Print metadata in DumpChunks. (asrabkin)
CHUKWA-74. Agent shouldn't log exceptions for down collectors. (Rushin Barot via asrabkin)
CHUKWA-365. Improved DumpArchive tool. (asrabkin)
CHUKWA-364. Design and Architecture document in documentation. (asrabkin)
CHUKWA-333. Copy release notes from 0.2 forward to Trunk. (asrabkin)
CHUKWA-362. Archiver group-by-cluster conf name shouldn't be hardcoded. (asrabkin)
CHUKWA-355. Remove obsolete datacollection.protocol package. (asrabkin)
CHUKWA-270. ChukwaAgentController no longer uses stdout. (Rushin Barot via asrabkin)
CHUKWA-356. More powerful file extractor. (asrabkin)
CHUKWA-350. Improve docs, add programmer guide. (asrabkin)
CHUKWA-282. Demux should output detailed per-operation ClientTrace records (Jiaqi Tan via asrabkin)
CHUKWA-341. Heap space in HICC Jetty is configurable. (Jiaqi Tan via asrabkin)
CHUKWA-284. Checkpoint includes last byte Acked, not sent. (asrabkin)
CHUKWA-25. Archiver can group by cluster name. (asrabkin)
CHUKWA-308. Added capability to start hicc without tomcat. (Eric Yang)
CHUKWA-326. If the chukwa records produce errors in post process, move to InErrorDirectory. (Jerome Boulon via Eric Yang)
CHUKWA-318. Added check for disk capacity for collector to write. (Jerome Boulon via Eric Yang)
CHUKWA-276. Fix hourly and daily rolling to use a single reducer. (Jerome Boulon via Eric Yang)
CHUKWA-313. Removed the 10 second ack from SeqFileWriter. (Eric Yang)
CHUKWA-5. Adaptors have durable names. (asrabkin)
CHUKWA-280. Added end to end test to detect iostat overflow. (Eric Yang)
CHUKWA-278. Improve post process manager and metric data loader to support data loading from pig aggregation. (Eric Yang)
CHUKWA-263. Added ability to configure demux parameters during build time. (Jerome Boulon via Eric Yang)
CHUKWA-195. Update demux compilation to use Hadoop 0.20 jar file. (Jerome Boulon via Eric Yang)
CHUKWA-268. Expose adaptor manager interface, refactor Agent class hierarchy. (asrabkin)
CHUKWA-259. More flexible CollectorStub. (asrabkin)
CHUKWA-274. Improve rate calculation for accumulated values for visualization. (Eric Yang)
CHUKWA-275. Add standard deviation to statics calculation for HICC charts. (Eric Yang)
CHUKWA-137. Add collector IP to the dataSink fileName. (Jerome Boulon via asrabkin)
CHUKWA-258. More detailed logging of SQL errors. (asrabkin)
CHUKWA-30. Remove HDFS flush and connection holding in Collector. (Jerome Boulon via asrabkin)
CHUKWA-142. Remove add(Chunk) from SeqFileWriter. (Jerome Boulon via asrabkin)
CHUKWA-82. Added min,max,average to HICC Charting widget. (Terence Kwan via Eric Yang)
CHUKWA-219. Improved usability of host selector, and updated caching for host list. (Eric Yang)
BUG FIXES
CHUKWA-407. PatternSyntaxExceptions should be sent to client. (thushara wijeratna via asrabkin)
CHUKWA-402. Added readme and license notice file. (asrabkin)
CHUKWA-401. Fix race condition in Dir Tailing adaptor test case. (asrabkin)
CHUKWA-398. Make ConstRateAdaptor more deterministic. (asrabkin)
CHUKWA-386. Fix a bug in echo id to the browser for the iframe servlet. (Eric Yang)
CHUKWA-380. FTA shouldn't emit empty chunks. (asrabkin)
CHUKWA-378. Disable TestArchive unit test. (asrabkin)
CHUKWA-377. Revised xtrace adaptor code. (asrabkin)
CHUKWA-372. select_multiple_callback does not work in hicc (Eric Yang via asrabkin)
CHUKWA-354. Fix corner case in SinkArchiver. (asrabkin)
CHUKWA-334. Fix LocalHdfsMover exception handling. (Jerome Boulon via asrabkin)
CHUKWA-335. Discard meta line in job history file. (Cheng Zhang via asrabkin)
CHUKWA-330. Make separation between fixed adaptor parameters and optional adaptor parameters for generating adaptor id. (Ari Rabkin via Eric Yang)
CHUKWA-37. Remove stale and unused bin scripts. (asrabkin)
CHUKWA-331. Fixed regular expression for down sampling base on record type. (Cheng Zhang via Eric Yang)
CHUKWA-329. Correction to user job summary script. (Cheng Zhang via Eric Yang)
CHUKWA-328. Handle empty sequence file more gracefully for MetricDataLoader. (Cheng Zhang via Eric Yang)
CHUKWA-327. Remove START_TIME="0", FINISH_TIME="0" from JobHistory. (Cheng Zhang via Eric Yang)
CHUKWA-325. Removed hourly rolling flag from daily rolling. (Jerome Boulon via Eric Yang)
CHUKWA-324. For 0 matched field, changed MDL to skip empty SQL statement. (Eric Yang)
CHUKWA-321. Added 2 seconds sleep to avoid chukwa agent race condition. (Eric Yang)
CHUKWA-319. Added cluster name to pig user aggregation script. (Cheng Zhang via Eric Yang)
CHUKWA-316. Added explicit rule for MetricDataLoader to prevent loading of data. (Eric Yang)
CHUKWA-42. Clean up logging of adaptor creation. (Ari Rabkin)
CHUKWA-312. Fixed chukwa-pig.jar class path for pig aggregation. (Cheng Zhang via Eric Yang)
CHUKWA-304. Propagate exception upward for database test cases. (Jerome Boulon via Eric Yang)
CHUKWA-314. Added filtering for start_time to remove invalid value. (Eric Yang)
CHUKWA-304. Propagate exception upward for database test cases. (Eric Yang)
CHUKWA-309. Fix MDL configuration for mr_job_conf table. (Cheng Zhang via Eric Yang)
CHUKWA-301. Set TestFileExpirationPolicy controller port to 0 to avoid race condition. (Jerome Boulon via asrabkin)
CHUKWA-302. Removed hard coded dependency of Hadoop jar file. (Eric Yang)
CHUKWA-298. Update branch 0.2 and trunk version number. (Eric Yang)
CHUKWA-296. Fix jsp syntax error. (Terence Kwan via Eric Yang)
CHUKWA-294. Added post process data loader configuration. (Eric Yang)
CHUKWA-10. Fix TestAgentConfig (asrabkin)
CHUKWA-291. Append slash to collector paths. (asrabkin)
CHUKWA-287. Fix collector crash on startup when no args. (Eric Yang via asrabkin)
CHUKWA-273. Agents correctly use configured collector port numbers. (asrabkin)
CHUKWA-216. postProcess.sh should use HADOOP_CONF_DIR to find hadoop-site.xml. (Jerome Boulon via asrabkin)
CHUKWA-261. Added X axis label for non Time Series graphs. (Eric Yang)
Release 0.2 - Unreleased
Release 0.1.2 - 2009-05-14
INCOMPATIBLE CHANGES
NEW FEATURES
CHUKWA-97. Refactored FileTailers, added LWFTAdaptor. (asrabkin)
CHUKWA-236. Added migration script for moving database schema for Chukwa 0.1.1 to Chukwa 0.1.2. (Eric Yang)
CHUKWA-78. Added down sample SQL aggregation for job data, task data and utilization data. (Eric Yang)
CHUKWA-17. Collect PS output every 10 minutes and parse into key/value pairs sequence file. (Cheng Zhang via Eric Yang)
CHUKWA-55. Aggregate data from HDFS bytes usage, and Mapreduce Job slot time to compute user usage of the cluster. (Eric Yang)
CHUKWA-50. Added parser for extract Job History and Job Conf into key value pairs,
and database loader dictionary file. (Contribute by Cheng Zhang via Eric Yang)
CHUKWA-69. Calculate trapezoid area for a given series of data. (Contribute by Cheng Zhang via Eric Yang)
CHUKWA-62. Add script to start tailing files. (asrabkin)
CHUKWA-12. Add instrumention Api for Chukwa components. (Jerome Boulon via asrabkin)
CHUKWA-14. Added permalink support to HICC GUI. (Eric Yang)
CHUKWA-45. Added docs to folder structure. (Corinne Chandel via asrabkin)
CHUKWA-13. Create a new Macro class for macro substitution.
Changed Aggregator to use Macro class.
Added macro substitution support to Chukwa Charting.
Added Macro test case. (Eric Yang)
HADOOP-4989. Added capability to add scatter chart. (Eric Yang)
IMPROVEMENTS
CHUKWA-226. Changed HDFS usage collection frequency from 10 minutes to 60 minutes. (Cheng Zhang via Eric Yang)
CHUKWA-104. Remove permission setting code for Log4JMetricsContext and ChukwaDailyRollingFileAppender. (Jerome Boulon via Eric Yang)
CHUKWA-178. Extend aggregation by 5 minutes for cluster metrics. (Eric Yang)
CHUKWA-176. Rearrange the parameter order for aggregator.sh. (Eric Yang)
CHUKWA-173. Parameterize configuration, and enable substitution at build time. (Jerome Boulon via Eric Yang)
CHUKWA-131. Added additional Mapred job/task metrics. (Eric Yang)
CHUKWA-145. Tuned hadoop parameters for demux. (Jerome Boulon via Eric Yang)
CHUKWA-163. Updated reference to mdl.xml file. (Terence Kwan via Eric Yang)
CHUKWA-157. Added javadoc target, api-xml, api-report, and change log 2 html. (Eric Yang)
CHUKWA-141. Updated Chukwa Docs overview page for 0.1.2. (Corinne Chandel via Eric Yang)
CHUKWA-138. Updated Chukwa Admin Guide. (Corinne Chandel via Eric Yang)
CHUKWA-128. Added tools to compute aggregation in database. (Eric Yang)
CHUKWA-112. Updated README file. (Corinne Chandel via Eric Yang)
CHUKWA-134. Add release audit target. (Giridharan Kesavan via Eric Yang)
CHUKWA-80. Extracted rpm spec file from build.xml file. Fix start up script, and config script. (Eric Yang)
CHUKWA-26. * DemuxManager, ArchiveManager and PostProcessorManager are now a single daemon process each.
* Each one working independently from others, as soon as something is available.
* Start-data-processor is now using those new daemons instead of pocessSink.sh
* Daily will process a daily compaction only when all hourly would have been done.
* Demux is now able to send NSCA commands to Nagios. (Jerome Boulon via Eric Yang)
CHUKWA-81. Fix file descriptor leak. (asrabkin)
CHUKWA-29. Added TaskTracker and DataNode client trace log parser and database loader. (Chris Douglas via Eric Yang)
CHUKWA-86. Improved JDBC compatibility layer (Cheng Zhang via Eric Yang)
CHUKWA-64. Changed version number to 0.1.2, and release number to alpha. (Eric Yang)
CHUKWA-59. Collect HDFS Usage information for users. (Contribute by Cheng Zhang via Eric Yang)
CHUKWA-54. Filters web based input from HICC to prevent cross site scripting attack. (Eric Yang)
CHUKWA-31. Added duration logging for SQL statements. (Eric Yang)
HADOOP-5373. Collectors track lifetime-received chunks. (asrabkin)
HADOOP-5370. Collectors don't write empty sink files (asrabkin)
HADOOP-5228. Chukwa tests shouldn't write to /tmp. (asrabkin)
HADOOP-5035. Improved Y axis ticker labelling.
Used TreeMap to build non-time series data for charting.
Improved handling of "not a number "values. (Eric Yang)
HADOOP-5030. Changed RPM install location to the value specified by build.properties file. (Eric Yang)
HADOOP-5029. Added mdl script to manually load chukwa sequence file to database. (Eric Yang)
HADOOP-4843. Enable job history log file streaming into Chukwa by using JobTrackerInstrumentation API. (Eric Yang)
HADOOP-4827. * Replace Consolidator with Aggregator macros
* Updated database table schema
* Fix interval for aggregation execute
* Improved SQL macros support, add group_avg() and past_*_minutes key word (Eric Yang)
OPTIMIZATIONS
BUG FIXES
CHUKWA-249. Use ChukwaConfiguration in Demux. (Jerome Boulon via asrabkin)
CHUKWA-257. fixed right arrow image for date picker. (Eric Yang)
CHUKWA-251. Added per widget Time range input. (Terence Kwan via Eric Yang)
CHUKWA-239. Demux settings now work out of the box. (Ari Rabkin)
CHUKWA-250. Updated hadoop jar reference in chukwa config. (Cheng Zhang via Eric Yang)
CHUKWA-211. Manage symlink correctly for RPM upgrade and uninstall. (Eric Yang)
CHUKWA-245. Includes ChukwaJobTrackerInstrumentation class as part of the input tools compilation. (Eric Yang)
CHUKWA-234. Improved handling of null session for XssFilter. (Eric Yang)
CHUKWA-232. Fixed classpath for chukwa core jar file. (Eric Yang)
CHUKWA-215. Corrected postProcess.sh environment setup. (Jerome Boulon via Eric Yang)
CHUKWA-223. Renamed read only session caching object to cache.*, and skip those values for caching. (Eric Yang)
CHUKWA-238. Resolve race condition in archiving. (Ari Rabkin)
CHUKWA-228. Added rpm.hdfsusage.uid option to run HDFS usage as a separate user. (Cheng Zhang via Eric Yang)
CHUKWA-227. Improved the user privilege for reading HDFS usage. (Cheng Zhang via Eric Yang)
CHUKWA-229. Fix descriptor leak for ExecPlugin. (Ari Rabkin via Eric Yang)
CHUKWA-212. Fix file descriptor leak in MDL. (Jerome Boulon via Eric Yang)
CHUKWA-220. Correct min, max settings for yaxis charting. (Terence Kwan via Eric Yang)
CHUKWA-222. Reduce Sar version dependency. (Ari Rabkin)
CHUKWA-103. Fix documentation broken link. (Eric Yang)
CHUKWA-208. Skip copying of test/sample directory if it doesn't exist. (Eric Yang)
CHUKWA-206. Removed hard coded path from configuration files, and sample data. (Eric Yang)
CHUKWA-201. Removed sourcing of Timeline widget from simile.mit.edu. (Eric Yang)
CHUKWA-193. Remove unrecognized tag from Job History Log file parser. (Cheng Zhang via Eric Yang)
CHUKWA-189. Added JAVA_LIBRARY_PATH to chukwa-env.sh for enabling compression. (Jerome Boulon via Eric Yang)
CHUKWA-186. Remove duplicated timestamp from aggregation script. (Eric Yang)
CHUKWA-187. Correction to status for job and task status. (Cheng Zhang via Eric Yang)
CHUKWA-183. Added HICC startup script. (Eric Yang)
CHUKWA-177. Added test case for verify the value between JSON values and database values. (Terence Kwan via Eric Yang)
CHUKWA-181. Changed chukwa check point file to $CHUKWA_LOG_DIR. (Jerome Boulon via Eric Yang)
CHUKWA-182. Added nagios appender configuration. (Eric Yang)
CHUKWA-158. Consolidate namenode address to a single configuration object. (Eric Yang)
CHUKWA-168. Added watchdog for database. (Eric Yang)
CHUKWA-175. Removed error message for shutting down data processors. (Eric Yang)
CHUKWA-155. Store final job status only (Cheng Zhang via Eric Yang)
CHUKWA-164. Use year corresponding to the sender time stamp. (Cheng Zhang via Eric Yang)
CHUKWA-166. Handle null parameter case for XSSFilter. (Terence Kwan via Eric Yang)
CHUKWA-156. Test Macro testcase changed to use timestamp check for the generated macros. (Eric Yang)
CHUKWA-154. Handle adaptor exception, close file pointers on failure condition. (Jerome Boulon via Eric Yang)
CHUKWA-139. Rewrite collector bail out code. (Cheng Zhang via Eric Yang)
CHUKWA-119. Removed dependency of ChukwaAgent from ChunkImpl for preventing multiple
MetricsContext to be initialized in the same VM. (Jerome Boulon via Eric Yang)
CHUKWA-121. Added logic to detect partition number less than 0. (Eric Yang)
CHUKWA-132. Handle multi-line output in Job History file more gracefully. (Cheng Zhang via Eric Yang)
CHUKWA-124. Fixed JDBC connection initialization and closing. Added escapeQuotes utility method. (Eric Yang)
CHUKWA-122. Updated logic for finding time partition for Macro. (Eric Yang)
CHUKWA-125. Updated chukwa demux template to include JobHistory parser. (Eric Yang)
CHUKWA-98. Added Daemon watcher to capture signal for pid file management. (Cheng Zhang via Eric Yang)
CHUKWA-120. Added the missing commons-cli library. (Jerome Boulon via Eric Yang)
CHUKWA-61. Added report widget for accounting information. (Eric Yang)
CHUKWA-77. Added Database Schema for Hadoop accounting information. (Eric Yang)
CHUKWA-92. AbstractMetricsContext was using the wrong value (Jerome Boulon via asrabkin)
CHUKWA-70. Rewrite FileAdaptor. (Jerome Boulon via asrabkin)
CHUKWA-93. Fix NPE in SeqFileWriter. (Jiaqi Tan via asrabkin)
CHUKWA-1. Remove lzo job configuration from Chukwa data processors. (Contribute by Jerome Boulon via Eric Yang)
CHUKWA-40. Check for null pointer exception before unregister adaptor. (Contribute By Jerome Boulon via Eric Yang)
CHUKWA-75. Filter out iostat values that is greater than 1+e10.
CHUKWA-47. ChukwaJobTrackerInstrumentation class extends JobTrackerMetricsInst and added
finalizedJob method for stop streaming job history log file. (Jerome Boulon via Eric Yang)
CHUKWA-66. Corrected TerminatorThread logger class name. (Eric Yang)
CHUKWA-65. Redirect metrics log file to CHUKWA_LOG_DIR, and setup read/writable permission on the metrics
log files. (Eric Yang)
CHUKWA-73. Added Socket Timeout 60 seconds. (Jerome Boulon via Eric Yang)
CHUKWA-58. Changed watchdog to look for CHUKWA_PID_DIR. (Eric Yang)
CHUKWA-43. ChukwaLog4jAppender should send the current file offset instead of sending 0. (Jerome Boulon via asrabkin)
CHUKWA-41. ChukwaLog4jAppender does not escape \n for exception. (Jerome Boulon via asrabkin)
CHUKWA-28. Late initalization of log4j adaptor. (Jerome Boulon via asrabkin)
CHUKWA-48. Cleanup code to resolve compiler warnings. (asrabkin)
CHUKWA-9. MetricDataLoader should close JDBC connection. (Jerome Boulon via asrabkin)
CHUKWA-37. Remove ChuckwaArchiveBuilder. (asrabkin)
CHUKWA-33. Reformat code to fit hadoop style. (asrabkin)
CHUKWA-8. Remove deprecated conf files. (eyang via asrabkin)
CHUKWA-11. Remove non .template files from conf dir. (asrabkin)
HADOOP-5409. Changed opt directory copy to a optional operation, if opt exist. (Eric Yang)
HADOOP-5138. Fixing failing test cases by separating TestCharFileTailing
and TestFileTailing adaptors. (Jerome Boulon via acmurthy)
HADOOP-5401. Standardize control port conf option name as chukwaAgent.control.port (asrabkin)
HADOOP-5087. Fix regex for command parsing. (asrabkin)
HADOOP-5057. Better test coverage for missing checkpoint cases. (asrabkin)
HADOOP-5055. Changed alert.conf location from $CHUKWA_HOME/conf/alert.conf to $CHUKWA_CONF_DIR/alert.conf. (Eric Yang)
HADOOP-5054. Improved database partitioning by date. (Eric Yang)
HADOOP-5051. * Added macro token subsitution for sum(table_name)
* Added correct hdfs throughput aggregation SQL macros. (Eric Yang)
HADOOP-5032. Export CHUKWA_CONF_DIR in chukwa-config.sh.
HADOOP-4960. Changed metrics time from current system time to data source time.
HADOOP-4959. Support parsing of top output for Redhat EL 5.1. (Eric Yang)
HADOOP-4916. * Added external property file to reference location and ownership of chukwa
* Added ability to control the user name to run chukwa (Eric Yang)
HADOOP-4914. Added description fields to chukwa init.d scripts.
HADOOP-4889. Move chown from post install phase to build phase of the RPM file. (Eric Yang)
HADOOP-4884. Change date format from dd/mm/yyyy to yyyy/mm/dd for display in the chart tool tips. (Eric Yang)
HADOOP-4860. Changed test cases into 3 different test classes to prevent Agent from loading chukwa_check_point file which interfered with test cases. (Eric Yang)
HADOOP-4825. Replaced jps with ps ax for shutdown scripts.
Clean up reference of jar file from top level instead of build directory. (Eric Yang)
HADOOP-4805. Removed black list collector feature from Chukwa Agent HTTP Sender. (Eric Yang)
HADOOP-4796. Change component unit test targets from "ant test-agent" to "ant -Dtestcase=TestAgent test". (Eric Yang)
HADOOP-4791. Add build configuration parameter to specify where Chukwa will be installed for RPM packaging. (Eric Yang)
Release 0.1.1 - Unreleased
IMPROVEMENTS
HADOOP-4431. Add versionning/tags to Chukwa Chunk.
(Jerome Boulon via Johan)
HADOOP-4433. Improve data loader for collecting metrics and log files.
(Eric Yang via omalley)
HADOOP-5205. Change the value of CHUKWA_IDENT_STRING from demo to TODO-AGENTS-INSTANCE-NAME in chukwa-env.sh.template (Jerome Boulon via asrabkin)
HADOOP-5033. Simplified ChukwaWriter API. (asrabkin)
NEW FEATURES
HADOOP-3719. Initial checkin of Chukwa, which is a data collection and
analysis framework. (Jerome Boulon, Andy Konwinski, Ari Rabkin,
and Eric Yang)