/
CHANGES.txt
1669 lines (1030 loc) · 61.6 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Nutch Change Log
Release 2.1 - Current Development
* NUTCH-1440 reconfigure non-existent stopwords_en.txt in schema-solr4.xml (shekhar sharma via lewismc)
* NUTCH-1439 Define boost field as type float in schema-solr4.xml (shekhar sharma via lewismc)
* NUTCH-1438 ParserJob support for option -reparse (ferdy)
* NUTCH-1437 HostInjectorJob to accept lines with or without protocol (ferdy)
* NUTCH-1435 Host jobs throw NullPointerException with MySQL (ferdy via lewismc)
* NUTCH-1428 GeneratorMapper should not initialize filters/normalizers when they are disabled (ferdy)
* NUTCH-1427 Reuse SelectorEntry in Generator. (ferdy)
* NUTCH-1411 nutchgora fetcher.store.content does not work (Alexander Kingson via ferdy)
* NUTCH-1426 HostDb close() should close store instead of flush (ferdy)
* NUTCH-1425 DbUpdaterJob declares PREV_SIGNATURE on input twice (ferdy)
* NUTCH-1424 fix fetcher timelimit logging (ferdy)
* NUTCH-1423 Remove unused fields in LanguageIndexingFilter (ferdy)
* NUTCH-1306 Add option to not commit and clarify existing solr.commit.size (ferdy)
Release 2.0 (08/06/2012) ddmmyyy
Full Jira report - https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=10680&version=12314893
* NUTCH-1391 readdb -stats fires java.io.EOFException (jnioche)
* NUTCH-1400 Remove developer -core option for bin/nutch (jnioche)
* NUTCH-1399 TestProtocolHttpClient fails (jnioche)
* NUTCH-1404 Nutch script fails to find job file in deploy mode (sidabatra, jnioche)
* NUTCH-1401 Upgrade to Hadoop 1.0.3 (jnioche)
* NUTCH-1396 Upgrade Tika 1.1 (jnioche)
* NUTCH-1392 -force and -resume arguments being ignored in ParserJob (ferdy via lewismc)
* NUTCH-1379 NPE when reprUrl is null in ParseUtil (ferdy)
* NUTCH-1378 HostDb NullPointerException (ferdy)
* NUTCH-XX Commit to add configuration for separation of ant distribution targets (lewismc + jnioche)
* NUTCH-1364 Add a counter for malformed urls (Jason Trost via lewismc)
* NUTCH-1361 Fix mishandling of malformed urls in generator job (Jason Trost via lewismc)
* NUTCH-1366 speed up indexing by eliminating the indexreducer (ferdy)
* NUTCH-1362 Fix error handling of urls with empty fields (lewis, ferdy)
* NUTCH-1026 Strip UTF-8 non-character codepoints (markus, ferdy)
* NUTCH-1358 Do not accept bogus arguments (ferdy)
* NUTCH-1349 Make batchId explcit within debug logging and improve CLI (lewismc + ferdy)
* NUTCH-1352 Improve regex urlfilters/normalizers synchronization (ferdy)
* NUTCH-1356 ParseUtil use ExecutorService instead of manually thread handling. (ferdy)
* NUTCH-1355 nutchgora Configure minimum throughput for fetcher (ferdy)
* NUTCH-1354 nutchgora support fetcher.queue.depth.multiplier property (ferdy)
* NUTCH-1353 nutchgora DomainStatistics support crawlId, counter bug and reformatting (ferdy)
* NUTCH-1350 remove unused dependancy because of access restriction (ferdy)
* NUTCH-1205 Upgrade gora modules to 0.2 in ivy/ivy.xml (lewismc, ferdy)
* NUTCH-882 Design a Host table in GORA (jnioche, ab, dogacan, Mathijs Homminga, ferdy)
* NUTCH-1340 Increase scalability by only removing markers when they actually exist for DbUpdaterReducer (ferdy)
* NUTCH-1333 Introduce AvroStore, DataFileAvroStore and Accumulo Datastore implementations (lewismc)
* NUTCH-1312 Nutchgora to send HTTP-accept header (ferdy)
* NUTCH-1311 Add response headers to datastore for the protocol-httpclient plugin (Dan Rosher via ferdy)
* NUTCH-1304 GeneratorMapper.java dosen't return when skipping and already generated mark (Dan Rosher via lewismc)
* NUTCH-1307 Improve formatting of ant targets for clearer project help (lewismc)
* NUTCH-1302 nutchgora job failures should be noticed by submitter (ferdy)
* NUTCH-1298 Pass numTasks to FetcherJob (Dan Rosher via ferdy)
* NUTCH-1289 In distributed mode URL's are not partitioned (Dan Rosher, ferdy)
* NUTCH-1292 Better exception logging and debugging during fetch. (ferdy)
* NUTCH-1263 FetcherJob must put 'fetchTime' on input (ferdy)
* NUTCH-1296 nutchgora fetcher does not show correct 'threads' and 'resuming' properties (ferdy)
* NUTCH-1295 nutchgora restlet dependencies failing when remote repos is down (ferdy)
* NUTCH-965 Skip parsing for truncated documents (alexis, lewismc, ferdy)
* NUTCH-1287 Upgrade to hsqldb 2.2.8 (ferdy)
* NUTCH-1280 language-identifier should have option to use detected value by Tika even when uncertain (ferdy)
* NUTCH-1246 Upgrade to Hadoop 1.0.0 (lewismc)
* NUTCH-1279 Check if limit has been reached in GeneraterReducer must be the first check performance-wise. (ferdy)
* NUTCH-1255 Change ivy.xml of all plugins to remove "nutch.root" property (ferdy)
* NUTCH-1189 add commented out default settings to gora.properties file (lewismc, Ferdy)
* NUTCH-1138 remove LogUtil from trunk and nutchgora (lewismc)
* NUTCH-1237 Improve javac arguements for more verbose output (lewismc)
* NUTCH-1217 Update NOTICE.txt to drop some copyrights (lewismc)
* NUTCH-1216 Add trivial comment to lib/native/README.txt (lewismc)
* NUTCH-1198 Less verbose logging when unmapped mimetypes are trying to be parsed. (ferdy)
* NUTCH-1196 Update job should impose an upper limit on the number of inlinks (nutchgora) (ferdy)
* NUTCH-1185 Decrease solr.commit.size to 250 (markus)
* NUTCH-1172 AbstractNuchTest should have a generic testdir instead of specific 'inject' dir (ferdy)
* NUTCH-1192 Add '/runtime' to svn ignore (ferdy)
* NUTCH-1191 Port NUTCH-1102 to nutchgora - consistent use of fetcher.parse (ferdy)
* NUTCH-1187 Port NUTCH-1028 to nutchgora - log parser keys (ferdy)
* NUTCH-902 Add all necessary files and configuration so that nutch can be used with different backends out-of-the-box (lewismc)
* NUTCH-1081 & 1135 ant tests fail & Fix TestGoraStorage for Nutchgora (Ferdy via lewismc)
* NUTCH-1156 building errors with gora-hbase as a backend; update ivy.xml to use correct dependancies (Ferdy via lewismc)
* NUTCH-1109 Add Sonar targets to Ant build.xml (lewismc)
* NUTCH-1097 application/xhtml+xml should be enabled in plugin.xml of parse-html; allow multiple mimetypes for plugin.xml (Ferdy via lewismc)
* Change plugin source directory "languageidentifier" to "language-identifier" (lewismc)
* NUTCH-1132, 1133 & 1134 Fix TestGenerator, TestInjector & TestFetcher respectively (lewismc)
* NUTCH-1154 Upgrade to Tika 0.10. NOTE: Tika's new RTF parser may ignore more
text in malformed documents than previously - see TIKA-748 for details. (ab)
* NUTCH-1152 Upgrade SolrJ to version 3.4.0 (ab)
* NUTCH-1136 Ant pmd target is broken
* NUTCH-1058 Upgrade Solr schema version to 1.4 (markus)
* NUTCH-672 allow unit tests to be run from bin/nutch (Todd Lipton via lewismc)
* NUTCH-937 Put plugins in classes/plugins in job file (Claudio Martella, Ferdy Galema, jnioche)
* NUTCH-1131 Rely on published artefacts for GORA (jnioche)
* NUTCH-1099 Adds HBase and Cassandra storage properties to nutch-default.xml (lewismc)
* NUTCH-1096 Empty (not null) ContentLength results in failure of fetch (Ferdy Galema via jnioche)
* NUTCH-1089 Short compressed pages caused exception in protocol-httpclient (Simone Frenzel via jnioche)
* NUTCH-1085 Nutch script does not require HADOOP_HOME (jnioche)
* NUTCH-1083 ParserChecker implements Tools (jnioche)
* NUTCH-1004 Do not index empty values for title field (markus)
* NUTCH-914 Implement Apache Project Branding Requirements (lewismc via jnioche)
* NUTCH-1065 New mvn.template (lewismc)
* NUTCH-1045 MimeUtil to rely on default config provided by Tika (jnioche)
* NUTCH-1037 Option to deduplicate anchors prior to indexing (markus)
* NUTCH-1055 upgrade package.html file in language identifier plugin (lewismc)
* NUTCH-1043 Add pattern for filtering .js in default url filters (jnioche)
* NUTCH-1027 Degrade log level of `can't find rules for scope` (markus)
* NUTCH-1011 Normalize duplicate slashes in URL's (markus)
* NUTCH-1013 Migrate RegexURLNormalizer from Apache ORO to java.util.regex (markus)
* NUTCH-1016 Strip UTF-8 non-character codepoints and add logging for SolrWriter (markus)
* NUTCH-1012 Cannot handle illegal charset $charset (markus)
* NUTCH-295 Description for fetcher.threads.fetch property (kubes via markus)
* NUTCH-1006 MetaEquiv with single quotes not accepted (markus)
* NUTCH-1010 ContentLength not trimmed (markus)
* NUTCH-995 Generate POM file using the Ivy makepom task (mattmann, jnioche, Gabriele Kahlout)
* NUTCH-1003 task 'package' does not reflect the new organisation of the code (jnioche)
* NUTCH-994 Fine tune Solr schema (markus)
* NUTCH-999 Normalise String representation for Dates in IndexingFilters (jnioche)
* NUTCH-996 Indexer adds solr.commit.size+1 docs (markus)
* NUTCH-983 Upgrade SolrJ to 3.1 (markus, jnioche)
* NUTCH-989 Index-basic plugin and Solr schema now use date fieldType for tstamp field (markus)
* NUTCH-888 Remove parse-rss and add tests for rss to parse-tika (jnioche)
* NUTCH-991 SolrDedup must issue a commit (markus)
* NUTCH 986 SolrDedup fails due to date incorrect format (markus)
* NUTCH-977 SolrMappingReader uses hardcoded configuration parameter name for mapping file (markus)
* NUTCH-976 Rename properties solrindex.* to solr.* (markus)
* NUTCH-975 Fix missing/wrong headers in source files (markus, jnioche)
* NUTCH-980 Fix IllegalAccessError with slf4j used in Solrj (markus)
* NUTCH-982 Remove copying of ID and URL field in solrmapping (markus)
* NUTCH-891 Subcollection plugin won't require blacklist any more (markus)
* NUTCH-967 Upgrade to Tika 0.9 (jnioche)
* NUTCH-955 Ivy configuration improvements. Upgrade to Xerces 2.9.1 and Restlet 2.0.5 (alexis via ab)
* NUTCH-962 max. redirects not handled correctly: fetcher stops at max-1 redirects (Sebastian Nagel via ab)
* NUTCH-964 Upgraded Xerces to 2.91 (markus)
* NUTCH-824 Crawling - File Error 404 when fetching file with an hexadecimal character in the file name (Michela Becchi via jnioche)
* NUTCH-954 Strict application of Content-Length limit for http protocols (Alexis Detreglode via jnioche)
* NUTCH-953 Fixed crawl command in Nutch script (Alexis Detreglode via jnioche)
* NUTCH-950 DomainURLFilter throws NPE on bogus urls (Alexis Detreglode via jnioche)
* NUTCH-935 basicurlnormalizer removes unnecessary /./ in URLs (Stondet via markus)
* NUTCH-912 MoreIndexingFilter does not parse docx and xlsx date formats (Markus Jelsma, jnioche)
* NUTCH-936 LanguageIdentifier should not set empty lang field on NutchDocument (Markus Jelsma via jnioche)
* NUTCH-949 Conflicting ANT jars in classpath (jnioche)
* NUTCH-825 Publish nutch artifacts to central maven repository (mattmann)
* NUTCH-913 Nutch should use new namespace for Gora (dogacan)
* NUTCH-714 Need a SFTP and SCP Protocol Handler (Sanjoy Ghosh, mattmann)
* NUTCH-894 Move statistical language identification from indexing to parsing step
(Sertan Alkan via dogacan)
* NUTCH-901 Make index-more plug-in configurable (Markus Jelsma via mattmann)
* NUTCH-862 HttpClient null pointer exception (Sebastian Nagel via ab)
* NUTCH-904 "-resume" option is always processed as "false" in FetcherJob
(Faruk Berksöz via dogacan)
* NUTCH-905 Configurable file protocol parent directory crawling (Thorsten Scherler, mattmann, ab)
* NUTCH-716 Make subcollection index filed multivalued (Dmitry Lihachev via jnioche)
* NUTCH-884 FetcherJob should run more reduce tasks than default (ab)
* NUTCH-883 Remove unused parameters from nutch-default.xml (jnioche)
* NUTCH-886 A .gitignore file for Nutch (dogacan)
* NUTCH-872 Change the default fetcher.parse to FALSE (ab).
* NUTCH-861 Renamed HTMLParseFilter into ParseFilter
* NUTCH-876 Remove remaining robots/IP blocking code in lib-http (ab)
* NUTCH-851 Port logging to slf4j (jnioche)
* NUTCH-564 External parser supports encoding attribute (Antony Bowesman, mattmann)
* NUTCH-873 Ivy configuration settings don't include Gora (mattmann)
* NUTCH-870 Injector should add the metadata before calling injectedScore (jnioche via mattmann)
* NUTCH-867 Port Nutch benchmark to Nutchbase (ab)
* NUTCH-869 Add parse-html back (jnioche)
* NUTCH-871 MoreIndexingFilter missing date format (Max Lynch via mattmann)
* NUTCH-696 Timeout for Parser (ab, jnioche)
* NUTCH-774 Retry interval in crawl date is set to 0 (Reinhard Schwab via mattmann)
* NUTCH-697 Generate log output for solr indexer and dedup (Dmitry Lihachev, Jeroen van Vianen via mattmann)
* NUTCH-844 Improve NutchConfiguration (ab)
* NUTCH-850 SolrDeleteDuplicates needs to clone the SolrRecord objects (jnioche)
* NUTCH-845 Native hadoop libs not available through maven (ab)
* NUTCH-843 Separate the build and runtime environments (ab)
* NUTCH-821 Use ivy in nutch builds (Enis Soztutar, jnioche)
* NUTCH-838 Add timing information to all Tool classes (Jeroen van Vianen, mattmann)
* NUTCH-837 Remove search servers and Lucene dependencies (ab)
* NUTCH-836 Remove deprecated parse plugins (jnioche)
* NUTCH-835 Document deduplication failed using MD5Signature (Sebastian Nagel via ab)
* NUTCH-278 Fetcher-status might need clarification: kbit/s instead of kb/s shown (Alex McLintock via mattmann)
* NUTCH-833 Website is still Lucene branded (mattmann, Alex McLintock)
* NUTCH-832 Website menu has lots of broken links - in particular the API docs (Alex McLintock via mattmann)
* NUTCH-921 Reduce dependency of Nutch on config files (ab)
* NUTCH-907 DataStore API doesn't support multiple storage areas for multiple disjoint crawls (Sertan Alkan via ab)
* NUTCH-880 REST API for Nutch (ab)
* NUTCH-930 Remove remaining dependencies on Lucene API (ab)
* NUTCH-931 Simple admin API to fetch status and stop the service (ab)
* NUTCH-932 Bulk REST API to retrieve crawl results as JSON (ab)
Release 1.1 - 2010-06-06
* NUTCH-819 Included Solr schema.xml and solrindex-mapping.xml don't play together (ab)
* NUTCH-818 Bugfix : Parse-tika uses minorCodes instead of majorCodes in ParseStatus (jnioche)
* NUTCH-816 Add zip target to build.xml (mattmann)
* NUTCH-732 Subcollection plugin not working (Filipe Antunes, ab)
* NUTCH-815 Invalid blank line before If-Modified-Since header (Pascal Dimassimo via ab)
* NUTCH-814 SegmentMerger bug (Rob Bradshaw, ab)
* NUTCH-812 Crawl.java incorrectly uses the Generator API resulting in NPE (Phil Barnett via mattmann and ab)
* NUTCH-810 Upgrade to Tika 0.7 (jnioche)
* NUTCH-785 Copy metadata from origin URL when redirecting in Fetcher + call scfilters.initialScore on newly created URL (jnioche)
* NUTCH-779 Mechanism for passing metadata from parse to crawldb (jnioche)
* NUTCH-784 CrawlDBScanner (jnioche)
* NUTCH-762 Generator can generate several segments in one parse of the crawlDB (jnioche)
* NUTCH-740 Configuration option to override default language for fetched pages (Marcin Okraszewski via jnioche)
* NUTCH-803 Upgrade to Hadoop 0.20.2 (ab)
* NUTCH-787 Upgrade Lucene to 3.0.1. (Dawid Weiss via ab)
* NUTCH-796 Zero results problems difficult to troubleshoot due to lack of logging (ab)
* NUTCH-801 Remove RTF and MP3 parse plugins (jnioche)
* NUTCH-798 Upgrade to SOLR1.4 and its dependencies (jnioche)
* NUTCH-799 SOLRIndexer to commit once all reducers have finished (jnioche)
* NUTCH-782 Ability to order htmlparsefilters (jnioche)
* NUTCH-719 fetchQueues.totalSize incorrect in Fetcher (Steven Denny via jnioche)
* NUTCH-790 Some external javadoc links are broken (siren)
* NUTCH-766 Tika parser (jnioche via mattmann)
* NUTCH-786 Improvement to the list of suffix domains (jnioche)
* NUTCH-775 Enhance searcher interface (siren)
* NUTCH-781 Update Tika to v0.6 (jnioche)
* NUTCH-269 CrawlDbReducer: OOME because no upper-bound on inlinks count (stack + jnioche)
* NUTCH-655 Injecting Crawl metadata (jnioche)
* NUTCH-658 Use counters to report fetching and parsing status (jnioche)
* NUTCH-777 Upgrading to jetty6 broke unit tests (mattmann)
* NUTCH-767 Update Tika to v0.5 for the MimeType detection (Julien Nioche via ab)
* NUTCH-769 Fetcher to skip queues for URLS getting repeated exceptions
(Julien Nioche via ab)
* NUTCH-768 - Upgrade Nutch 1.0 to use Hadoop 0.20.1, also upgrades Xerces to
version 2.9.1. (kubes)
* NUTCH-712 ParseOutputFormat should catch java.net.MalformedURLException
coming from normalizers (Julien Nioche via ab)
* NUTCH-741 Job file includes multiple copies of nutch config files
(Kirby Bohling via ab)
* NUTCH-739 SolrDeleteDuplications too slow when using hadoop (Dmitry Lihachev via ab)
* NUTCH-738 Close SegmentUpdater when FetchedSegments is closed
(Martina Koch, Kirby Bohling via ab)
* NUTCH-746 NutchBeanConstructor does not close NutchBean upon contextDestroyed,
causing resource leak in the container. (Kirby Bohling via ab)
* NUTCH-772 Upgrade Nutch to use Lucene 2.9.1 (ab)
* NUTCH-760 Allow field mapping from Nutch to Solr index (David Stuart, ab)
* NUTCH-761 Avoid cloning CrawlDatum in CrawlDbReducer (Julien Nioche, ab)
* NUTCH-753 Prevent new Fetcher from retrieving the robots twice (Julien Nioche via ab)
* NUTCH-773 - Some minor bugs in AbstractFetchSchedule (Reinhard Schwab via ab)
* NUTCH-765 - Allow Crawl class to call Either Solr or Lucene Indexer (kubes)
* NUTCH-735 - crawl-tool.xml must be read before nutch-site.xml when
invoked using crawl command (Susam Pal via dogacan)
* NUTCH-721 - Fetcher2 Slow (Julien Nioche via dogacan)
* NUTCH-702 - Lazy Instanciation of Metadata in CrawlDatum (Julien Nioche via dogacan)
* NUTCH-707 - Generation of multiple segments in multiple runs returns only 1 segment
(Michael Chen, ab)
* NUTCH-730 - NPE in LinkRank if no nodes with which to create the WebGraph
(Dennis Kubes via ab)
* NUTCH-731 - Redirection of robots.txt in RobotRulesParser (Julien Nioche via ab)
* NUTCH-757 - RequestUtils getBooleanParameter() always returns false
(Niall Pemberton via ab)
* NUTCH-754 - Use GenericOptionsParser instead of FileSystem.parseArgs() (Julien
Nioche via ab)
* NUTCH-756 - CrawlDatum.set() does not reset Metadata if it is null (Julien Nioche
via ab)
* NUTCH-679 - Fetcher2 implementing Tool (Julien Nioche via ab)
* NUTCH-758 - Set subversion eol-style to "native" (Niall Pemberton via ab)
Release 1.0 - 2009-03-23
1. NUTCH-474 - Fetcher2 crawlDelay and blocking fix (Dogacan Guney via ab)
2. NUTCH-443 - Allow parsers to return multiple Parse objects.
(Dogacan Guney et al, via ab)
3. NUTCH-393 - Indexer should handle null documents returned by filters.
(Eelco Lempsink via ab)
4. NUTCH-456 - Parse msexcel plugin speedup (Heiko Dietze via siren)
5. NUTCH-446 - RobotRulesParser should ignore Crawl-delay values of other
bots in robots.txt (Dogacan Guney via siren)
6. NUTCH-482 - Remove redundant plugin lib-log4j (siren)
7. NUTCH-483 - Remove redundant commons-logging jar from ontology plugin
(siren)
8. NUTCH-161 - Change Plain text parser to
use parser.character.encoding.default property for fall back encoding
(KuroSaka TeruHiko, siren)
9. NUTCH-61 - Support for adaptive re-fetch interval and detection of
unmodified content. (ab)
10. NUTCH-392 - OutputFormat implementations should pass on Progressable.
(cutting via ab)
11. NUTCH-495 - Unnecessary delays in Fetcher2 (dogacan)
12. NUTCH-443 - allow parsers to return multiple Parse object, this will speed
up the rss parser (dogacan via mattmann). This update is a fix and semantics
change from the original patch for NUTCH-443. The original patch did not tell
the Indexer to read crawl_parse too so that it can pickup sub-urls' fetch
datums. This patch addresses that issue. Now, if Fetcher gets a null content,
instead of pushing an empty content, it filters the null content.
13. NUTCH-485 - Change HtmlParseFilter 's to return ParseResult object instead of
Parse object. (Gal Nitzan via dogacan)
14. NUTCH-489 - URLFilter-suffix management of the url path when the url contains
some query parameters. (Emmanuel Joke via dogacan)
15. NUTCH-502 - Bug in SegmentReader causes infinite loop.
(Ilya Vishnevsky via dogacan)
16. NUTCH-444 Possibly use a different library to parse RSS feed for improved
performance and compatibility. This patch introduced a new plugin, feed,
that includes an index filter and a parse plugin for feeds that uses ROME.
There was discussion to remove parse-rss, in light of the feed plugin,
however, this patch does not explicitly remove parse-rss. (dogacan, mattmann)
17. NUTCH-471 - Fix synchronization in NutchBean creation.
(Enis Soztutar via dogacan)
18. Upgrade to Lucene 2.2.0 and Hadoop 0.12.3. (ab)
19. NUTCH-468 - Scoring filter should distribute score to all outlinks at
once. (dogacan)
20. NUTCH-504 - NUTCH-443 broke parsing during fetching. (dogacan)
21. NUTCH-497 - Extreme Nested Tags causes StackOverflowException in
DomContentUtils...Spider Trap. (kubes)
22. NUTCH-434 - Replace usage of ObjectWritable with something based on
GenericWritable. (dogacan)
23. NUTCH-499 - Refactor LinkDb and LinkDbMerger to reuse code. (dogacan)
24. NUTCH-498 - Use Combiner in LinkDb to increase speed of linkdb generation.
(Espen Amble Kolstad via dogacan)
25. NUTCH-507 - lib-lucene-analyzers jar defintion is wrong in plugin.xml.
(Emmanuel Joke via dogacan)
26. NUTCH-503 - Generator exits incorrectly for small fetchlists.
(Vishal Shah via dogacan)
27. NUTCH-505 - Outlink urls should be validated. (dogacan)
28. NUTCH-510 - IndexMerger delete working dir. (Enis Soztutar via dogacan)
29. NUTCH-513 - suffix-urlfilter.txt does not have a template. (dogacan)
30. NUTCH-515 - Next fetch time is set incorrectly. (dogacan)
30. NUTCH-506 - Nutch should delegate compression to Hadoop. (dogacan)
31. NUTCH-517 - build encoding should be UTF-8. (Enis Soztutar via dogacan).
32. NUTCH-518 - Fix OpicScoringFilter to respect scoring filter chaining.
(Enis Soztutar via dogacan)
33. NUTCH-516 - Next fetch time is not set when it is a
CrawlDatum.STATUS_FETCH_GONE. (Emmanuel Joke via dogacan)
34. NUTCH-525 - DeleteDuplicates generates ArrayIndexOutOfBoundsException
when trying to rerun dedup on a segment. (Vishal Shah via dogacan)
35. NUTCH-514 - Indexer should only index pages with fetch status SUCCESS.
(dogacan) Note: There is a bigger problem, i.e how to deal
with redirected pages, and this issue can be considered as a band-aid
for the time being. See NUTCH-273 and NUTCH-353 for more details.
36. NUTCH-533 - LinkDbMerger: url normalized is not updated in the key and
inlinks list. (Emmanuel Joke via dogacan)
37. NUTCH-535 -ParseData's contentMeta accumulates unnecessary values during
parse. (dogacan)
38. NUTCH-522 - Use URLValidator in the Injector. (Emmanuel Joke, dogacan)
39. NUTCH-536 - Reduce number of warnings in nutch core. (dogacan)
40. NUTCH-439 - Top Level Domains Indexing / Scoring. Also adds
domain-related utilities. (Enis Soztutar via dogacan)
41. NUTCH-544 - Upgrade Carrot2 clustering plugin to the newest stable
release (2.1). (Dawid Weiss via dogacan)
42. NUTCH-545 - Configuration and OnlineClusterer get initialized in every
request. (Dawid Weiss via dogacan)
43. NUTCH-532 - CrawlDbMerger: wrong computation of last fetch time.
(Emmanuel Joke via dogacan)
44. NUTCH-550 - Parse fails if db.max.outlinks.per.page is -1. (dogacan)
45. NUTCH-546 - file URL are filtered out by the crawler. (dogacan)
46. NUTCH-554 - Generator throws IOException on invalid urls.
(Brian Whitman via ab)
47. NUTCH-529 - NodeWalker.skipChildren doesn't work for more than 1 child.
(Emmanuel Joke via dogacan)
48. NUTCH-25 - needs 'character encoding' detector.
(Doug Cook, dogacan, Marcin Okraszewski, Renaud Richardet via dogacan)
49. NUTCH-508 - ${hadoop.log.dir} and ${hadoop.log.file} are not propagated
to the tasktracker. (Mathijs Homminga, Emmanuel Joke via dogacan)
50. NUTCH-562 - Port mime type framework to use Tika mime detection framework.
(mattmann)
51. NUTCH-488 - Avoid parsing uneccessary links and get a more relevant outlink
list. (Emmanuel Joke, Marcin Okraszewski via kubes)
52. NUTCH-501 - Implement a different caching mechanism for objects cached in
configuration. (dogacan)
53. NUTCH-552 - Upgrade Nutch to Hadoop 0.15.x. (kubes)
54. NUTCH-565 - Arc File to Nutch Segments Converter. (kubes)
55. NUTCH-547 - Redirection handling: YahooSlurp's algorithm.
(dogacan, kubes via dogacan)
56. NUTCH-548 - Move URLNormalizer from Outlink to ParseOutputFormat.
(Emmanuel Joke via dogacan)
57. NUTCH-538 - Delete unused classes under o.a.n.util. (dogacan)
58. NUTCH-494 - FindBugs: CrawlDbReader and DeleteDuplicates. (dogacan)
59. NUTCH-574 - Including inlink anchor text in index can create irrelevant
search results. Created index-anchor plugin, removed functionality from
index-basic plugin. For backwards compatibility, add index-anchor plugin to
nutch-site.xml plugin.includes. (kubes)
60. NUTCH-581 - DistributedSearch does not update search servers added to
search-servers.txt on the fly. (Rohan Mehta via kubes)
61. NUTCH-586 - Add option to run compiled classes without job file
(enis via ab)
62. NUTCH-559 - NTLM, Basic and Digest Authentication schemes for web/proxy
server. (Susam Pal via dogacan)
63. NUTCH-534 - SegmentMerger: add -normalize option (Emmanuel Joke via ab)
64. NUTCH-528 - CrawlDbReader: add some new stats + dump into a CSV format
(Emmanuel Joke via ab)
65. NUTCH-597 - NPE in Fetcher2 (Remco Verhoef via ab)
66. NUTCH-584 - urls missing from fetchlist (Ruslan Ermilov, ab)
67. NUTCH-580 - Remove deprecated hadoop api calls (FS) (siren)
68. NUTCH-587 - Upgrade to Hadoop 0.15.3 (kubes)
69. NUTCH-604 - Upgrade to Lucene 2.3.0 (ab)
70. NUTCH-602 - Allow configurable number of handlers for search servers
(hartbecke via kubes)
71. NUTCH-607 - Update build.xml to include tika jar when building war (kubes)
72. NUTCH-608 - Upgrade nutch to use released apache-tika-0.1-incubating (mattmann)
73. NUTCH-606 - Refactoring of Generator, run all urls through checks (kubes)
74. NUTCH-605 - Change deprecated configuration methods for Hadoop (kubes)
75. NUTCH-603 - Add more default url normalizations (kubes)
76. NUTCH-611 - Upgrade Nutch to use Hadoop 0.16 (kubes)
77. NUTCH-44 - Too many search results, limits max results returned from a
single search. (Emilijan Mirceski and Susam Pal via kubes)
78. NUTCH-567 - Proper (?) handling of URIs in TagSoup. TagSoup library is
updated to 1.2 version. (dogacan)
79. NUTCH-613 - Empty summaries and cached pages (kubes via ab)
80. NUTCH-612 - URL filtering was disabled in Generator when invoked
from Crawl (Susam Pal via ab)
81. NUTCH-601 - Recrawling on existing crawl directory (Susam Pal via ab)
82. NUTCH-575 - NPE in OpenSearchServlet (John H. Lee via ab)
83. NUTCH-126 - Fetching https does not work with a proxy (Fritz Elfert via ab)
84. NUTCH-615 - Redirected URL-s fetched without setting fetchInterval.
Guard against reprUrl being null. (Emmanuel Joke, ab)
85. NUTCH-616 - Reset Fetch Retry counter when fetch is successful (Emmanuel
Joke, ab)
86. NUTCH-220 - Upgrade to PDFBox 0.7.3 (ab)
87. NUTCH-223 - Crawl.java uses Integer.MAX_VALUE (Jeff Ritchie via ab)
88. NUTCH-598 - Remove deprecated use of ToolBase. Use generics in Hadoop API.
(Emmanuel Joke, dogacan, ab)
89. NUTCH-620 - BasicURLNormalizer should collapse runs of slashes with a
single slash. (Mark DeSpain via ab)
90. NUTCH-500 - Add hadoop masters configuration file into conf folder.
(Emmanuel Joke via kubes)
91. NUTCH-596 - ParseSegments parse content even if its not
CrawlDatum.STATUS_FETCH_SUCCESS (dogacan)
92. NUTCH-618 - Tika error "Media type alias already exists" (mattmann,kubes)
93. NUTCH-634 - Upgrade Nutch to Hadoop 0.17.1 (Michael Gottesman, Lincoln
Ritter, ab)
94. NUTCH-641 - IndexSorter inorrectly copies stored fields (ab)
95. NUTCH-645 - Parse-swf unit test failing (ab)
96. NUTCH-642 - Unit tests fail when run in non-local mode (ab)
97. NUTCH-639 - Change LuceneDocumentWrapper visibility from
private to _public_ (Guillaume Smet via dogacan)
98. NUTCH-651 - Remove bin/{start|stop}-balancer.sh from svn
tracking. (dogacan)
99. NUTCH-375 - Add support for Content-Encoding: deflated
(Pascal Beis, ab)
100. NUTCH-633 - ParseSegment no longer allow reparsing.
(dogacan)
101. NUTCH-653 - Upgrade to hadoop 0.18. (dogacan)
102. NUTCH-621 - Nutch needs to declare it's crypto usage (mattmann)
103. NUTCH-654 - urlfilter-regex's main does not work.
(dogacan)
104. NUTCH-640 - confusing description "set it to Integer.MAX_VALUE".
(dogacan)
105. NUTCH-662 - Upgrade Nutch to use Lucene 2.4. (kubes)
106. NUTCH-663 - Upgrade Nutch to use Hadoop 0.19 (kubes)
107. NUTCH-647 - Resolve URLs tool (kubes)
108. NUTCH-665 - Search Load Testing Tool (kubes)
109. NUTCH-667 - Input Format for working with Content in Hadoop Streaming
(kubes)
110. NUTCH-635 - LinkAnalysis Tool for Nutch. (kubes)
111. NUTCH-646 - New Indexing Framework for Nutch. (kubes)
112. NUTCH-668 - Domain URL Filter. (kubes)
113. NUTCH-594 - Serve Nutch search results in multiple formats including
XML and JSON. (kubes)
114. NUTCH-442 - Integrate Solr/Nutch. (dogacan, original version by siren)
115. NUTCH-652 - AdaptiveFetchSchedule#setFetchSchedule doesn't calculate
fetch interval correctly. (dogacan)
116. NUTCH-627 - Minimize host address lookup (Otis Gospodnetic)
117. NUTCH-678 - Hadoop 0.19 requires an update of jets3t.
(julien nioche via dogacan)
118. NUTCH-681 - parse-mp3 compilation problem.
(Wildan Maulana via dogacan)
119. NUTCH-676 - MapWritable is written inefficiently and confusingly.
(dogacan)
120. NUTCH-579 - Feed plugin only indexes one post per feed due to identical
digest. (dogacan)
121. NUTCH-571 - parse-mp3 plugin doesn't always index album of mp3.
(Joseph Chen, dogacan)
122. NUTCH-682 - SOLR indexer does not set boost on the document.
(julien nioche via dogacan)
123. NUTCH-279 - Additions to urlnormalizer-regex (Stefan Neufeind, ab)
124. NUTCH-671 - JSP errors in Nutch searcher webapp (Edwin Chu via ab)
125. NUTCH-643 - ClassCastException in PDF parser (Guillaume Smet, ab)
126. NUTCH-636 - Httpclient plugin https doesn't work on IBM JRE
(Curtis d'Entremont, ab)
127. NUTCH-683 - NUTCH-676 broke CrawlDbMerger. (dogacan)
128. NUTCH-631 - MoreIndexingFilter fails with NoSuchElementException
(Stefan Will, siren)
129. NUTCH-691 - Update jakarta poi jars to the most relevant version
(Dmitry Lihachev via siren)
130. NUTCH-563 - Include custom fields in BasicQueryFilter
(Julien Nioche via siren)
131. NUTCH-695 - Incorrect mime type detection by MoreIndexingFilter plugin
(Dmitry Lihachev via siren)
132. NUTCH-694 - Distributed Search Server fails (siren)
133. NUTCH-626 - Fetcher2 breaks out the domain with db.ignore.external.links
set at cross domain redirects (Remco Verhoef, dogacan via siren)
134. NUTCH-247 - Robot parser to restrict (kubes, siren)
135. NUTCH-698 - CrawlDb is corrupted after a few crawl cycles (dogacan
via siren)
136. NUTCH-699 - Add an "official" solr schema for solr integration (dogacan,
Dmitry Lihachev via siren)
137. NUTCH-703 - Upgrade to Hadoop 0.19.1 (ab)
138. NUTCH-419 - Unavailable robots.txt kills fetch (Carsten Lehmann,
Doug Cook via ab)
139. NUTCH-700 - Neko1.9.11 goes into a loop (Julien Nioche, siren)
140. NUTCH-669 - Consolidate code for Fetcher and Fetcher2 (siren)
141. NUTCH-711 - Indexer failing after upgrade to Hadoop 0.19.1 (ab)
142. NUTCH-684 - Dedup support for Solr. (dogacan)
143. NUTCH-715 - Subcollection plugin doesn't work with default
subcollections.xml file (Dmitry Lihachev via siren)
144. NUTCH-722 - Nutch contains JAI jars that we cannot redistribute
Release 0.9 - 2007-04-02
1. Changed log4j confiquration to log to stdout on commandline
tools (siren)
2. NUTCH-344 - Fix for thread blocking issue (Greg Kim via siren)
3. NUTCH-260 - Update hadoop version to 0.5.0 (Renaud Richardet,
siren)
4. Optionally skip pages with abnormally large values of Crawl-Delay
(Dennis Kubes via ab)
5. Change readdb -stats to use CombiningCollector (ab)
6. NUTCH-348 - Fix Generator to select highest scoring pages (Chris
Schneider and Stefan Groschupf via ab)
7. NUTCH-347 - Adjust plugin build script not to emit warnings when copying
dependant jars (siren)
8. NUTCH-338 - Remove the text parser as an option for parsing PDF files
in parse-plugins.xml (Chris A. Mattmann via siren)
9. NUTCH-105 - Network error during robots.txt fetch causes file to
be ignored (Greg Kim via siren)
10. NUTCH-367 - DistributedSearch thown ClassCastException (siren)
11. NUTCH-332 - Fix the problem of doubling scores caused by links pointing
to the current page (e.g. anchors). (Stefan Groschupf via ab)
12. NUTCH-365 - Flexible URL normalization (ab)
13. NUTCH-336 - Differentiate between newly discovered pages and newly
injected pages (Chris Schneider via ab) NOTE: this changes the
scoring API, filter implementations need to be updated.
14. NUTCH-337 - Fetcher ignores the fetcher.parse value (Stefan Groschupf
via ab)
15. NUTCH-350 - Urls blocked by http.max.delays incorrectly marked as GONE
(Stefan Groschupf via ab)
16. NUTCH-374 - when http.content.limit be set to -1 and
Response.CONTENT_ENCODING is gzip or x-gzip , it can not fetch any thing
(King Kong via pkosiorowski)
17. NUTCH-383 - upgrade to Hadoop 0.7.1 and Lucene 2.0.0. (ab)
****************************** WARNING !!! ********************************
* This upgrade breaks data format compatibility. A tool 'convertdb' *
* was added to migrate existing CrawlDb-s to the new format. Segment data *
* can be partially migrated using 'mergesegs', however segments will *
* require re-parsing (and consequently re-indexing). *
****************************** WARNING !!! ********************************
18. NUTCH-371 - DeleteDuplicates now correctly implements both parts of
the algorithm. (ab)
19. NUTCH-391 - ParseUtil logs file contents to log file when it cannot
find parser (siren)
20. NUTCH-379 - ParseUtil does not pass through the content's URL to the
ParserFactory (Chris A. Mattmann via siren)
21. NUTCH-361, NUTCH-136 - When jobtracker is 'local' generate only one
partition. (ab)
22. NUTCH-399 - Change CommandRunner to use concurrent api from jdk (siren)
23. NUTCH-395 - Increase fetching speed (siren)
24. NUTCH-388 - nutch-default.xml has outdated example for urlfilter.order
(reported by Jared Dunne)
25. NUTCH-404 - Fix LinkDB Usage - implementation mismatch (siren)
26. NUTCH-403 - Make URL filtering optional in Generator (siren)
27. NUTCH-405 - Content object is not properly initialized in map method
of ParseSegment (siren)
28. NUTCH-362 - Remove parse-text from unsupported filetypes in
parse-plugins.xml (siren)
29. NUTCH-305 - Update crawl and url filter lists to exclude
jpeg|JPEG|bmp|BMP, suffix-urlfilter.txt (contributed by Stefan
Neufeind) is also updated (siren)
30. NUTCH-406 - Metadata tries to write null values (mattmann)
31. NUTCH-415 - Generator should mark selected records in CrawlDb.
Due to increased resource consumption this step is optional.
Application-level locking has been added to prevent concurrent
modification of databases. (ab)
32. NUTCH-416 - CrawlDatum status and CrawlDbReducer refactoring. It is
now possible to correctly update CrawlDb from multiple segments.
Introduce new status codes for temporary and permanent
redirection. (ab)
33. NUTCH-322 - Fix Fetcher to store redirected pages and to store
protocol-level status. This also should fix NUTCH-273. (ab)
34. Change default Fetcher behavior not to follow redirects immediately.
Instead Fetcher will record redirects as new pages to be added to CrawlDb.
This also partially addresses NUTCH-273. (ab)
35. Detect and report when Generator creates 0-sized segments. (ab)
36. Fix Injector to preserve already existing CrawlDatum if the seed list
being injected also contains such URL. (ab)
37. NUTCH-425, NUTCH-426 - Fix anchors pollution. Continue after
skipping bad URLs. (Michael Stack via ab)
38. NUTCH-325 - UrlFilters.java throws NPE in case urlfilter.order contains
Filters that are not in plugin.includes (Stefan Groschupf, siren)
39. NUTCH-421 - Allow predeterminate running order of indexing filters
(Alan Tanaman, siren)
40. When indexing pages with redirection, drop all intermediate pages and
index only the final page. (ab)
41. Upgrade to Hadoop 0.10.1. (ab)
42. NUTCH-420 - Fix a bug in DeleteDuplicates where results depended on the
order in which IndexDoc-s are processed. (Dogacan Guney via ab)
43. NUTCH-428 - NullPointerException thrown when agent name is not
configured properly. Changed to throw RuntimeException instead.
(siren)