-
Notifications
You must be signed in to change notification settings - Fork 745
/
CHANGES.txt
3220 lines (2195 loc) · 114 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release 3.0.0-BETA - 12/01/2023
BREAKING CHANGES
* Require Java 11 (TIKA-4128).
* The boilerpipe handler has been moved to the tika-handler-boiler-pipe
package (TIKA-4138).
* We've migrated HTML parsing to the JSoup parser instead of TagSoup. If
you have a custom configuration on the HTMLParser, you'll need to change
that to o.a.t.p.html.JSoupParser (TIKA-1599).
* Removed xerces2 as a dependency (TIKA-4135).
* tika-core now has a scope of "provided" in most non-app modules (TIKA-4191).
* Tika will look for "custom-mimetypes.xml" directly on the classpath, NOT
under "/org/apache/tika/mime/". (TIKA-4147).
* Return media type "text/javascript" instead of "application/javascript"
to follow RFC-9239. (TIKA-4119).
Other Changes/Updates
* Improve detection of sqlite3-based file formats (TIKA-4187).
* Upgrade PDFBox to 3.0.1 (TIKA-3347)
* Deprecated AbstractParser for removal in 4.x (TIKA-4132).
* Fix bug in DateUtils that stripped timezone information from
incoming Calendar objects (TIKA-4126).
* The InputStreamDigester now calculates stream length (TIKA-4016).
Release 2.9.0 - 8/23/2023
* With user configuration, the PDFParser can now throw an EncryptedDocumentException
for Microsoft IRM PDF containers with encrypted payloads. Separately,
the PDFParser now throws an EncryptedDocumentException instead of an IOException
if the security handler cannot be found (TIKA-4082).
* Fix bug that led to duplicate extraction of macros from some OLE2 containers (TIKA-4116).
* Parse iframe's srcdoc as an embedded file (TIKA-3109).
* Add detection of warc.gz as a specialization of gz and parse as if a standard WARC (TIKA-4048).
* Allow users to modify the attachment limit size in the /unpack resource (TIKA-4039)
* Fixed write limit bug in RecursiveParserWrapper (TIKA-4055).
* Add mime detection for many files with thanks to Gregory Lepore (TIKA-3992).
* Fixed iWork 13 keynote detection on files with wrong extension (TIKA-4111).
Release 2.8.0 - 5/11/2023
* Enable counting and/or parsing of incremental updates in PDFs. This
is an experimental feature and may change in later releases (TIKA-4017).
* Fixed bug that prevented the the loading of CompositeExternalParser in tika-app and
tika-server-standard. This parser will call exiftool and ffmpeg if those are installed, as was
the behavior in Tika 1.x. Exclude org.apache.tika.parser.external.CompositeExternalParser
if you do not want this behavior (TIKA-4022).
* Removed the shading of tika-parsers-standard-module (TIKA-4038).
* Enable optional extraction of file system metadata in FileSystemFetcher (TIKA-4035).
* Allow pretty printing in FileSystemEmitter (TIKA-4034).
* Add detection for and a new mime type for older postscript-based
Adobe Illustrator "application/illustrator+ps" files (TIKA-3971).
* Add magic detection for canon raw file types: crw, cr2 and cr3 (TIKA-3991).
* Add detection for ONIX message files (TIKA-4011).
* Add detection and a parser for ActiveMime files (TIKA-3987).
* Add extraction of rendition layout value and version from Epub (TIKA-4013).
* Improve embedded file extraction from PDFs (TIKA-4012).
* Improve metadata extraction from WARCs (TIKA-4018).
* Update to PDFBox 2.0.28 (TIKA-4016).
* Users may now avoid the ZeroByteFileException via a
setting on the AutoDetectParserConfig (TIKA-3976).
* Fix bug in closing <a> elements in the presence of <b> elements
in RTF files (TIKA-3972).
* Improve extraction of embedded file names in .docx (TIKA-3968).
* Normalize author, title, subject and description to their Dublin Core
properties in the HTMLParser (TIKA-3963).
Release 2.7.0 - 1/31/2023
* Add SVG detection for svg files that lack the xml header (TIKA-3308).
* Migrate to a live fork of Universal Charset Detector (TIKA-3213).
* Improve handling of text-based attachments inside .eml files (TIKA-3959).
* Add tika-parser-nlp-package to release artifacts (TIKA-3958).
* Remove need for <params/> element in classes that extend ConfigBase (TIKA-3946).
* Add X-TIKA:embedded_id_path to ensure unique embedded file paths (TIKA-3942).
* Fix bug that prevented digests when the fallback/EmptyParser
was called (TIKA-3939).
* Remove log4j 1.2.x (and slf4j-log4j12 which now redirects to slf4j-reload4j) from
all modules (TIKA-3935).
* Upgrade mime4j to 0.8.9 (TIKA-3950).
* Refactor date parsing for emails (TIKA-3957)
* Upgrade to Bouncy Castle 1.71 and jdk18on jars (TIKA-3933).
* Add a JDBCPipesReporter (TIKA-3931).
* Add multivalued field strategy option in jdbc-emitter (TIKA-3930).
Default is now 'concatenate' with ', ' as the delimiter.
* Downgrade logging in PipesClient for each parse from info to debug.
Release 2.6.0 - 11/3/2022
* Add optional Siegfried detector (TIKA-3901).
* Move OverrideDetector's functionality to the CompositeDetector (TIKA-3904).
* The FileCommandDetector has been refactored to have the same
behavior as the Siegfried detector; see setUseMime in the javadoc (TIKA-3902).
* Fix bug in OpenSearch emitter that prevented upserts on
documents with embedded files (TIKA-3882).
* Extract PDF actions and triggers into the file's metadata (TIKA-3887).
* Add a tika-async-cli module (TIKA-3885).
* Fetch keys sent via headers to tika server are now URL decoded (TIKA-3864).
Release 2.5.0 - 09/30/2022
* Improved extraction of PDF subset info for PDF/UA, PDF/VT, and PDF/X.
NOTE: we no longer append PDF/A information, e.g. 'version="A-1b"'
to the 'dc:format'. Users must now get that information from the
'pdfa:PDFVersion' key or from 'pdfaid:conformance'
and 'pdfaid:part' (TIKA-3844).
* Avoid infinite loop in bookmark extraction from PDFs (TIKA-3832).
* Upgraded to slf4j 2.0.1 (TIKA-3842).
* Added upsert option for the OpenSearch emitter (TIKA-3855).
* Extract PDF signature information at the document level
into the metadata (TIKA-3852).
* Enable configuration of digests via AutoDetectParserConfig (TIKA-3853).
* Use commons-io byte array streams via PJ Fanning (TIKA-3843).
* Upgrade to PDFBox 2.0.27 (TIKA-3866).
* Upgrade to JempBox 1.8.17 (TIKA-3856).
* Add extraction of ODF version from ODF files (TIKA-3840).
* tika-parser-html-commons (BoilerPipeHandler) is no longer a
a dependency of tika-parser-html-module. tika-app and tika-server-standard
have added a dependency on tika-parser-html-commons. However,
users who are managing custom dependencies and who want the BoilerPipeHandler
will have to now include the tika-parser-html-commons dependency
(TIKA-1484).
* Add unrar as an optional parser (TIKA-3800).
* Refactor FuzzingCLI to use PipesParser (TIKA-3799).
* ServiceLoader's loadServiceProviders() now guarantees
unique classes (TIKA-3797).
* Fix bug that prevented setting of includeHeadersAndFooters
for xls, xlsx, doc and docx via tika-config (TIKA-3796).
* Fix bug that prevented specification of rendered image type
via http header in the PDFParser (TIKA-3794).
* Fix bug causing some Exif dates to be decoded wrongly on
timezones different than UTC (TIKA-3815).
* Numerous dependency upgrades (TIKA-3795).
* Add ALPHA-level initial releases of JDBCEmitter,
FileSystemStatusReporter and OpenSearchPipesReporter.
These may have breaking changes in subsequent releases.
Release 2.4.1 - 06/14/2022
* Implement bulk upload in the OpenSearch emitter (TIKA-3791).
* Implement tika-server client via pipes mode (TIKA-3790).
* Custom embedded parsers and EmbeddedDocumentHandlers
can now add metadata to the container file's
metadata (TIKA-3789).
* Record embedded file exceptions in the container
file's metadata (TIKA-3788).
* Allow continuation of parsing after write limit has
been reached (TIKA-3787).
* Allow pass-through of 'Content-Length' header to metadata
in TikaResource (TIKA-3786).
* Add embedded depth to profiles tables in tika-eval (TIKA-3775).
* Add stop() method to TikaServerCli so that it can be run
with Apache Commons Daemon (TIKA-1570).
* Fixed bug in ordering of Parsers during service loading (TIKA-3750).
* Users can expand system properties from the forking
process into forked tika-server processes (TIKA-3748).
* Fix a few files being wrongly detected as EML (TIKA-3771).
* Fix ignoreCharsets param of Icu4jEncodingDetector (TIKA-3774).
Release 2.4.0 - 04/23/2022
* NOTE: To save on resources, we no longer include the
deeplearning4j dependencies in the tika-dl jar. The dependencies for the
tika-dl package must be provided by users. See:
https://github.com/apache/tika/blob/main/tika-parsers/tika-parsers-ml/tika-dl/pom.xml
for the dependencies that must be provided at run-time (TIKA-3676).
* NOTE: Added prefix "dwg-custom:" to DWG custom metadata properties (TIKA-3731).
* Add initial, BETA-grade TLS encryption option for tika-server;
configuration may change in future releases (TIKA-3719).
* Allow specification of fetcherName and fetchKey via query parameters
in request URI in tika-server (TIKA-3714).
* Add basic parsers for WARC and WACZ in tika-parsers-standard (TIKA-3697).
* Add MetadataWriteFilter capability to improve memory profile in
Metadata objects (TIKA-3695).
* Allow configurability of the ContentHandlerDecorator used
by the AutoDetectParser (TIKA-3723).
* Allow configurability of the EmbeddedDocumentExtractor used
by the AutoDetectParser (TIKA-3711).
* Add detection for Frictionless Data packages and WACZ (TIKA-3696).
* Add detection for DGN files with gratitude and credit
to Steven Frew's tika-dgn-detector (TIKA-3721).
* Add parser for metadata from DGN 8 files via Dan Coldrick (TIKA-3721).
* Add a fetcher and emitter for Azure blob storage (TIKA-3707).
* Add detection for files encrypted by Microsoft's Rights Management Service
(TIKA-3666).
* Fixed regression in 2.3.0 that led to more embedded filenames
than appropriate being written to the content (TIKA-3711).
* tika-server now clones forking process' environment variables
into forked process (TIKA-3715).
* Add an optional /eval endpoint for tika-eval profile or compare
capabilities in tika-server (TIKA-3689).
* Add a Parsed-By-Full-Set metadata item to record all parsers that processed
a file (TIKA-3716).
* Add metadata filters for Optimaize and OpenNLP language detectors (TIKA-3717).
* Upgrade to PDFBox 2.0.26 (TIKA-3726).
* Upgrade deeplearning4j to 1.0.0-M2 (TIKA-3458 and PR#527).
* Various dependency upgrades, including POI, dl4j, gson, jackson,
twelvemonkeys, log4j2 and others (TIKA-3675 and many PRs from dependabot).
* Switch cipher from ECB to GCM in HttpClientFactory (TIKA-3724).
Release 2.3.0 - 02/02/2022
* Upgrade to Apache POI 5.2.0. This is the first upgrade to POI
5.x and represents a major refactoring. Users may experience
significantly more logging (TIKA-3164).
* Upgrade to log4j2 2.17.1 (TIKA-3638).
* Improve consistency in reporting package-entry divs across
all parsers for embedded files (TIKA-3644). This leads
to some more text (embedded file names) in files with
many embedded attachments.
* Improve configuration of maps as params for parsers in
TikaConfig (TIKA-3645).
* Improve identification of iWorks 13 files and add parsing
for thumbnails, some metadata and attachments (TIKA-3634).
Skip handling of .iwa files, which are not yet supported.
* Limit the default in-memory processing (maxMainMemoryBytes) in
the PDFParser to 512MB as in the 1.x branch (TIKA-3642).
* Added IDML Parser from 1.x series to 2.x series (TIKA-3188).
* Extract annotation types and subtypes for PDFs into metadata (TIKA-3653).
* Add metadata value for PDFs that contain 3D annotations (TIKA-3653).
* Add parser for Translation Memory eXchange (TMX) files (TIKA-3660).
* Add Bill of Materials (Maven BOM) for centralized module version management (TIKA-3367).
Release 2.2.1 - 12/19/2021
* Fix multithreading bug for ooxml files (TIKA-3627).
* Upgrade log4j to 2.17.0 (TIKA-3625).
* Upgrade to PDFBox 2.0.25 (TIKA-3622)
* Fix bug that prevented metadata keys in the UnpackerResource
in tika-server (TIKA-3624).
* Upgrade log4j to 2.16.0 (TIKA-3623)
Release 2.2.0 - 12/13/2021
* Add support for OneNote files downloaded from O365 (TIKA-3446).
* Fix logic bug in PipesServer that prevented concatenation of
content from attachments (TIKA-3609).
* Improve extraction of embedded files from MSOffice files created
by non-Microsoft tools (TIKA-3526).
* Added back ability to ignore load errors in TikaConfig (TIKA-3575).
* Make SecureContentHandler and other parameters configurable in
AutoDetectParser programmatically and via tika-config.xml (TIKA-3594).
* Fix default logging in tika-app in batch mode (TIKA-3589).
* Fix bug that prevented specifying a config with the long
--config= option in tika-app in batch mode (TIKA-3589).
* Fix thread starvation after numerous restarts in
PipesClient (TIKA-3588).
* Fix race condition when starting multiple forked
servers on multiple ports (TIKA-3586).
* Add timeout per task to be configured via headers
for tika-server's legacy endpoints /tika and /rmeta.
Note that this timeout greater than taskTimeoutMillis (TIKA-3582).
* Add metadata item for whether or not a PDF has a collection/
is a Portfolio PDF (TIKA-3579).
* Add detection of ESRI Layer files (TIKA-3570).
* Add detection of JPEG XL, MARC, ICC profiles, NES-ROM file types
(TIKA-3562 and TIKA-3563)
* Remove duplicate "subject" metadata keys that were intended
for backwards compatibility within 1.x only (TIKA-3564).
* Fix Open Office mime types to be subclasses of application/zip
and no longer require OPCPackageDetector-last ordering of zip
detectors (TIKA-3556).
* Improve robustness and features of the httpfetcher (TIKA-3543)
* Add optional fetch ranges to FetchEmitTuple to allow range fetching from,
e.g. http or s3 (TIKA-3542).
* Exclude dependencies on jsoup and ehcache in ucar grib/cdm (TIKA-3003).
Release 2.1.0 - 08/18/2021
MAJOR CHANGES in 2.1.0:
* Improved packaging for tika-parsers-extended. Use the tika-parser-scientific-package and
tika-parser-sqlite3-package artifacts if you want fat jars with dependencies. (TIKA-3510)
* Tika app writes UTF-8 when an encoding is not specified; the legacy behavior
was UTF-8 on Mac OS, but System default on other OSs (TIKA-3515).
* Change the default rendering strategy for PDFs from NO_TEXT to ALL (TIKA-3520).
Other changes:
* Fixed bug that pointed to the wrong tessdata directory if the user specified
a tesseract path but not also a tessdata path (TIKA-3518).
* Fixed bug in Icu4j's encoding detector where it would return non-standard
names for charsets, e.g. IBM424_rtl is now returned as IBM424 (TIKA-3516).
* Add a simple UrlFetcher in tika-core as a basic alternative
to tika-fetcher-http (TIKA-3527).
* Add tika-pipes support for Google Cloud Storage (TIKA-3524).
* Fix markup ordering errors in xhtml output for ODT files (TIKA-2242).
* Fix serialization of embedded docs in OpenSearch emitter
and fix embedded documents not being indexed in some use
cases in the Solr emitter (TIKA-3490).
* Add pipesClientId system property to PipesServer so that each
forked process can log to its own logger (TIKA-3480).
* Add DateNormalizingMetadataFilter let users ensure that all dates
emitted to Solr/OpenSearch are in UTC. Users can configure which
timezone they'd like to use in cases where the file format does
not store a timezone (TIKA-3496).
* Breaking change in the Solr and OpenSearch emitters. To achieve
the SKIP or CONCATENATE attachment strategy, modify the
parseMode in the pipesiterators or in the FetchEmitTuple (TIKA-3494).
Release 2.0.0 - 07/07/2021
* Cleanup of fetcher integration with tika-server.
* Update dependencies.
Release 2.0.0-BETA - 05/19/2021
* Refactor pipes module for resilience
* Add transcribe capability (TIKA-94).
Release 2.0.0-ALPHA - 01/13/2021
BREAKING CHANGES in 2.0.0
* General
* OCR is now triggered automatically for PDFs if tesseract
is on the user's path see (https://cwiki.apache.org/confluence/display/TIKA/TikaOCR#TikaOCR-disable-ocr)
for how to disable OCR.
* We upgraded from log4j to log4j2 in tika-app, tika-server and anywhere else
we used to use log4j.
* By default, when rendering a page for OCR, the PDFParser does not render glyphs/text.
* Removed deprecated Metadata keys/properties (TIKA-1974).
* Removed deprecated PDFPreflightParser (TIKA-3437).
* Removed dangerous calls to read an inputstream or convert to bytes
without specifying a charset
* Parsers can be configured via tika-config.xml on instantiation.
We have moved away from configuration via .properties files because
of confusion among users. This affects the PDFParser, TesseractOCRParser
and the StringsParser.
* Changed namespaces of translator implementations (o.a.t.language.translate.impl) to avoid
split-package with tika-core
* tika-parsers
* The parser modules have been broken into three main modules:
tika-parsers-standard, tika-parsers-extended and tika-parsers-ml.
Users may now need to add tika-parsers-extended's
tika-parser-scientific-module or tika-parser-sqlite3-module to tika-app and
tika-server to include parsers that used to be included by default
(for example: envi, gdal, grib, isatab, netcdf, sqlite3).
* PDFParser -- a) see above on OCR. b) This parser no longer warns if the jpeg2000
dependency is not included. Tika now relies on PDFBox to log an error if a jpeg2000
image should be processed but can't because the required external dependency is
not available. See https://pdfbox.apache.org/2.0/dependencies.html#jai-image-io
for the non-ASF-2.0-compatible jpeg2000 library.
* CompressorParser -- users must add the com.github.luben:zstd-jni dependency to
the classpath to process zstd files. This is an optional library that is no longer bundled
in tika-parsers-standard-package because it contains native libs.
* ChmParser was moved to org.apache.tika.parser.microsoft.chm
* RTFParser was moved to org.apache.tika.parser.microsoft.rtf
* We are now using non-shaded versions of xmpcore with namespaces com.adobe.internal.*
vs com.adobe.*.
* tika-app
* See above on default inclusion of only tika-parsers-standard.
* tika-server
* tika-server now by default forks a process to isolate the parsing
in the forked process (this was called the -spawnChild option
in tika-1.x). Clients must now expect that tika-server
will restart on OOM, timeouts, crashes or after parsing a
large number of files. When this happens tika-server will restand and not
receive connections for brief periods. The less robust, legacy behavior
of not forking a process is available with "-noFork"=
* Most of tika-server's legacy configuration via the commandline has been moved
into configuration via a tika-config.xml file.
* tika-server's "enableFileUrl" has been removed in favor of a FileSystemFetcher.
* tika-server's /metadata endpoint requires tika-server-standard to write XMP/rdf output.
This output is not available in tika-server-core.
* In tika-server, for those parsers that can be configured per parse via a config object
passed in through the ParseContext, the config object will only update those fields
that the user has modified. The config object will no longer
fully reset all settings to the default settings per parse.
This has a more intuitive "update the base/configured settings" with
what has been changed in the config object.
* tika-eval
* tika-eval's default profile and comparison reports no longer include tag reports.
Users can get the report configs that include tags (*-tags.xml):
https://github.com/apache/tika/tree/main/tika-eval/tika-eval-app/src/main/resources
Release 1.27 - 06/30/2021
* Migrate MP4 parsing to Drew Noakes' metadata-extractor (TIKA-3459).
To revert to legacy parser turn off NoakesMP4Parser and turn on MP4Parser
via tika-config.xml.
* Prevent rare infinite loop in tika-server's -spawnChild mode
when restart fails because of failure to bind to the port (TIKA-3441).
* Improve likelihood that tesseract will not be orphaned on
jvm restart in tika-server (TIKA-3441).
* Deprecate experimental PDFPreflightParser (TIKA-3437).
* Apply encoding detection to zip entry names via Ryan421 (TIKA-3374).
* Add json output for /tika endpoint in tika-server (TIKA-3352).
* Tika's PDFParser should use the underlying file if one is passed in
via a TikaInputStream (TIKA-3350)
Release 1.26 - 03/24/2021
* Fix thread safety bug in OpenOffice parser (TIKA-3334).
* The "writeLimit" header now pertains to the combined characters
written per container document (and embedded documents) in the /rmeta
endpoint in tika-server (TIKA-3325); it no longer functions only
per container or embedded document.
* Extract more embedded files in PDFs by recursively processing the
embedded file tree (TIKA-3332).
* Allow for case insensitive headers for configuration of the PDFParser
and the TesseractOCRParser in tika-server via Subhajit Das (TIKA-3320).
* Improve detection and parsing of XPS files (TIKA-3316).
* General dependency upgrades (TIKA-3244).
* Great optimization in ForkParser (TIKA-3237).
* Fix parsing of emails attached to other emails in PST files (TIKA-3004).
* MP3 parser should output the xmpDM:duration metadata as seconds not
milliseconds, consistent with the other Audio and Video parsers (TIKA-3318).
* MP4 parser check if any of the Compatible Brands match when identifying
the subtype (TIKA-3310).
Release 1.25 - 11/25/2020
* Fix inconsistent license in xmpcore (TIKA-3204).
* General upgrades including some dependencies with
recently found security vulnerabilities (TIKA-3119).
* Add detection and a parser for flat ODF files (TIKA-3159).
* Add extraction of macros from ODF files (TIKA-3161).
* Add mime detection for hprof and hprof text files (TIKA-3144).
* Add TextSignature and TextProfileSignature to tika-eval (TIKA-3145 and TIKA-3146)
* Create a metadata filter to trigger tika-eval stats post parsing (TIKA-3140)
* Add a configurable metadata-filter for the RecursiveParserWrapper (TIKA-3137)
* Parameterize writeLimit and maxEmbeddedResources for RecursiveParserWrapper
in tika-server (TIKA-3133)
* Add status endpoint to tika-server (TIKA-3129).
* Remove whitelist/blacklist terminology (TIKA-3120)
* Add detection for parquet files (TIKA-3115).
* Add detection and parsing for bplist (TIKA-3104).
* Enable metadata value filtering for RecursiveParserWrapper (TIKA-3137)
* Add a basic parser for plist files based on com.googlecode.plist:dd-plist (TIKA-3104).
* Read hyperlinked images from ODT files (TIKA-3156).
* Updated GrobidRESTParser to use new API location (TIKA-3191).
* Add FileProfiler to tika-eval (TIKA-3216).
* Add status endpoint to tika-server (TIKA-3129).
* Improved handling of zip files with STORED entries with
data descriptor (TIKA-3196).
* Add parsers for XLZ, IDML and MIF (TIKA-2976, TIKA-3188 and TIKA-3189).
* Add the beginnings of a format-aware fuzzing module (TIKA-3083).
* Add wrapper for Linux 'file' command for mime detection (TIKA-3215).
* Added ability to skip parsing of embedded files in Tika Server (TIKA-3227).
Release 1.24.1 - 4/17/2020
* Allow gzip compression of input and output streams for tika-server (TIKA-3073).
Release 1.24 - 3/11/2019
* Add scripts to run tika-server as a service via Eric Pugh,
and add these scripts and jar as a new artifact in the release (TIKA-3010).
* Upgrade Drew Noakes' metadata-extractor (TIKA-2952).
* Enable optional extraction of structural tags in PDFs (alpha-grade) (TIKA-3026).
* Tika app's --extract mode now outputs to STDOUT (TIKA-3035).
* Add an optional Preflight parser for PDFs (TIKA-3055).
* Improve detection of some zip-based formats (TIKA-3057).
* Upgrade metadata-extractor to 2.13.0 (TIKA-2952).
* Upgrade to POI 4.1.2 (TIKA-3047).
* Extract XMP from PSD files (TIKA-3050).
* Added XMLProfiler as an optional parser to profile XFA and XMP
in PDFs (TIKA-3045).
* Extract inline images that rely on the DCT filter from PDFs (TIKA-3041).
* Upgrade to PDFBox 2.0.19 (TIKA-3033).
* Fix bug in ASM parser configuration (TIKA-2992).
* Upgrade to java-libpst 0.9.3 (TIKA-2546).
* Fixed XLIFF12Parser failures with ToXMLHandler (TIKA-3014).
Release 1.23 - 12/02/2019
* NOTE: The PDFParser now relies on OCRDPI to render page images when
users configure OCR on rendered page images. This will have the effect
of increasing rendered image size (TIKA-2624).
* NOTE: tika-server no longer returns 415 for file types for which there
is no parser.
* Fix bug in AUTO OCR strategy in the PDFParser (TIKA-3002).
* Fix incorrect height and width metadata extraction from JPEG images (TIKA-2630).
* Upgrade to POI 4.1.1 (TIKA-2851).
* Upgrade to PDFBox 2.0.17 (TIKA-2951).
* Ensure that the PDFParser respects custom configuration of Tesseract
from tika-config.xml via Eric Pugh (TIKA-2970).
* Add parser for XLIFF v1.2 files (TIKA-2975).
* Add mime type detection support for WebAssembly (TIKA-2894),
HEIF / HEIC images (TIKA-2942), Digilite FDF (TIKA-2988);
and xml-root detection for XFDF (TIKA-2990) and XDP (TIKA-2989).
* Add an XLZ Parser (TIKA-2976).
* Fix deadlock with ForkParser when InputStream throws IOException (TIKA-2892).
Release 1.22 - 07/29/2019
* NOTE: tika-server no longer hard-codes the HtmlParser to handle
XML files (TIKA-2910). Users must now configure that behavior
via a tika-config.xml file.
* NOTE: Known regression: PDFBOX-4587 -- PDF passwords with codepoints
between 0xF000 and 0XF0000 will cause an exception.
* Add parser for HWP v5 files via SooMyung Lee (soomyung) and
JinSup Kim (ddoleye) (TIKA-2909).
* Fix order of closing streams to avoid "Failed to close temporary resource"
exception in TesseractOCRParser (TIKA-2908).
* Improve AutoDetectReader performance by caching encoding
detector (TIKA-1568).
* Prevent RTFParser from outputting illegal tag combinations (TIKA-2889).
* Fix RereadableInputStream to release all resources (TIKA-2903).
* Implement custom language identifier in the tika-eval module based on
OpenNLP's language detector; add 18 languages and add common words
lists for all 121 languages (TIKA-2790).
* Fix NPE in MimeTypesReader.releaseParser() via Eamonn Saunders (TIKA-2896).
* Fix RTFParser to extract more content (TIKA-2883).
* Add clientSubmitTime to the metadata extracted from PST files (TIKA-2898).
* Improve StreamingZipContainerDetector for xltx, xltm and
several other file formats (TIKA-2886).
Release 1.21 - 05/14/2019
* Add optional AUTO mode to OCR'ing of PDFs. If tesseract is installed
and on the path, and this option is selected programmatically
or via TikaConfig(), the PDFParser will use heuristics to decide
whether or not to run OCR per page on PDFs. (TIKA-2749)
* The ZipContainerDetector's default behavior was changed to run
streaming detection up to its markLimit. Users can get the
legacy behavior (spool-to-file/rely-on-underlying-file-in-TikaInputStream)
by setting markLimit=-1. The POIFSContainerDetector requires an underlying file;
it will try to spool the file to disk; if the file's length is > markLimit,
it will not attempt detection; set markLimit to -1 for legacy behavior (TIKA-2849).
* Upgrade PDFBox to 2.0.14 (TIKA-2834).
* Add CSV detection and replace TXTParser with TextAndCSVParser;
users can turn off CSV detection by excluding the TextAndCSVParser
and adding back the TXTParser via tika-config (TIKA-2833).
* Add a CSVParser. CSV detection is currently based solely on filename
and/or information conveyed via Metadata (TIKA-2826).
* General upgrades: asm, bouncycastle, commons-codec, commons-lang3, cxf,
guava, h2, httpcomponents, jackcess, junrar, Lucene, mime4j, opennlp, parso,
sqlite-jdbc (provided), zstd-jni (provided) (TIKA-2824)
* Bundle xerces2 with tika-parsers (TIKA-2802).
* Upgrade jaxb to 2.3.2 (TIKA-2819).
* Upgrade jackson to 2.9.8 (TIKA-2717).
* Update tika-eval's common tokens lists (TIKA-2822).
* Handle bad tags in tika-eval more robustly (TIKA-2810).
* Add reports for tags in tika-eval (TIKA-2809).
* Extract text from SDT element within textboxes in .docx files (TIKA-2807).
* Try to handle truncated OOXML files more robustly (TIKA-2765).
Release 1.20 - 12/17/2018
* Upgrade to POI 4.0.1 (TIKA-2751).
* Integrate/parameterize new angles handling in
PDFBox (TIKA-2779).
* Upgrade to PDFBox 2.0.13 (TIKA-2788).
* Prevent content within <style/> and <script/> elements
to be written in the ToTextContentHandler (TIKA-2550).
* Switch child to parent communication to a shared memory-mapped
file in tika-server's -spawnChild mode.
* Fix bug in tika-server when run in legacy mode (not -spawnChild)
that caused it to return 503 on documents submitted after
it hit an OutOfMemoryError (TIKA-2776).
* Upgrade jaxb-runtime and javax.activation (TIKA-2778).
* tika-app in batch mode now requires an interrupt or
kill signal to the parent process to stop the parent
and the child processes (TIKA-2780).
* Bulk upgrade of dependencies (TIKA-2775).
* Improve language id efficiency in tika-eval (TIKA-2777).
* Upgrade sqlite "provided" dependency to 3.25.2 (TIKA-2773).
* Remove duplication of notes in PPT slides (TIKA-2735)
* Use -javaHome or $JAVA_HOME (if they exist) when
spawning child in tika-server's -spawnChild mode.
* Fixed closing of styles around Hyperlinks in Word Parser
Contributed by Ronan O'Sullivan (TIKA-2599).
Release 1.19.1 - 10/4/2018
* Update PDFBox to 2.0.12, jempbox to 1.8.16
and jbig2 to 3.0.2 (TIKA-2745).
* Fix regression in parser for MP3 files (TIKA-2730).
* Updated Python Dependency Check for TesseractOCR (TIKA-2740).
* Improve SAXParser robustness (TIKA-2727).
* Remove dependency on slf4j-log4j12 by upgrading jmatio (TIKA-2742).
* Replace com.sun.xml.bind:jaxb-impl and jaxb-core with
org.glassfish.jaxb:jaxb-runtime and jaxb-core (TIKA-2743)
Release 1.19 - 9/14/2018
* Require Java 8 (TIKA-2679).
* Enable building with Java 11 (TIKA-2668)
* Add an option to make tika-server robust against infinite loops,
OOMs, and memory leaks (TIKA-2725).
* Allow configuration of the Tesseract parser via the standard
tika-config.xml options (TIKA-2705).
* Improve handling of empty cells across table-based
formats (TIKA-2479).
* Add a Standards compliant HTML encoding detector
via Gerard Bouchar (TIKA-2673).
* Improved XML parsing -- limited default entity expansions to 20.
To raise this limit, add -Djdk.xml.entityExpansionLimit=XXX to
your commandline.
* Mime magic improvements for Olympus RAW (TIKA-2658), interpreted
server-side languages via HTTP (TIKA-2648), MHTML (TIKA-2723)
* Add absolute timeout to ForkParser rather than testing
for active (TIKA-2656).
* Make the RecursiveParserWrapper work with the ForkParser (TIKA-2655).
* Allow the ForkParser to specify a directory containing tika-app.jar
for use by the ForkServer. This allows users to keep most of the
parser dependencies out of their code; and it allows for an easy
addition of optional jars for Parser dependencies,
such as the xerial sqlite jar (TIKA-2653).
* Use a pool for SAXParsers and DOMBuilders rather than creating
a new parser/builder for every parse.
For better performance, set XMLReaderUtils.setPoolSize() to the
number of threads you're using with Tika (TIKA-2645).
* Add the RecursiveParserWrapperHandler to improve the RecursiveParserWrapper
API slightly (TIKA-2644).
* Upgraded to Commons-Compress 1.18 (TIKA-2707).
* Upgraded to Apache POI 4.0.0 (TIKA-2552).
* Upgraded to Apache PDFBox 2.0.11 (TIKA-2681).
* Upgraded to deeplearning4j 1.0.0-beta2 (TIKA-2672).
* Upgraded jmatio to 1.4 (TIKA-2667)
* Upgraded Apache Lucene to 7.4.0 in tika-eval and tika-examples (TIKA-2695).
* Upgraded junrar to 1.0.1 (TIKA-2664).
* Numerous other upgrades (TIKA-2692).
* Excluded Spring as a transitive dependency (TIKA-2721).
Release 1.18 - 4/20/2018
* Upgrade jackson to 2.9.5 (TIKA-2634).
* Add support for brotli (TIKA-2621).
* Upgrade PDFBox to 2.0.9 and include new jbig2-imageio
from org.apache.pdfbox (TIKA-2579 and TIKA-2607).
* Support for TIFF images in PDF files (TIKA-2338)
* Detection of full encrypted 7z files (TIKA-2568)
* Various new mimes and typo fixes in tika-mimetypes.xml
via Andreas Meier (TIKA-2527).
* Revert to listenForAllRecords=false in ExcelExtractor
via Grigoriy Alekseev (TIKA-2590)
* Add workaround to identify TIFFs that might confuse
commons-compress's tar detection via Daniel Schmidt
(TIKA-2591)
* Ignore non-IANA supported charsets in HTML meta-headers
during charset detection in HTMLEncodingDetector
via Andreas Meier (TIKA-2592)
* Add detection and parsing of zstd (if user provides
com.github.luben:zstd-jni) via Andreas Meier (TIKA-2576)
* Allow for RFC822 detection for files starting with "dkim-"
and/or "x-" via Andreas Meier (TIKA-2578 and TIKA-2587)
* Extract xlsx files embedded in OLE objects within PPT and PPTX
via Brian McColgan (TIKA-2588).
* Extract files embedded in HTML and javascript inside HTML
that are stored in the Data URI scheme (TIKA-2563).
* Extract text from grouped text boxes in PPT (TIKA-2569).
* Extract language metadata item from PDF files via Matt Sheppard (TIKA-2559)
* RFC822 with multipart/mixed, first text element should be treated
as the main body of the email, not an attachment (TIKA-2547).
* Swap out com.tdunning:json for com.github.openjson:openjson to avoid
jar conflicts (TIKA-2556).
* No longer hardcode HtmlParser for XML files in tika-server (TIKA-2551).
* Require Java 8 (TIKA-2553).
* Add a parser for XPS (TIKA-2524).
* Mime magic for Dolby Digital AC3 and EAC3 files
* Fixed bug where TesseractOCRParser ignores configured ImageMagickPath,
and set rotation script to ignore Python warnings (TIKA-2509)
* Upgrade geo-apis to 3.0.1 (TIKA-2535)
* Mime definition and magic improvements for text-based programming
and config formats (TIKA-2554, TIKA-2567, TIKA-1141)
* Added local Docker image build using dockerfile-maven-plugin to allow
images to be built from source (TIKA-1518).
* Support for SAS7BDAT data files (TIKA-2462)
* Handle .epub files using .htm rather than .html extensions for the
embedded contents (TIKA-1288)
* Mime magic for ACES Images (TIKA-2628) and DPX Images (TIKA-2629)
* For sparse XLSX and XLSB files, always output missing cells to
the left of filled ones (matching XLS), and optionally output
missing rows on all 3 formats if requested via the
OfficeParserContext (TIKA-2479)
Release 1.17 - 12/8/2017
***NOTE: THIS IS THE LAST VERSION OF TIKA THAT WILL RUN
ON Java 7. The next versions will require Java 8***
* Fix thread-safety in ChmExtractor (TIKA-2519).
* Upgrade cxf to 3.0.16 (TIKA-2516).
* Allow users to configure maxMainMemoryBytes for PDFs via shrike (PR-213).
* Extract underline and strikethrough in docx (TIKA-2347 and TIKA-2512).
* Cache TikaConfig in EmbeddedDocumentUtil for better performance
in documents with large number of attachments (TIKA-2511).
* Extract media files from ooxml (TIKA-2510).
* Standardize the way the Image and Video captioning
dockers and extraction work (TIKA-2400, GitHub-208)
* Upgrade to xmpcore 5.1.3 (TIKA-2034).
* Upgrade to metadata-extractor 2.10.1 (TIKA-2486).