Mode 1527 - Ported & updated text extractors from 2.x #430
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
June 29, 2012 15:50
…nd updated the binary store to extract the text and mime-type of binary values Working on this, exposed how fragile - lock-wise - is working with the SharedLockingInputStream (FileSystemBinaryStore). Therefore, I've updated the mime-type detection so that mark & reset are avoided as much as possible, also making sure that streams are closed after each detector finishes with them. The Tika version was bumped to 1.1 which required also the update of the POI version to 3.8.
…th the indexing process. This meant that a few changes were needed: - the text extractors configuration has been updated to resemble that of the sequencers - the binary store interface has been updated to be able to store and retrieve extracted text for a given binary (source) value - the TextExtractors class was changed to become the entry point into text extraction
…ptively by the binary storage, when a binary value is created. For this to be possible, the context of the extractor cannot contain any node-specific information. Also, this exposed an issue with the SharedLockingInputStream: if the stream is closed in the "read" methods, Tika's parsers will keep reading it over and over (effectively reopening it each time) either causing OOM errors or duplicate text. This means the "close" call from the read methods has been removed.
…tractors. To validate the configuration and Arquillian integration test was added as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Tika text extractor is now located in extractors/modeshape-extractor-tika. In addition to porting the text extractors, a few important changes were made: