RAT-54: Tika based document analyzer

[Claudenw]

Switches to Tika to determine document type.

** this change depends upon #233 **
After the above change is merged this change will be approx 10 files -- most of them tests.

json files are now STANDARD not BINARY

fixes RAT-20 Detection of binaries should be smarter
fixes RAT-54 MIME Detection Using Tika
fixes RAT-147 binary guesser design improvement
fixes RAT-150 RAT should use Apache Tika to simply guess ignored [application/X] file types and focus on the [text/Y] family as a sensible default
fixes RAT-211 Generated rat-output.xml must be well-formed, even if BinaryGuesser fails
fixes RAT-301 Chinese characters comments are not recognized as binary anymore (thanks to Tika)

Checklist

• add documentation for default file exclusion listing *.json files as the excluded file. Also note that setting the exclusion will replace the existing entry, so it should be included in replacements. NOT NEEDED as .json is no longer excluded but again categorized as BINARY.
• add documentation for the default directory exclusion of hidden (.*) files. Also note that setting the exclusion will replace the existing entry. (There is no non-programatic access to changing the directory)
• Add tests for ArchiveWalker. (https://issues.apache.org/jira/browse/RAT-372)

[ottlinger]

I'd prefer a file filter that allows ignoring no-comment plaintext files such as JSON.

[Claudenw]

I am adding a file filter to remove json files. Initially this will be hard coded. I will open a subsequent ticket to generalize it so that we can define a list of extensions to remove. This will probably mean more command line options. So I'll have to open that can of worms as well.

[ottlinger]

Pls add a reference to all the old tickets in the changelog & thanks for taking care of the old tickets/bugs.

[Claudenw]

Well this blew up to something bigger than I wanted but...

I added default exclusion for "*.json" files in the Default class and used that to configure the ReportConfiguration defaults.
In the future the default exclusions should be configured in Default class.

I changed the parameter names and instance variables to "filesToIgnore" and "directoriesToIgnore" to make it clear what the filters were doing.

I cleaned up the DirectoryWalker and Walker classes.

I ensured that the filesToIgnore and directoriesToIgnore alwasy have a value (not null). If set to null in the configuration the value is translated into a filter that always returns false.

I think there may be an issue in the directoreisToIgnore processing, but it has been there from the beginning. I will investigate and open a ticket if necessary.

[ottlinger]

Thanks for all the fixes while introducing Tika to RAT - big kudos.

Should we add a note to the webpage or is it just an entry in the release changelog concerning the use of Tika for file type detection? WDYT?

[ottlinger]

Isn't that what RAT-301 is all about? I got curious when I read your comment in the changelog here.

[Claudenw]

147 is improve the guesser so that non UTF-8 text is not detected as binary.
301 is extended (Chinese in the report) characters that are UTF-8 encoded being detected as binary.

related but not the same. I was waiting for example to test 301 with.

[ottlinger]

https://github.com/apache/linkis/blob/master/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/udf/entity/UDFVersion.java

[ottlinger]

@Claudenw would you mind integrating above file in order to see if its properly detected with Tika? This could allow solving RAT-301 as well :)

[ottlinger]

@Claudenw I brought in changes related to RAT-301 - is that fine for you or should the Chinese character example go somewhere else? Thanks

[ottlinger]

@Claudenw the extraction into the Tika-class looks very nice - thanks.

[Claudenw]

@ottlinger If you approve I can merge this. If you want more eyes on it, lets's invite a few reviewers.

[ottlinger]

@ottlinger If you approve I can merge this. If you want more eyes on it, lets's invite a few reviewers.

In the PR's main description you've created a check list - is that already done?
I'm still thinking if we should add more documentation about the change or leave to a bigger entry in the release notes about possibly changing behaviour of RAT.

[Claudenw]

I updated the checklist.

[ottlinger]

@Claudenw pls review my latest additions concerning RAT-301, after that go ahead with the merge. Thanks for your work and the cool addition of more functionality to RAT #kudos

[ottlinger]

Thanks for all your work.