Tika-Similarity uses the Tika-Python package (Python port of Apache Tika) to compute file similarity based on Metadata features.
Switch branches/tags
Clone or download
chrismattmann Merge pull request #94 from skgb-1990/fix_error_space_in_filename
Handling case where there is a space in the filename
Latest commit 5407d37 Mar 19, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Solr-ES-Similarity add solr similarity Aug 29, 2016
snapshots Merge pull request #85 from akarshgoyal/master Mar 7, 2018
testdata ATF crawl similar cluster and circle cluster examples. Apr 21, 2015
wordlists Delete swear.txt Aug 18, 2016
.DS_Store Created D3 Treemap Visualization Mar 1, 2018
.gitignore fix for issue 49 Feb 23, 2016
LICENSE.txt ALv2 license. Dec 13, 2014
README.md fixed README.md for loading wrong sunburst image URL #2 Mar 8, 2018
affinity_propagation.py Add ALv2 license header. Mar 7, 2018
argK-means.py refactor k-means Mar 2, 2016
argK_tika_similarity.py argK with similarity implemented within it Apr 5, 2016
circle-packing.py fix header. Apr 30, 2015
circlepacking.html added circle packing of D3 Mar 13, 2015
cluster-d3.html update w/ tika-python(sync w/ tika-1.8) May 7, 2015
cluster-scores.py accept specific mime types Jan 18, 2016
cluster_barchart.html Removed unnecessary code and changed style Feb 28, 2018
cluster_force.html Added cluster force d3 visualizations Mar 1, 2018
cluster_force.png Added cluster force d3 visualizations Mar 1, 2018
cluster_force_hover.png Added cluster force d3 visualizations Mar 1, 2018
compositeViz.html update d3 viz May 7, 2015
correlation-matrix-d3.html A D3 Correlation Matrix for depicting visualizations Mar 2, 2018
cosine_similarity.py Cleanup and added comments Mar 2, 2018
dynamic-circlepacking.html really interesting circle packing, click on the circle, and wait a li… Apr 3, 2015
dynamic-cluster.html update d3 viz May 7, 2015
edit-cosine-circle-packing.py Handling case where there is a space in the filename Mar 19, 2018
edit-cosine-cluster.py Handling case where there is a space in the filename Mar 19, 2018
edit-cosine-correlation-matrix.py A D3 Correlation Matrix for depicting visualizations Mar 2, 2018
edit-value-similarity.py Update edit-value-similarity.py Mar 9, 2018
features.py moved imports to top Aug 15, 2016
generateLevelCluster.py add viz image May 11, 2015
indented_tree.html Adding indented tree visualization using d3 Mar 2, 2018
index.html change default to clusters2 example. Mar 6, 2015
jaccard_similarity.py fixed incorrect message Mar 2, 2016
levelCluster-d3.html changed for pdf format May 7, 2015
metalevenshtein.py Added command line options for stylictic feature similarity. Added mo… Aug 15, 2016
psykey.py Added command line options for stylictic feature similarity. Added mo… Aug 15, 2016
similarity.py fix: max retries exceeded Mar 4, 2016
sk_kmeans.py trivial commit Apr 16, 2016
sunburst.html added sunburst visualization (team #13) Mar 2, 2018
sunburst.py added sunburst visualization (team #13) Mar 2, 2018
tree_map.html Created D3 Treemap Visualization Mar 1, 2018
tree_map.py Add ALv2 license. Mar 7, 2018
value-similarity.py recursively list should use itertools Apr 12, 2017
vector.py updated to select attributes from config file Mar 4, 2018

README.md

Apache Tika File Similarity based on Jaccard distance, Edit distance & Cosine distance

This project demonstrates using the Tika-Python package (Python port of Apache Tika) to compute file similarity based on Metadata features.

The script can iterate over all files in the current directory or given files by command line and derives their metadata features, then computes the union of all features. The union of all features become the "golden feature set" that all document features are compared to via intersect. The length of that intersect per file divided by the length of the unioned set becomes the similarity score.

Scores are sorted in reverse (descending) order which can be shown in three different Data-Driven document visualizaions. A companion project to this effort is Auto Extractor which uses Apache Spark and Apache Nutch to take web crawl data, and produce D3-visualizations and clusters of similar pages.

Pre-requisite

  1. Install Tika-Python

Installation

pip install editdistance
git clone https://github.com/chrismattmann/tika-img-similarity

You can also check out ETLlib

How to use

Optional: Compute similarity only on specific IANA MIME Type(s) inside a directory using --accept

Key-based comparison

This compares metadata feature names as a golden feature set

#!/usr/bin/env python2.7
python similarity.py -f [directory of files] [--accept [jpeg pdf etc...]]
or 
python similarity.py -c [file1 file2 file3 ...]

Value-based comparison

This compares metadata feature names together with its value as a golden feature set

#!/usr/bin/env python2.7
python value-similarity.py -f [directory of files] [--accept [jpeg pdf etc...]]
or 
python value-similarity.py -c [file1 file2 file3 ...]

Edit Distance comparison on Metadata Values

  • This computes pairwise similarity scores based on Edit Distance Similarity.
  • Similarity Score of 1 implies an identical pair of documents.
#!/usr/bin/env python2.7
python edit-value-similarity.py [-h] --inputDir INPUTDIR --outCSV OUTCSV [--accept [png pdf etc...]] [--allKeys]

--inputDir INPUTDIR  path to directory containing files

--outCSV OUTCSV      path to directory for storing the output CSV File, containing pair-wise Similarity Scores based on Edit distance

--accept [ACCEPT]    Optional: compute similarity only on specified IANA MIME Type(s)

--allKeys            Optional: compute edit distance across all metadata keys of 2 documents, else default to only intersection of metadata keys

Eg: python edit-value-similarity.py --inputDir /path/to/files --outCSV /path/to/output.csv --accept png pdf gif

Cosine Distance comparison on Metadata Values

  • This computes pairwise similarity scores based on Cosine Distance Similarity.
  • Similarity Score of 1 implies an identical pair of documents.
#!/usr/bin/env python2.7
python cosine_similarity.py [-h] --inputDir INPUTDIR --outCSV OUTCSV [--accept [png pdf etc...]]

--inputDir INPUTDIR  path to directory containing files

--outCSV OUTCSV      path to directory for storing the output CSV File, containing pair-wise Similarity Scores based on Cosine distance

--accept [ACCEPT]    Optional: compute similarity only on specified IANA MIME Type(s)

Similarity based on Stylistic/Authorship features

  • This calculates pairwise cosine similarity on bag of signatures/features produced by extracting stylistic/authorship features from text.
#!/usr/bin/env python2.7
python psykey.py --inputDir INPUTDIR --outCSV OUTCSV --wordlists WRODLIST_FOLDER

--inputDir INPUTDIR  path to directory containing files

--outCSV OUTCSV      path to directory for storing the output CSV File, containing pair-wise Similarity Scores based on Cosine distance of stylistic and authorship features

--wordlists WRODLIST_FOLDER    path to the folder that contains files that are word list belonging to different classes. eg Use the wordlist folder provided with the tika-similarity library. If adding your own, make sure that the file is .txt with one word per line. Also, the name of the file will be considered the name of the class.

Metalevenshtein string distance

  • This calculates Metalevenshtein (Inspired by the paper : Robust Similarity Measures for Named Entities Matching by Erwan et al.) distance between two strings.
#!/usr/bin/env python2.7

Usage:

import metalevenshtein as metalev
print metalev.meta_levenshtein('abacus1cat','cat1cus')


To use all the argument options in this function:

def meta_levenshtein(string1,string2,Sim='levenshtein',theta=0.5,strict=-1,idf=dict()):

    Implements ideas from the paper : Robust Similarity Measures for Named Entities Matching by Erwan et al.
    Sim = jaro_winkler, levenshtein : can be chosen as the secondary matching function.
    theta is the secondary similarity threshold: If set higher it will be more difficult for the strings to match.
    strict=-1 for doing all permutations of the substrings
    strict=1 for no permutations
    idf=provide a dictionary for {string(word),float(idf od the word)}: More useful when mathings multi word entities (And word importances are very important)
    like: 'harry potter', 'the wizard harry potter'

Bell Curve fitting and overlap

  • Fits two datasets into bel curves and finds the area of overlap between the bell curves.
#!/usr/bin/env python2.7
import features as feat
data1=[1,2,3,3,2,1]
data2=[4,5,6,6,5,4]
area,error=feat.gaussian_overlap(data1,data2)
print area

D3 visualization

Cluster viz

  • Jaccard Similarity
* python cluster-scores.py [-t threshold_value] (for generating cluster viz)
* open cluster-d3.html(or dynamic-cluster.html for interactive viz) in your browser
  • Edit Distance & Cosine Similarity
* python edit-cosine-cluster.py --inputCSV <PATH TO CSV FILE> --cluster <INTEGER OPTION> (for generating cluster viz)

  <PATH TO CSV FILE> - Path to CSV file generated by running edit-value-similarity.py or cosine_similarity.py
  <INTEGER OPTION> - Pass 0 to cluster based on x-coordinate, 1 to cluster based on y-coordinate, 2 to cluster based on similarity score

* open cluster-d3.html(or dynamic-cluster.html for interactive viz) in your browser

Default threshold value is 0.01.

Circlepacking viz

  • Jaccard Similarity
* python circle-packing.py (for generating circlepacking viz)
* open circlepacking.html(or dynamic-circlepacking.html for interactive viz) in your browser
  • Edit Distance & Cosine Similarity
* python edit-cosine-circle-packing.py --inputCSV <PATH TO CSV FILE> --cluster <INTEGER OPTION> (for generating circlepacking viz)

  <PATH TO CSV FILE> - Path to CSV file generated by running edit-value-similarity.py or cosine_similarity.py
  <INTEGER OPTION> - Pass 0 to cluster based on x-coordinate, 1 to cluster based on y-coordinate, 2 to cluster based on similarity score


* open circlepacking.html(or dynamic-circlepacking.html for interactive viz) in your browser

Composite viz

This is a combination of cluster viz and circle packing viz. The deeper color, the more the same attributes in the cluster.

* open compositeViz.html in your browser

Image of composite viz

Sunburst viz

Visualization of clustering from Jaccard Similarity result

* python sunburst.py (for generating circlepacking viz)
* open sunburst.html

Image of sunburst viz

Big data way

if you are dealing with big data, you can use it this way:

* python generateLevelCluster.py (for generating level cluster viz)
* open levelCluster-d3.html in your browser

You can set max number for each node _maxNumNode(default _maxNumNode = 10) in generateLevelCluster.py Image of level composite viz

Treemap viz

* python tree_map.py (for generating treemap viz)
* open tree_map.html in your browser

Image of treemap viz

Questions, comments?

Send them to Chris A. Mattmann.

Contributors

  • Chris A. Mattmann, JPL
  • Dongni Zhao, USC
  • Harshavardhan Manjunatha, USC
  • Thamme Gowda, USC
  • Ayberk Yılmaz, USC
  • Aravind Ram, USC
  • Aishwarya Parameshwaran, USC
  • Rashmi Nalwad, USC
  • Asitang Mishra, JPL
  • Suzanne Stathatos, JPL

License

This project is licensed under the Apache License, version 2.0.