Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. #865

alexiudice · 2018-07-18T13:43:46Z

Adds text similarity/distance methods and double metaphone text encoding.

Added Levenshtein Distance and Similarity code and test.

Added Hamming Distance code and test.

Added Jaro-Winkler Distance code and test.

Added Double Metaphone text encoding and test.

Added Apache commons-text dependency.

jexp · 2018-07-18T16:13:04Z

build.gradle

@@ -127,6 +127,7 @@ dependencies {
    compile group: 'com.github.javafaker', name: 'javafaker', version: '0.10'

    compile group: 'org.apache.commons', name: 'commons-math3', version: '3.6.1'
+    compile group: 'org.apache.commons', name: 'commons-text', version: '1.2'


how big is this dependency?

Just checked on my machine, commons-text-1.2.jar is 136,544 bytes (139 KB on disk)

jexp · 2018-07-18T16:13:55Z

src/main/java/apoc/algo/Similarity.java

@@ -64,4 +67,52 @@ public double euclideanDistance(@Name("vector1") List<Number> vector1, @Name("ve
    public double euclideanSimilarity(@Name("vector1") List<Number> vector1, @Name("vector2") List<Number> vector2) {
        return 1.0d / (1 + euclideanDistance(vector1, vector2));
    }
+
+    @UserFunction
+    @Description("apoc.algo.levenshteinDistance(lhs, rhs) return the Levenshtein distance of two texts")


we already had that as "distance" function in apoc.text
your contributions should go there.

jexp · 2018-07-18T16:14:33Z

src/main/java/apoc/algo/Similarity.java

@@ -64,4 +67,52 @@ public double euclideanDistance(@Name("vector1") List<Number> vector1, @Name("ve
    public double euclideanSimilarity(@Name("vector1") List<Number> vector1, @Name("vector2") List<Number> vector2) {
        return 1.0d / (1 + euclideanDistance(vector1, vector2));
    }
+
+    @UserFunction
+    @Description("apoc.algo.levenshteinDistance(lhs, rhs) return the Levenshtein distance of two texts")


nothing should go into the apoc.algo package until further notice to reduce confusion with the neo4j-graph-algorithms library

jexp · 2018-07-18T16:15:00Z

src/main/java/apoc/algo/Similarity.java

+            return 0.0;
+        }
+
+        JaroWinklerDistance jwd = new JaroWinklerDistance();


how expensive is the construction? e.g. in a tight loop?

alexiudice · 2018-07-19T13:22:03Z

I’ve updated the PR with the following changes:

I've moved all of the added functions in apoc.algo to apoc.text and the tests from SimilarityTest.java to StringsTest.java.
I've changed apoc.text.distance function a bit. The apoc.text.levenshteinDistance function has a more descriptive name so I kept it in. It now it just calls apoc.text.distance. I edited the apoc.text.distance so it does not use the deprecated method StringUtils.getLevenshteinDistance
All three distance functions were instantiating a new object every call which can be expensive in a loop. To remedy this I've replaced these with calls to three static final objects added in the Strings.java class so they are instantiated only one time.

jexp · 2018-07-19T14:19:36Z

Great, thanks so much.

For the reuse, it is only an issue if the object creation is really expensive (think jackson ObjectMapper) can you make sure the static instances are thread-safe? Otherwise please revert it back if it's not expensive to create the instances.

nammmm · 2018-07-19T15:35:18Z

I think they are all thread safe as they are all static final fields. In addition, I just finished cleaning up the code (code style, description and parameter names).

jexp · 2018-07-19T16:58:56Z

Can you please add it to the docs, at least to overview.adoc ?
Thanks so much.

alexiudice · 2018-07-19T17:06:39Z

I'm pretty sure the added methods are thread safe since the Distance objects are immutable once instantiated and the apply methods parameters are final. We could add a synchronized block around the apply method calls if we really want to make sure they are thread safe.

@nammmm said he will add the documentation.

nammmm · 2018-07-19T17:48:23Z

I have added the functions to overview.adoc. Let me know if there is anything else I need to do.

alexiudice · 2018-07-19T18:08:23Z

~~Please do not merge yet! Testing some last things.~~

EDIT: All ready to go!

…ouble metaphone text encoder Added Apache commons-text dependency and used the LevenshteinDistance, HammingDistance and JaroWinklerDistance objects from the dependency to create the similarity/distance methods in Strings. Replaced deprecated StringUtils.getLevenshteinDistance with LevenshteinDistance. Added Double Metaphone encoding method in Phonetic.

…ne text encoder (#865) Added Apache commons-text dependency and used the LevenshteinDistance, HammingDistance and JaroWinklerDistance objects from the dependency to create the similarity/distance methods in Strings. Replaced deprecated StringUtils.getLevenshteinDistance with LevenshteinDistance. Added Double Metaphone encoding method in Phonetic.

…ouble metaphone text encoder (neo4j-contrib#865) Added Apache commons-text dependency and used the LevenshteinDistance, HammingDistance and JaroWinklerDistance objects from the dependency to create the similarity/distance methods in Strings. Replaced deprecated StringUtils.getLevenshteinDistance with LevenshteinDistance. Added Double Metaphone encoding method in Phonetic.

alexiudice changed the title ~~fixes #864: Adds text similarity/distance methods and double metaphone text encoder~~ Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. Jul 18, 2018

jexp reviewed Jul 18, 2018

View reviewed changes

jexp requested changes Jul 18, 2018

View reviewed changes

alexiudice force-pushed the story-864-3.4-distanceAndMetaphoneMethods branch from 5bace00 to ffb5c65 Compare July 19, 2018 13:14

nammmm force-pushed the story-864-3.4-distanceAndMetaphoneMethods branch 3 times, most recently from f42e6bf to d69f15e Compare July 19, 2018 15:32

nammmm force-pushed the story-864-3.4-distanceAndMetaphoneMethods branch 2 times, most recently from 5acbedf to c1167c8 Compare July 19, 2018 15:58

nammmm force-pushed the story-864-3.4-distanceAndMetaphoneMethods branch from c1167c8 to d0a4dba Compare July 19, 2018 17:44

nammmm force-pushed the story-864-3.4-distanceAndMetaphoneMethods branch from d0a4dba to d4eae9f Compare July 19, 2018 18:25

jexp merged commit bbc36ef into neo4j-contrib:3.4 Jul 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. #865

Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. #865

alexiudice commented Jul 18, 2018

jexp Jul 18, 2018

nammmm Jul 18, 2018

jexp Jul 18, 2018

jexp Jul 18, 2018

jexp Jul 18, 2018

alexiudice commented Jul 19, 2018

jexp commented Jul 19, 2018

nammmm commented Jul 19, 2018

jexp commented Jul 19, 2018

alexiudice commented Jul 19, 2018 •

edited

nammmm commented Jul 19, 2018

alexiudice commented Jul 19, 2018 •

edited

Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. #865

Fixes #864 - Adds text similarity/distance methods and double metaphone text encoder. #865

Conversation

alexiudice commented Jul 18, 2018

jexp Jul 18, 2018

Choose a reason for hiding this comment

nammmm Jul 18, 2018

Choose a reason for hiding this comment

jexp Jul 18, 2018

Choose a reason for hiding this comment

jexp Jul 18, 2018

Choose a reason for hiding this comment

jexp Jul 18, 2018

Choose a reason for hiding this comment

alexiudice commented Jul 19, 2018

jexp commented Jul 19, 2018

nammmm commented Jul 19, 2018

jexp commented Jul 19, 2018

alexiudice commented Jul 19, 2018 • edited

nammmm commented Jul 19, 2018

alexiudice commented Jul 19, 2018 • edited

alexiudice commented Jul 19, 2018 •

edited

alexiudice commented Jul 19, 2018 •

edited