EntityExtractor().setRequireSentences(True) throws error.

[clayms]

EntityExtractor().setRequireSentences(True)

throws this error:

Py4JJavaError: An error occurred while calling o461.getParam.
: java.util.NoSuchElementException: Param requireSentences does not exist.

From the EntityExtractor.scala file, I found the sentences parameter defined on lines 99 to 105, but when I changed lines 191 and 192 in annotator.py from:

def setRequireSentences(self, value):
    return self._set(requireSentences=value)

to:

def setRequireSentences(self, value):
    return self._set(sentences=value)

The error was basically the same:

Py4JJavaError: An error occurred while calling o474.getParam.
: java.util.NoSuchElementException: Param sentences does not exist.

[aleksei-ai]

It seems that EntityExtractor works totally incorrect. Will rewrite it in a day.

[aleksei-ai]

@clayms Thank you for the report.
I've recreated EntityExtractor in branch https://github.com/JohnSnowLabs/spark-nlp/tree/rewrite_entity_extractor
Here is an example notebook how to use it https://github.com/JohnSnowLabs/spark-nlp/blob/rewrite_entity_extractor/python/example/entities-extractor/extractor.ipynb

Will release it soon

[sofianeh]

I can't have the EntityExtractor() match with any of my entities even though my documents contain those. Here's a simple example I've built in Databricks, as you can see entitites column is an empty list
https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/3306898732834076/1537424650694939/3546260386063631/latest.html

[aleksei-ai]

@sofianeh It seems that file /dbfs/testfile.txt is local file not an hdfs file.

Can you try file:///dbfs/testfile.txt instead?

[sofianeh]

Already tried with both file:///dbfs/testfile.txt and file:/dbfs/testfile.txt. They end up with this error:
Caused by: java.io.FileNotFoundException: file:/dbfs/testfile.txt (No such file or directory)

[saif-ellafi]

@sofianeh there is a param called entitiesFormat. By default, this param is set to "TXT" which means it only reads from local files without any prefix (we could add parsing of the prefix). To use with prefix and/or to read from hdfs with prefix, you should set this param to "TXTDS"

This will use spark to read such file, and spark allows file:// or hdfs:// prefix. Either that or you remove the file:// prefix (this will usually be faster than TXTDS if the amount of data input is small)

use setEntitiesFormat("TXTDS")

Let us know how it goes! Thanks

[sofianeh]

Yes, using TXTDS prevents the error, but I still end up with empty entities. Is there something wrong in my example (content of testfile.txt, my documents, my Pipeline, ...)?
Also, the documentation mentions a setMaxLen(number) parameter but it seems no longer working.

[saif-ellafi]

@sofianeh what is the content of testfile.txt?? and thanks, setMaxLen doesn't exist anymore.

I have just tried this and worked for me (this is one of the unit tests we have there, I have changed it to TXTDS though)

    val entityExtractor = new EntityExtractor()
      .setInputCols("sentence", "normalized")
      .setInsideSentences(false)
      .setEntitiesFormat(TXTDS)
      .setEntitiesPath("./src/test/resources/entity-extractor/test-phrases.txt")
      .setOutputCol("entity")

en this example test-phrases looks like this:

dolore magna aliqua
Lorem ipsum dolor sit
laborum
anim id est laborum

so it basically searches this lines in the text

[sofianeh]

testfile.txt is created on the fly within the example notebook I've provided above

[saif-ellafi]

Right, sorry about that. Let me check i'll try the same thing