Skip to content
This repository has been archived by the owner. It is now read-only.

Add entity extraction capability #21

Merged
merged 6 commits into from Jun 22, 2017
Merged

Add entity extraction capability #21

merged 6 commits into from Jun 22, 2017

Conversation

@c-w
Copy link
Contributor

@c-w c-w commented Jun 20, 2017

The Cassandra schema contains a field for entities separate from places so this pull request ensures that we have the necessary data available to write the entity information to Cassandra.

@c-w c-w requested a review from kevinhartman Jun 20, 2017
@c-w c-w force-pushed the extract-entities branch from 09ff41d to 8ea6649 Jun 21, 2017
@c-w c-w requested a review from erikschlegel Jun 21, 2017
@c-w c-w force-pushed the extract-entities branch 2 times, most recently from dfac34a to 4f00634 Jun 21, 2017
Copy link
Contributor

@kevinhartman kevinhartman left a comment

LGTM

Loading


import scala.collection.JavaConversions._
import scala.util.{Failure, Success, Try}
import com.microsoft.partnercatalyst.fortis.spark.transforms.nlp.OpeNER.entityIsPlace
Copy link
Contributor

@kevinhartman kevinhartman Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: In this case, I think qualifying the name inline would make it more obvious to the reader that this helper is part of OpeNER.

Loading

Copy link
Contributor Author

@c-w c-w Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2934106.

Loading


def extractPeople(text: String, language: String): List[Tag] = {
entityRecognizer.extractEntities(text, language).filter(entityIsPerson)
.map(entity => Tag(name = entity.getStr, confidence = 1.0))
Copy link
Contributor

@kevinhartman kevinhartman Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my own understanding, why are we using tags for people but not place entities?

Loading

Copy link
Contributor Author

@c-w c-w Jun 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, not happy with that inconsistency too. Fixed in e0daddc.

Loading

Copy link
Contributor

@jcjimenez jcjimenez left a comment

LGTM

Loading

@c-w c-w force-pushed the extract-entities branch from 4f00634 to 2934106 Jun 22, 2017
@c-w c-w merged commit a86af0d into master Jun 22, 2017
2 checks passed
Loading
@c-w c-w deleted the extract-entities branch Jun 22, 2017
@c-w c-w removed the in progress label Jun 22, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants