Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detecting names in text fields (deprecated) #428

Closed
wants to merge 46 commits into from
Closed

Detecting names in text fields (deprecated) #428

wants to merge 46 commits into from

Commits on Oct 7, 2019

  1. First pass at name identification with European Union JRC data

    @MWYang
    MWYang committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    5f9f99d View commit details
    Browse the repository at this point in the history

  2. Perform dictionary check with one UDF

    @MWYang
    MWYang committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    180c7f1 View commit details
    Browse the repository at this point in the history

  3. Replaced expensive Spark counts with approximate versions

    @MWYang
    MWYang committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    fb5da80 View commit details
    Browse the repository at this point in the history

  4. Bad attempt at representing output of name identification with NameList

    @MWYang
    MWYang committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    b42469a View commit details
    Browse the repository at this point in the history

  5. Revert "Bad attempt at representing output of name identification wit… 

    …h NameList"

This reverts commit b42469a.
    @MWYang
    MWYang committed Oct 7, 2019
    Configuration menu
    Copy the full SHA
    f391e07 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2019

  1. Changed output type to NameMap

    @MWYang
    MWYang committed Oct 8, 2019
    Configuration menu
    Copy the full SHA
    b2c1663 View commit details
    Browse the repository at this point in the history

  2. First pass at gender identification with SSA data

    @MWYang
    MWYang committed Oct 8, 2019
    Configuration menu
    Copy the full SHA
    ea0ac44 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2019

  1. Made gender identification more robust to missing values; not attempt… 

    …ing inference for full names yet
    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    aeba797 View commit details
    Browse the repository at this point in the history

  2. Started on test workflow

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    06388ec View commit details
    Browse the repository at this point in the history

  3. Renamed NameMap to the more accurate descriptor NameStats

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    93195c0 View commit details
    Browse the repository at this point in the history

  4. First pass at postal code detection

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    0fe071a View commit details
    Browse the repository at this point in the history

  5. Added lat-long extraction with geonames data

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    ca7cc24 View commit details
    Browse the repository at this point in the history

  6. Added RichText implicit for name identifiers, and Name identifier ret… 

    …urns empty map (i.e. null) when not a name
    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    74f8efc View commit details
    Browse the repository at this point in the history

  7. Aded RichText implicit for postal code identifiers

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    52f86f6 View commit details
    Browse the repository at this point in the history

  8. Handle postal code matches in arbitrary locations of the string

    @MWYang
    MWYang committed Oct 9, 2019
    Configuration menu
    Copy the full SHA
    685de4e View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2019

  1. Minor change to output map from postal code identification

    @MWYang
    MWYang committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    45a3eaf View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2019

  1. Added handling of leading zeros in postal codes & refactored a bit

    @MWYang
    MWYang committed Oct 15, 2019
    Configuration menu
    Copy the full SHA
    f6656ad View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2019

  1. Merge branch 'master' into my/detect-sensitive

    @tovbinm
    tovbinm committed Oct 19, 2019
    Configuration menu
    Copy the full SHA
    d06ef91 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2019

  1. Added gender identification for full names

    @MWYang
    MWYang committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    865414e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'my/detect-sensitive' of https://github.com/MWYang/Trans… 

    …mogrifAI into my/detect-sensitive
    @MWYang
    MWYang committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    0eb9715 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2019

  1. Take the grand mean of dictionary checked words per row instead of th… 

    …resholding for name identifying
    @MWYang
    MWYang committed Oct 22, 2019
    Configuration menu
    Copy the full SHA
    e7ca36c View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2019

  1. Started integrating HumanNameIdentifier into SmartTextVectorizer

    @MWYang
    MWYang committed Oct 24, 2019
    Configuration menu
    Copy the full SHA
    c7ef496 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2019

  1. Continued migrating name identification into SmartTextVectorizer; Add… 

    …ed guard check functionality; Added some typing back to name identifcation
    @MWYang
    MWYang committed Oct 26, 2019
    Configuration menu
    Copy the full SHA
    1df72be View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2019

  1. Implemented gender checking in SmartTextVectorizer

    @MWYang
    MWYang committed Oct 30, 2019
    Configuration menu
    Copy the full SHA
    f6c317a View commit details
    Browse the repository at this point in the history

  2. Changed logging and project settings

    @MWYang
    MWYang committed Oct 30, 2019
    Configuration menu
    Copy the full SHA
    88a5a61 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2019

  1. Added logging back into name identification

    @MWYang
    MWYang committed Oct 31, 2019
    Configuration menu
    Copy the full SHA
    d8a3f33 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2019

  1. Moved common Spark functions to utils and fixed bug with empty Row

    @MWYang
    MWYang committed Nov 1, 2019
    Configuration menu
    Copy the full SHA
    f7350c0 View commit details
    Browse the repository at this point in the history

  2. Added type safety to guardCheck, averageCol, and extractDouble functi… 

    …ons (no more DataFrame, yay); moved latter two to separate utils function and added tests
    @MWYang
    MWYang committed Nov 1, 2019
    Configuration menu
    Copy the full SHA
    dd811a6 View commit details
    Browse the repository at this point in the history

  3. Fixed missing pctMale/Female/Other values and added redundant tests

    @MWYang
    MWYang committed Nov 1, 2019
    Configuration menu
    Copy the full SHA
    8240bf9 View commit details
    Browse the repository at this point in the history

  4. REVERT EVENTUALLY: Added logging for a sample of the row to sanity ch… 

    …eck results
    @MWYang
    MWYang committed Nov 1, 2019
    Configuration menu
    Copy the full SHA
    3f282bb View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2019

  1. Cleaned and switched back to using JRC names

    @MWYang
    MWYang committed Nov 5, 2019
    Configuration menu
    Copy the full SHA
    f2b81ae View commit details
    Browse the repository at this point in the history

  2. First pass at converting code to use only one pass over the dataset w… 

    …ith treeAggregate
    @MWYang
    MWYang committed Nov 5, 2019
    Configuration menu
    Copy the full SHA
    025a768 View commit details
    Browse the repository at this point in the history

  3. Finished using treeAggregate for one pass over the data; using broadc… 

    …ast variables for name data; broke off some variables into separate utils file
    @MWYang
    MWYang committed Nov 5, 2019
    Configuration menu
    Copy the full SHA
    db98115 View commit details
    Browse the repository at this point in the history

  4. Set default timeout for countApprox; use exact count via treeAggregat… 

    …e (since we're doing that computation anyway) for SmartTextVectorizer; renamed some parameters
    @MWYang
    MWYang committed Nov 5, 2019
    Configuration menu
    Copy the full SHA
    358f587 View commit details
    Browse the repository at this point in the history

  5. Removed TODOs and added parameter for removing names

    @MWYang
    MWYang committed Nov 5, 2019
    Configuration menu
    Copy the full SHA
    e1a09a8 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2019

  1. Figured out where to put column removal functionality for name columns

    @MWYang
    MWYang committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    b7ccb18 View commit details
    Browse the repository at this point in the history

  2. Merged SmartTextVectorizerWithBias into SmartTextVectorizer; moved Na… 

    …meIdentificationFun trait into utils file
    @MWYang
    MWYang committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    a914354 View commit details
    Browse the repository at this point in the history

  3. Removed unnecessary implicits (replaced with default arguments) and p… 

    …roperly initialize SparkSession (with the help of abstract and lazy vals)
    @MWYang
    MWYang committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    4730b73 View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2019

  1. Added documentation to NameIdentificationUtils and updated dictionaries

    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    580a182 View commit details
    Browse the repository at this point in the history

  2. Finished adding documentation and made small naming changes

    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    006394c View commit details
    Browse the repository at this point in the history

  3. Reverted property file changes

    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    76b14c8 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into my/detect-sensitive

    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    82ca17f View commit details
    Browse the repository at this point in the history

  5. Hopefully fixed master URL configuration error with SparkSession

    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    e8984e2 View commit details
    Browse the repository at this point in the history

  6. Fixed (hopefully) broadcast variable issue by declaring them when dat… 

    …aset.sparkSession is in scope
    @MWYang
    MWYang committed Nov 7, 2019
    Configuration menu
    Copy the full SHA
    582a83a View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2019

  1. Changed SmartTextVectorizer sensitive feature args to an enum from mu… 

    …ltipl bools
    @MWYang
    MWYang committed Nov 11, 2019
    Configuration menu
    Copy the full SHA
    613cf45 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2019

  1. Merge branch 'master' into my/detect-sensitive

    @tovbinm
    tovbinm committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    9ecad2f View commit details
    Browse the repository at this point in the history