Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unary estimator for detecting names and transforming to gender #445

Merged
merged 64 commits into from
Jan 14, 2020
Merged

Unary estimator for detecting names and transforming to gender #445

merged 64 commits into from
Jan 14, 2020

Commits on Dec 5, 2019

  1. Re-added unary estimator code and started porting logic to Algebird m… 

    …onoids instead of custom accumulators
    @MWYang
    MWYang committed Dec 5, 2019
    Configuration menu
    Copy the full SHA
    2acf3fc View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2019

  1. Re-added JRC name dictionary and cleaned up names of methods

    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    b55c31e View commit details
    Browse the repository at this point in the history

  2. Fixed bug with AveragedValue computation; Trying to debug current Alg… 

    …ebird problem
    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    f443952 View commit details
    Browse the repository at this point in the history

  3. Fixed wrong inequality direction in guard checks

    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    557ef39 View commit details
    Browse the repository at this point in the history

  4. Added HLL back to monoid accumulator and code now compiles correctly;… 

    … Still need to fix HLL serialization in Spark issue
    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    b6728ec View commit details
    Browse the repository at this point in the history

  5. Fixed HLL in NameDetectStats not serializing correctly; Now need to f… 

    …ix printing bug for NameDetectStats
    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    ff1b2ef View commit details
    Browse the repository at this point in the history

  6. Fixed NameDetectStats printing

    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    33b77c9 View commit details
    Browse the repository at this point in the history

  7. Fixed guard stat calculation computing moments of number of tokens in… 

    …stead of moments of text length; Still need to fix no moments higher than the 1st being calculated
    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    4caec75 View commit details
    Browse the repository at this point in the history

  8. Fixed moments calculation and fixed divide by zero error when list of… 

    … tokens is empty
    @MWYang
    MWYang committed Dec 6, 2019
    Configuration menu
    Copy the full SHA
    b60dc4a View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2019

  1. Added gender identification code transforming; All previous tests now… 

    … pass
    @MWYang
    MWYang committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    469111b View commit details
    Browse the repository at this point in the history

  2. Undid SparkUtils changes, which are no longer necessary

    @MWYang
    MWYang committed Dec 7, 2019
    Configuration menu
    Copy the full SHA
    e5f169e View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2019

  1. Renamed class names to be more consistent + small fixes

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    b701612 View commit details
    Browse the repository at this point in the history

  2. Added honorific detection

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    8342dae View commit details
    Browse the repository at this point in the history

  3. Implemented RegEx checking for gender

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    2e0e85a View commit details
    Browse the repository at this point in the history

  4. Implemented mixed gender identification strategies

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    b079a27 View commit details
    Browse the repository at this point in the history

  5. Removed TODOs and extraneous functions in preparation for PR

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    a1197a7 View commit details
    Browse the repository at this point in the history

  6. Updated documentation

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    19bad0b View commit details
    Browse the repository at this point in the history

  7. Ignore null values in detecting names

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    3172bde View commit details
    Browse the repository at this point in the history

  8. Added flag for ignoring nulls

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    0d82eef View commit details
    Browse the repository at this point in the history

  9. Added sir/madam to list of honorifics

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    7812d12 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'master' into my/unary-detect-names

    @MWYang
    MWYang committed Dec 9, 2019
    Configuration menu
    Copy the full SHA
    345508f View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2019

  1. Fixed typo when adding sir/madam to list of honorifics that caused fa… 

    …iling test
    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    a80e382 View commit details
    Browse the repository at this point in the history

  2. Fixed failing test due to divide by zero NA on some inputs

    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    f7817d9 View commit details
    Browse the repository at this point in the history

  3. Cleaned up redundant import in tests

    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    cf3eff0 View commit details
    Browse the repository at this point in the history

  4. Made small changes based on PR comments (updated inline comment and i… 

    …mport statement)
    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    b928a49 View commit details
    Browse the repository at this point in the history

  5. Created metadata case class per PR review; Added tests for metadata; … 

    …Cleaned up test code
    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    048c084 View commit details
    Browse the repository at this point in the history

  6. Added test for name threshold

    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    5d30e79 View commit details
    Browse the repository at this point in the history

  7. Updated comment about NameDetectStats.toJson

    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    78c321c View commit details
    Browse the repository at this point in the history

  8. Added tests for new NameStats feature type

    @MWYang
    MWYang committed Dec 10, 2019
    Configuration menu
    Copy the full SHA
    a597d21 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2019

  1. Added private declaration to methods in NameDetectFun trait

    @MWYang
    MWYang committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    997a132 View commit details
    Browse the repository at this point in the history

  2. Abstracted out even more name detection logic into NameDetectUtils

    @MWYang
    MWYang committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    6b8a039 View commit details
    Browse the repository at this point in the history

  3. Added default dictionaries to NameDetectUtils object (for lazy and pe… 

    …rsistent loading)
    @MWYang
    MWYang committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    1250559 View commit details
    Browse the repository at this point in the history

  4. Fixed tests sometimes failing because they were not using the same na… 

    …me dictionary as NameDetectUtils; Removed now unncessary change to RandomText test helper
    @MWYang
    MWYang committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    2d804ee View commit details
    Browse the repository at this point in the history

  5. Updated NameDetectStats.toJson to be less verbose and use custom seri… 

    …alizers
    @MWYang
    MWYang committed Dec 11, 2019
    Configuration menu
    Copy the full SHA
    5336e71 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2019

  1. Added shortcut for unary name detector

    @MWYang
    MWYang committed Dec 12, 2019
    Configuration menu
    Copy the full SHA
    464fe52 View commit details
    Browse the repository at this point in the history

  2. Delete accidentally committed temporary test file

    @MWYang
    MWYang committed Dec 12, 2019
    Configuration menu
    Copy the full SHA
    bff4d1b View commit details
    Browse the repository at this point in the history

  3. Removed type parameter from NameDetectFun because of later conflict w… 

    …ith SmartTextMapVectorizer
    @MWYang
    MWYang committed Dec 12, 2019
    Configuration menu
    Copy the full SHA
    639af3d View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2019

  1. Removed Pythonic i.e. not Scala-ic index thing and added separate cas… 

    …e for checkign the last token instead, per PR comment
    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    440068c View commit details
    Browse the repository at this point in the history

  2. Removed extraneous case classes for dictionaries, per PR comment

    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    e6136f9 View commit details
    Browse the repository at this point in the history

  3. Small fixes (updated comments, re-ordered things) per PR comments

    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    283d76e View commit details
    Browse the repository at this point in the history

  4. Removed usage of broadcast variables in transformer b/c it does not s… 

    …erialize correctly
    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    80d81ed View commit details
    Browse the repository at this point in the history

  5. Fixed serialization of GenderDetectStrategy, per PR recommendation to… 

    … test with local workflow
    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    e003eb2 View commit details
    Browse the repository at this point in the history

  6. Fixed missing plus sign in OpPipelineStageReaderWriter causing double… 

    … serialization test to fail
    @MWYang
    MWYang committed Dec 13, 2019
    Configuration menu
    Copy the full SHA
    b51e14b View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2019

  1. Synced changes from upstream feature branch for STV changes

    @MWYang
    MWYang committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    393275c View commit details
    Browse the repository at this point in the history

  2. Tidied up monoid definition for NameDetectStats after figuring out ho… 

    …w to use Algebird
    @MWYang
    MWYang committed Dec 14, 2019
    Configuration menu
    Copy the full SHA
    ad9574c View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2019

  1. Abstracted out ordering of gender detection strategies into utils file

    @MWYang
    MWYang committed Dec 18, 2019
    Configuration menu
    Copy the full SHA
    5d7716b View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2019

  1. Merge branch 'master' into my/unary-detect-names

    @MWYang
    MWYang committed Dec 20, 2019
    Configuration menu
    Copy the full SHA
    eaad6f8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'my/unary-detect-names' of https://github.com/MWYang/Tra… 

    …nsmogrifAI into my/unary-detect-names
    @MWYang
    MWYang committed Dec 20, 2019
    Configuration menu
    Copy the full SHA
    2bea311 View commit details
    Browse the repository at this point in the history

  3. Small fixes (better Scala code, more safe, better patterns) from Matt… 

    …hew's PR comments
    @MWYang
    MWYang committed Dec 20, 2019
    Configuration menu
    Copy the full SHA
    77f91a2 View commit details
    Browse the repository at this point in the history

  4. Improved gender detection strategy tests to check that the correct st… 

    …rategy was used
    @MWYang
    MWYang committed Dec 20, 2019
    Configuration menu
    Copy the full SHA
    b6b385a View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2019

  1. Broke out guard check numbers into their own params

    @MWYang
    MWYang committed Dec 21, 2019
    Configuration menu
    Copy the full SHA
    8ed02bd View commit details
    Browse the repository at this point in the history

Commits on Jan 6, 2020

  1. Added operationName as an argument to HumanNameDetectorModel for easi… 

    …er debugging
    @MWYang
    MWYang committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    799eb58 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into my/unary-detect-names

    @MWYang
    MWYang committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    ca69ed8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'my/unary-detect-names' of https://github.com/MWYang/Tra… 

    …nsmogrifAI into my/unary-detect-names
    @MWYang
    MWYang committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    b82440a View commit details
    Browse the repository at this point in the history

  4. Revert to using container Text class for NameDetectUtils per PR comments

    @MWYang
    MWYang committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    d747c92 View commit details
    Browse the repository at this point in the history

  5. Added NameStats to FeatureBuilder

    @MWYang
    MWYang committed Jan 6, 2020
    Configuration menu
    Copy the full SHA
    7d01d70 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2020

  1. Added NameStats to a few more places

    @MWYang
    MWYang committed Jan 7, 2020
    Configuration menu
    Copy the full SHA
    b4209c6 View commit details
    Browse the repository at this point in the history

  2. Added NameStats to TestFeatureBuilder and RandomMap

    @MWYang
    MWYang committed Jan 7, 2020
    Configuration menu
    Copy the full SHA
    23d7a57 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2020

  1. Merge branch 'master' into my/unary-detect-names

    @tovbinm
    tovbinm committed Jan 8, 2020
    Configuration menu
    Copy the full SHA
    b9522de View commit details
    Browse the repository at this point in the history

  2. Incorporated PR comments (using enumeratum for NameStats map keys/val… 

    …ues and proper camel casing for guard check params)
    @MWYang
    MWYang committed Jan 8, 2020
    Configuration menu
    Copy the full SHA
    e4e3ddd View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2020

  1. Incorporated PR comments (renamed GenderStrings to GenderValues and r… 

    …emoved BooleanStrings enum
    @MWYang
    MWYang committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    8b02dff View commit details
    Browse the repository at this point in the history

  2. Removed plural names from NameStats enums and factored out method in … 

    …identifyGender to hopefully reduce complexity
    @MWYang
    MWYang committed Jan 9, 2020
    Configuration menu
    Copy the full SHA
    8844ef5 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2020

  1. More small fixes from PR comments

    @MWYang
    MWYang committed Jan 10, 2020
    Configuration menu
    Copy the full SHA
    0cc10e0 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2020

  1. Removed emptiness check

    @MWYang
    MWYang committed Jan 14, 2020
    Configuration menu
    Copy the full SHA
    4fea007 View commit details
    Browse the repository at this point in the history