Named Entity Recognition + Valence Tests

[agshruti12]

Pull Request Template:
If you are merging in a feature or other major change, use this template to check your pull request!

Basic Info

What's this pull request about?

Tests for NER and valence. NER tests are a modified unit test, checking correct NUMBER of named entities, along with correct named entities themselves. Has it's own test dataset called 'test_named_entity.csv'. Performed valence testing as well using perturbation framework. For a given positive/negative sentence, have generated a corresponding INV and DIR perturbed sentence. Test checks to see if the magnitude of the DIR change is greater than the magnitude of the INV change. Has it's own test dataset called test_chat_level_complex.csv. We can input more tests that operated with the perturbation framework at the chat level in this csv.

Feature Documentation

Did you document your feature? Make sure you do the following before you pull request!

• Copy the Template. Go to the Feature Wiki and Copy/Paste the Feature Template into a new page.
• Fill out the Template. Fill out the basic information for the feature in the template. Use the template to document your plan for implementation and major design decisions; if anything changes along the way, update the documentation as you go.
• At the top of my feature, I indicate whether the feature is conversation level or chat level.

Code Basics

• My feature is a .py file.
• My feature uses snake case in the name. That means the name of the format is my_feature, NOT myFeature (camel case).
• My feature has the name, NAME_features.py, where NAME is the name of my feature.
• My feature is located in feature_engine/features.

Testing

• I have thought about test cases for my features, with inputs and expected outputs.
• I have linked to a location (e.g., .py or .ipynb) where I can run my test cases and show they work (inputs match expected outputs).

The location of my tests are here:

[PASTE LINK HERE]

If you check all the boxes above, then you ready to merge!

[xehu]

I notice that not all the NER tests pass --- but this is because the feature isn't perfect! Would it be possible to run the feature on the full test dataset in order to get metrics (e.g., precision/recall), but then only run the test on a subset of the NER features that we know are supposed to work? That way, we won't have all the tests return as 'failing' ...

[agshruti12]

Changes made:

1. assign_chunk_nums: removed temporal parameters
2. get_all_DD_features: removed temporal parameters from assign_chunk_nums call
3. burstiness: use wait_times starting at positions 1 (omitted first value), and ensuring time_diff parsing works for both timedelta (when passing in a datetime timstamp) and float (when passing in a unixtime or integer timestamp) objects.
4. fflow: cosine similarity --> cosine distance
5. tests for valence + NER + discursive diversity + variance in DD + incongruent modulation + within person discursive range + forward flow + team burstiness (across 3 different timestamps)