Filtering words using a dictionnary #84
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nicolasdugue
approved these changes
May 21, 2024
nicolasdugue
added a commit
that referenced
this pull request
Jun 14, 2024
* factory method to load embeddings at the w2v format (#53) * Revert "factory method to load embeddings at the w2v format (#53)" This reverts commit 7da0e99. * 52 loading vectors such as w2v or spine ones (#57) * factory method to load embeddings at the w2v format * Update graph_embeddings.py, small fix * Adding distRatio (#59) * moving dist ratio in sinr.text.evaluate, adding unit tests (#61) * commenting cosine_dist, pick_intruder, dist_ratio, dist_ratio_dim, intra_sim, inter_sim methods * moving distRatio from graph_embeddings to text/evaluate * tests unitaires distratio * cleaning comments * adding creation and deletion of w2v file for distRatio unit tests * fix with_value() argument (#65) Co-authored-by: Simon Guillot <simon.guillot@univ-lemans.fr> * load_from_word2vec model s name bug fixed (#70) * missing word list bug fixed (#68) * 73 wrong community memberships update when filtering dimensions (#75) * update of community_membership when filtering dimensions * sinr filtered: removing dimensions and updating communities_sets * fixed code to pass tests * comments * 76 preprocessing multiple documents (#77) * preprocessing by documents * Tests : preprocessing by sentences and by documents * adding size indicator for spacy model and downloading spacy model in tests workflow' * downloading spacy * 78 classification (#80) * preprocess : minimal length of documents kept + tests * vectorizer + test * classification's methods + tests * xgboost interpretable dimensions * adding xgboost for test workflow * classification, fit and score test modification * get_dimension_stereotypes on removed community fixed (#82) * Filtering words using a dictionnary (#84) * Exceptions list, path to save / load SINrVectors (#86) * not removing words when in exceptions list * add path to method save * exceptions list to set + test exceptions list * path parameter method load * new exceptions list for similarity * optionnal parameter path for load and save methods * 90 notebooks (#91) * fix save, load, dim_nnz_thresholds + add obj_nnz_count * add notebook with gutenberg example * bnc model for notebook * notebook bnc * notebook frwac * remove nb evaluate * add tqdm to sparsify method --------- Co-authored-by: Thibault PROUTEAU <thibault.prouteau@gmail.com> Co-authored-by: Anna B <72624798+aberanger@users.noreply.github.com> Co-authored-by: Simon Guillot <47661058+SimonGuillot@users.noreply.github.com> Co-authored-by: Simon Guillot <simon.guillot@univ-lemans.fr>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When preprocessing with the extract_text method, an optional parameter, a vocabulary list, can be entered to remove all words not belonging to this list from the corpus.
Fixes #83
Type of change
Please delete options that are not relevant.