This is a simple hack of spaCy streamlit. It rewrites some of the dependency visualizer code to display a subset of the dependencies, rather than the full dependency tree. Also included is some code for calculating measures of association strength, see Evert (2016) for details.
spaCy uses ClearNLP Dependency Labels. matcherPatterns.pickle in this repository leverages spaCy's dependency matcher to filter the ClearNLP dependencies into the POS pairing notation widely used in Applied Linguistics (NOUN + VERB, for example). The schema is designed to align with the ACL (Ackermann and Chen, 2013), but also includes some additional combinations. The additional collocations can be easily filterd out later, or the schema can be customized.
| label | headPOS | depPOS |
|---|---|---|
| NOUN + ADJ | NOUN | ADJ |
| NOUN + NOUN | NOUN | NOUN |
| VERB + NOUN | VERB | NOUN |
| VERB + ADJ | VERB | ADJ |
| VERB + ADV | VERB | ADV |
| ADJ + ADV | ADJ | ADV |
| NON-ACL below this point | ||
| NOUN + DET | NOUN | DET |
| NOUN + PREP | NOUN | ADP |
| PREP + NOUN | ADP | NOUN |
| VERB + PART | VERB | PART |
| VERB + PREP | VERB | ADP |
| VERB + VERB | VERB | VERB |
| VERB + AUX | VERB | AUX |
To investigate word patterns and lexical development in the ICNALE corpus. These are argumentative essays written by English language learners.
Currently, the software uses OANCBigramStats.pickle, a list of filtered dependency bigrams with co-occurrence frequencies pre-calculated. This bigram list is based on the Open American National Corpus. It seems to perform fairly well, but I will be making efforts to replace this corpus or add to it with more News and Gold-standard Student Papers -- published academic work and TV show subtitles are other corpus samples that might be helpful. However, the target corpus is exclusively argumentative essays, so I suspect a news corpus (particularly op-eds and opinion pieces) may be the most suitable for the task.
I also made a pullable docker image docker pull langdonholmes/collocationextractor. It is quite large (5.1GB), as it includes Python, PyTorch, several pre-trained NLP models, and all the relevant OANC dependency bigram statistics. I could get this down to probably 1-2 GB if I used only spaCy's 'small' pre-trained model, but the transformer-based model is significantly more accurate for distant dependencies, which are exactly the dependencies not captured by more traditional window/span collocation extraction methods, so there wouldn't be much point to all this with just the 'small' pre-trained model.
This package contains utilities for visualizing spaCy models and building interactive spaCy-powered apps with Streamlit.