v1.1
New Squad Score Version (#36)
New Squad Score version that still complies with restrictions of utilizing only features that are either representative of validated complexity models or requested by stakeholder and in a form that is least susceptible to errors we will encounter in stories and transcriptions (i.e. using characters for length rather than syllables or words).
- New changes
- Added feature of
adj_num
- Formula now requires an external library (
nltk
)
- Added feature of
- Reasoning
- Per stakeholder, one qualitative indication of skilled creative writing is descriptive language, specifically sensory descriptions. While this specific feature implementation is limited, in that the mere presence of an adjective does not inherently indicate sensory descriptions, this method was selected as a way to access the spirit of the creative writing feature while still respecting the limitations of this use case. "In most cases, NLTK correctly tags words that have typos" (p. 52) so it is a good candidate for use in our environment which likely will contain misspellings and mis-transcriptions.
- Dependencies
nltk
librarynltk
downloads:punkt
andaveraged_perceptron_tagger
, both included in DockerfileRUN
commands
- Features
- sl: story length (in characters)
- awl: average word length (in characters)
- qn: quotes number
- uw: unique words count (over two characters)
- an: number of adjectives
- Weights (no change from v1.0)
- Squad Score is initiated with only weights of 1 for each feature, as there were not enough labels on the data to be able to tune weights in a generalizable way.
- There is also a standardized “range scaler” of 30, meant to bring the overall Squad Score up to a closer range of 0-100, purely for ease of metric reading.
- Formula
- sl(1)(30) + awl(1)(30) + qn(1)(30) + uw(1)(30) + an(1)(30)
- Range (no change from v1.0)
- the score bottoms out at 0, but does not have a bounded upper range
- Metrics
- Similar to v1.0, the only labels available at the time of this development were a 1-25 ranking of 25 of the training set stories. Applying this Squad Score formula to these 25 stories resulted in a -.63 correlation coefficient of scores to rankings, which is an improvement of .03 over v1.0. The generalizability of this improvement is unconfirmed.