Skip to content

v1.1
Compare
Choose a tag to compare
@lorischl-otter lorischl-otter released this 15 Oct 19:43
· 78 commits to main since this release
v1.1
bdebc18
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

New Squad Score Version (#36)

New Squad Score version that still complies with restrictions of utilizing only features that are either representative of validated complexity models or requested by stakeholder and in a form that is least susceptible to errors we will encounter in stories and transcriptions (i.e. using characters for length rather than syllables or words).

  • New changes
    • Added feature of adj_num
    • Formula now requires an external library (nltk)
  • Reasoning
    • Per stakeholder, one qualitative indication of skilled creative writing is descriptive language, specifically sensory descriptions. While this specific feature implementation is limited, in that the mere presence of an adjective does not inherently indicate sensory descriptions, this method was selected as a way to access the spirit of the creative writing feature while still respecting the limitations of this use case. "In most cases, NLTK correctly tags words that have typos" (p. 52) so it is a good candidate for use in our environment which likely will contain misspellings and mis-transcriptions.
  • Dependencies
    • nltk library
    • nltk downloads: punkt and averaged_perceptron_tagger, both included in Dockerfile RUN commands
  • Features
    • sl: story length (in characters)
    • awl: average word length (in characters)
    • qn: quotes number
    • uw: unique words count (over two characters)
    • an: number of adjectives
  • Weights (no change from v1.0)
    • Squad Score is initiated with only weights of 1 for each feature, as there were not enough labels on the data to be able to tune weights in a generalizable way.
    • There is also a standardized “range scaler” of 30, meant to bring the overall Squad Score up to a closer range of 0-100, purely for ease of metric reading.
  • Formula
    • sl(1)(30) + awl(1)(30) + qn(1)(30) + uw(1)(30) + an(1)(30)
  • Range (no change from v1.0)
    • the score bottoms out at 0, but does not have a bounded upper range
  • Metrics
    • Similar to v1.0, the only labels available at the time of this development were a 1-25 ranking of 25 of the training set stories. Applying this Squad Score formula to these 25 stories resulted in a -.63 correlation coefficient of scores to rankings, which is an improvement of .03 over v1.0. The generalizability of this improvement is unconfirmed.
Assets 2