Coding Open Answers

Coding Open Answers

An open question ("Why did you give that score?") gives you text. To count it, each answer needs a theme. siamang.data.text_coding does that from a codeframe — a JSON file of themes, the answers you coded by hand, and rules that code the rest — with no model and no network, so the same file gives the same numbers at every run. This page shows how to write one, check it and use it.

---

How an answer is coded

Each answer is coded by the first of:

1. Your decision. The codeframe keeps the themes you gave an answer, by its fingerprint — sixteen hex characters of a hash of the answer's text, case, spacing and Unicode form set aside. It never keeps the text itself. You can decide an answer has one theme, several, or none ([]: read, and belongs to no theme).
2. The rules. Each theme can have terms that code the answers nobody decided — including the ones collected after you wrote the rules, because the rules run at every run.
3. Nothing. The answer is uncoded: left for you to read.

A blank answer — an empty cell, or a missing value however the column holds it (None, NaN, pd.NA, NaT) — is not an answer and is never coded.

---

A first codeframe

import pandas as pd
from siamang.core import Variable, VariableMap
from siamang.data import SurveyData, text_coding

answers = [
    "The delivery was late",
    "Box arrived damaged, and two days late",
    "The driver was rude",
    "Nothing",
    "wasn't late, all fine",
    "Great!",
    "",
]
variables = VariableMap()
variables.add(Variable("why", "nominal", label="Why that score?", dtype="str"))
data = SurveyData(frame=pd.DataFrame({"why": answers}), variables=variables)

codeframe = {
    "schema_version": "2.0",
    "variable": "why",
    "multiple": True,  # several themes an answer
    "themes": [
        {"code": 1, "label": "Late delivery", "group": "Delivery",
         "rules": {"include": ["late", "delay*"]}},
        {"code": 2, "label": "Damaged", "group": "Delivery",
         "rules": {"include": ["damag*", "broken|broke"]}},
        {"code": 3, "label": "Rude staff",
         "rules": {"include": ["rude", "unfriendly"], "require": ["staff", "driver", "courier"]}},
        {"code": 9, "label": "Nothing / Don't know", "exclusive": True,
         "rules": {"include": ["nothing", "don't know"]}},
    ],
    # "Great!" read by a coder: no theme.
    "assignments": {text_coding.fingerprint("Great!"): []},
}

print(text_coding.validate(codeframe).ok)   # True
cf = text_coding.parse(codeframe)
coded = text_coding.apply(data, cf)
print(coded.frame["why_theme"].tolist())
# [[1], [1, 2], [3], [9], None, [], None]
print(coded.report.themes(cf).to_markdown())

| Theme | N | % |
|---|---|---|
| Delivery (net) | 2 | 33.3 |
| Late delivery | 2 | 33.3 |
| Damaged | 1 | 16.7 |
| Nothing / Don't know | 1 | 16.7 |
| Rude staff | 1 | 16.7 |
| No theme | 1 | 16.7 |
| Coded | 5 | 83.3 |
| Coded by hand | 1 | 16.7 |
| Coded by rules | 4 | 66.7 |
| Uncoded | 1 | 16.7 |

Variable = why; Answered = 6; Themes = 4; Coverage = 83.3 % of the answers are coded; Coded by hand = 1; Coded by rules = 4; Distinct uncoded answers = 1; Percentages = of the respondents who answered; a respondent can have several themes, so the themes add up to more than 100 %; Nets = a net counts a respondent once, however many of its themes they have

Six people answered. wasn't late, all fine is uncoded: its only mention of late is negated. The net Delivery counts the respondent with both of its themes once. With "multiple": true the theme variable holds a list of codes per answer — a multiple-choice variable, which frequencies, crosstabs, the Bar chart and the tab book read as they read a multiple-choice question.

Save the codeframe as analysis/why.codeframe.json and text_coding.load(path) reads it back.

---

The file

Field	What it is
variable	The open-text variable.
into	The theme variable (default <variable>_theme).
language	"en". The words the rules know — negations, clause words, stop words — are English.
multiple	true: several themes an answer (a multiple-choice variable). false (default): one theme (a nominal variable).
max_codes	The most themes the rules give an answer; 0 (default) for no cap.
scope	Where the rules read: "clause" (default) or "answer". A theme's own rules.scope wins.
replace	[{"from": "delievery", "to": "delivery"}]: whole words or phrases replaced before the rules read — typos, synonyms. They are made in one pass: a word one replacement writes is not replaced again.
themes	code, label, and optionally definition, group (the net it belongs to), exclusive, priority and rules.
assignments	Your decisions: {fingerprint: code, [codes] or []}.

A theme's rules are include, require and exclude terms. The theme matches in a clause (or, with scope answer, in the whole answer) when an include term matches there, the require terms have a match there — a list is "any of these"; a list of lists is "one of each list" — and no exclude term matches there.

---

Writing terms

Term	Matches	Does not match
late	It was LATE!	latest, wasn't late
delay*	delay, delays, delayed	relay
slow|late	slow or late
customer service	the customer service was bad	service to the customer
staff ~3 rude	the staff were very rude, rude staff	rude. The staff, rude, the staff
not_late	wasn't late, never late	it was late
not_friendly staff	no friendly staff	friendly staff
don't know	I don't know, I dont know, I do not know	I don't really know (write don't ~2 know)
would not recommend	would not recommend, wouldn't recommend
not happy	not happy, wasn't happy	never happy

• * stands for any letters of a word, anywhere in it (*charg*).
• | gives alternatives at one place of a phrase: customer serv*|support.
• ~N finds two words (or phrases) within N words of each other, in either order, between two punctuation marks — a comma or a line break ends its reach as a full stop does.
• A term matches a mention that is not negated. not_word matches only a negated one. A negation written in the term (not late, no problems, don't know) is the term's own, and so is the one a not_ word asks for: in no friendly staff, not_friendly staff finds the staff under it.
• A term's words are split where an answer's are: e-mail and n/a are two words each (a -, / or · between two letters), so they find E-mail and N/A, and quotation marks around a term are dropped ("late" is late). One of several alternatives cannot be two words: e-mail|email warns, and e-mail and email are two terms.
• There are no regular expressions: re: is refused.
• A "word" longer than 200 characters — a pasted link or string — matches no term.

One n't form, however it is typed. dont is read as don't, cannot and can not as can't, and do not, was not, would not… as don't, wasn't, wouldn't (after do, does, did, is, are, was, were, has, have, had, can, could, will, would, shall, should, must, need, might) — in the answers and in your terms alike. So don't know and do not know are the same term, and each finds all three spellings. The word not in a term stands for every negation written with not: not happy finds wasn't happy. explain shows the words as they are read.

Negation. not, no, never, cannot, without, nothing, none, nobody, neither, nor, hardly, barely and every n't form negate the next three words, stopping at the end of a clause and at and, or, yet. So in not cheap, and fast only cheap is negated.

Clauses end at a line break, at punctuation (. , ; : ! ? ( ) [ ] { }, the dashes – —, …, a bullet •, |), at -, / and · when they do not join two letters (fast - cheap, late / broken, a list's - item; but e-mail and n/a stay whole), and at but, however, although, though, whereas, except, plus. In Delivery was quick but the box was damaged, a theme that requires delivery and includes damaged does not match in clause scope — they are in different clauses — and does with "scope": "answer". An answer written on several lines is read line by line: Not happy on one line does not negate Price too high on the next. (Its fingerprint, and so your decision for it, is that of the same words on one line.)

What negation costs. A mention under a negation does not match, so never received my parcel is not coded Parcel, nor not enough staff Staff. preview counts, per theme, the respondents it loses this way (negated) and lists them per answer, so you can read them: a not_ term or your own decision codes the ones that belong.

Other languages. Words are Unicode: Цена, café, नमस्ते are words, and terms in them match. Only negation is English — an answer in another language is never lost, it just has no negation read in it.

---

Several themes, exclusive themes, priority

When several themes match an answer:

1. an exclusive theme (Nothing / Don't know) is kept only if no other theme matched;
2. the themes are ranked by priority (highest first; ties by their order in the file);
3. max_codes keeps that many; a codeframe with "multiple": false keeps one.

Your own decisions are kept as you made them (a single-theme codeframe keeps the highest-ranked of several).

---

Checking your rules

validate lists every problem with its place in the file:

found = text_coding.validate({**codeframe, "themes": codeframe["themes"] + [
    {"code": 4, "label": "Mail", "rules": {"include": ["e-mail|email", "re:mail"]}}]})
for issue in found.errors + found.warnings:
    print(issue.level, issue.path, issue.message)

error ('themes', 4, 'rules', 'include', 1) theme 4 (Mail): include term 're:mail': regular expressions are not supported: write the words, with * for word forms (delay*), | for alternatives (slow|late) and ~N for words near each other (staff ~3 rude)
warning ('themes', 4, 'rules', 'include', 0) theme 4 (Mail): include term 'e-mail|email' has a part that can never match, 'e-mail': it is two words (an answer is split at '-'), which one of several alternatives cannot be — make it a term of its own

explain shows how one answer is read and why it got its themes:

e = text_coding.explain("The delivery wasn't late, but the box was broken", cf)
e["clauses"]   # ['the delivery wasn't late', 'the box was broken']
e["codes"]     # [2]
[(r["label"], r["status"], r["term"]) for r in e["rules"]]
# [('Late delivery', 'negation', 'late'), ('Damaged', 'fired', 'broken|broke')]

Each rule is fired, vetoed (a require missing or an exclude present, with the reason) or negation (the words are there, negated). dropped lists the themes that matched and were set aside — an exclusive theme, a cut by priority — and why. tokens are the words as the rules read them (do not as don't), each with the word that negates it.

preview codes a whole list of answers with their counts and gives, per distinct answer, its codes, whether they came from you or from a rule (with the term and the words it matched), and the counts per theme, per net and overall:

p = text_coding.preview({"The delivery was late": 12, "Great!": 3, "meh": 1}, cf)
p["coverage"]
# {'answered': 16, 'coded': 15, 'uncoded': 1, 'by_hand': 3, 'by_rules': 12,
#  'no_theme': 3, 'percent_coded': 93.8}

Each theme also has negated: the respondents who mention one of its include terms only negated and did not get it (see What negation costs).

It is fast enough to run as you edit: 50,000 different answers against 30 themes of 10 terms each take a few seconds, and what an answer costs grows with its length and no faster — a long pasted answer, or one word of a hundred thousand letters, takes no longer than its reading.

---

Finding new themes

suggest lists the words and two-word phrases that come up most in the answers still uncoded, with how many people used each and an example:

text_coding.suggest({"The app crashes": 3, "app crashes all the time": 2}, 5, codeframe=cf)
# {'words': [{'term': 'app', 'count': 5, 'example': 'The app crashes'}, ...],
#  'phrases': [{'term': 'app crashes', 'count': 5, 'example': 'The app crashes'}]}

English stop words are left out, and a negated mention is counted apart, as not_word — a term you can paste as it is — so a word can come both ways (fast and not_fast).

---

In a flow

The Code open answers node (prepare.text_code) applies a codeframe file and hands on the data with the theme variable, the theme table and its statistics (coverage, coded by hand and by rules). A codeframe that gives several themes an answer makes a multiple-choice variable, so a later chart or tab book treats it as one. See Reporting Tables for the table.

The theme variable holds the themes: an answer you decided has no theme and an uncoded one have none there. So a frequency or crosstab of it counts the respondents with a theme, while the theme table counts everyone who answered — its No theme and Uncoded rows are the difference. Quote the table for the share of respondents who said something.

When Theme variable is left empty, the variable takes the name the codeframe carries (into). Checking the flow, running it and writing its script know that name only when they are given the codeframe (check_flow, FlowRunner and generate_flow take codeframes=; siamang flow check, flow run and codegen read it where the flow runs) — writing the name in Theme variable makes a flow that never depends on it.

---

Version 1 codeframes

A codeframe with "schema_version": "1.0" — one theme per decided answer, no rules — is read and applied exactly as before: the same variable, the same table (theme shares of the coded answers) and the same generated code.

---

See also

• Working with Data
• Reporting Tables
• Analysis