Tokenizer & Word Frequency Counter

CS 121 — Assignment 1

Two small command-line programs in Python for text analysis:

• Part A — reads a text file, breaks it into tokens, and prints every unique token with the number of times it appears, ordered from most to least frequent.
• Part B — reads two text files and prints how many unique tokens they have in common.

Both are written with the standard library only. No dependencies, no installation.

---

What counts as a token

A token is a maximal sequence of alphanumeric ASCII characters, lowercased. Anything else — punctuation, whitespace, symbols — ends the current token and is discarded.

"Hello, world! It's 2024."   →   ["hello", "world", "it", "s", "2024"]

Files are read as UTF-8 with errors='ignore', so non-English characters and malformed bytes are skipped rather than crashing the program.

---

Usage

Part A — word frequencies

python PartA.py <file>

$ python PartA.py sample.txt
the = 42
and = 31
of = 27
...

Output is one token = count pair per line, sorted by count in descending order.

Part B — common unique tokens

python PartB.py <file1> <file2>

$ python PartB.py sample1.txt sample2.txt
137

Prints a single integer: the number of distinct tokens appearing in both files. Part B imports and reuses Part A's tokenize and computeWordFrequencies, so both files must sit in the same directory.

---

Error handling

Situation	Behavior
No file argument	Prints a prompt asking for a file
Too many arguments	Prints an error and exits
File does not exist	Exits with Cannot read file! (File not found)
Any other read error	Exits with Error reading file

---

Complexity

Let n, m = characters in each file, t = tokens produced, w = unique tokens.

Part A

Function	Runtime	Why
tokenize	O(n)	Each character in the file is visited exactly once.
computeWordFrequencies	O(t)	One pass over the token list; dict insert and lookup are O(1) on average.
printWords	O(w log w)	Sorting the unique tokens dominates the linear print pass.

Overall: O(n + w log w).

Part B

Function	Runtime	Why
checkCommonWords	O(w₁)	One pass over the first map, with an O(1) average-case membership check against the second.

Overall: O(n + m) to read and tokenize both files, then O(w₁) to intersect — linear in the total input size.

---

Files

PartA.py   tokenize() · computeWordFrequencies() · printWords()
PartB.py   checkCommonWords()  — imports PartA