Transcript offsets

[maxhawkins]

Calculate transcript offsets for all tokens and use those to reconstruct the text in-browser. This begins to address #11.

[strob]

Thanks for starting to work on this! This definitely seems like the path forward for addressing hyphenation and German compound words. (Are there other cases where this will make a substantive difference?)
Does this already fix hyphenation alignment? If not, I'd rather wait to merge until it adds that functionality.

[maxhawkins]

Yeah good point. This fixed the hyphenation stuff. It also preserves punctuation so you can make awesome ascii art in your transcripts.

★░░░░░░░░░░░████░░░░░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░███░██░░░░░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░██░░░█░░░░░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░██░░░██░░░░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░░██░░░███░░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░░░██░░░░██░░░░░░░░░░░░░░░░★ 
★░░░░░░░░░░░██░░░░░███░░░░░░░░░░░░░░★ 
★░░░░░░░░░░░░██░░░░░░██░░░░░░░░░░░░░★ 
★░░░░░░░███████░░░░░░░██░░░░░░░░░░░░★ 
★░░░░█████░░░░░░░░░░░░░░███░██░░░░░░★ 
★░░░██░░░░░████░░░░░░░░░░██████░░░░░★ 
★░░░██░░████░░███░░░░░░░░░░░░░██░░░░★ 
★░░░██░░░░░░░░███░░░░░░░░░░░░░██░░░░★ 
★░░░░██████████░███░░░░░░░░░░░██░░░░★ 
★░░░░██░░░░░░░░████░░░░░░░░░░░██░░░░★ 
★░░░░███████████░░██░░░░░░░░░░██░░░░★ 
★░░░░░░██░░░░░░░████░░░░░██████░░░░░★ 
★░░░░░░██████████░██░░░░███░██░░░░░░★ 
★░░░░░░░░░██░░░░░████░███░░░░░░░░░░░★ 
★░░░░░░░░░█████████████░░░░░░░░░░░░░★ 
★░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░★

[strob]

Getting there!

Unfortunately, the click/highlight behavior on view_alignment.html is behaving very poorly.

Here's a screenshot of the sort of thing I'm seeing (throughout a transcript)--let me know if you're able to reproduce or if you need a more elaborate bug report:

[maxhawkins]

That's a bug with the offset calculation. Can you send me the transcript text?

[strob]

Here's the transcript text and the alignment JSON:

wasteland.txt
wastleland-align.json.txt

[maxhawkins]

Ug. It's Python's utf-8 handling. Working on it...

[maxhawkins]

OK, that should fix it. Please give this a try.

This video on Unicode in Python in required watching. So much pain:
http://nedbatchelder.com/text/unipain.html

[maxhawkins]

In case you tried this in the last few minutes, please pull again. There was a bug, but the latest version should work for your file.

[strob]

Fabulous.