What pitfalls are there to watch for when parsing pokemon_species_flavor_text.csv?

[coreyog]

So far I've found form feeds in Bulbasaur's English Pokemon Red dex entry and Stantler's English Pokemon Gold entry has an interesting control sequence I hadn't seen of 0xC2AD right where it splits the word "reality." I don't know any of the other language that exist in the data so I don't know if I should be blindly replacing these characters with spaces or just removing them. Is there a rule of thumb for how to parse these characters? Are there other character sequences I should be parsing out that I just haven't found because there's 802 pokemon in 2 dozen games, in a dozen languages?

[magical]

It's not very well documented. Here's what's going on. Here's what stantler's flavor text looked like in Gold:

    The curved antlers
    subtly change the
    flow of air to

    create a strange
    space where real-
    ity is distorted.

The early games stored flavor text with explicit line breaks, hyphenation, and page breaks. We represent this is the database as special characters in the text.

• Newlines are newlines, easy
• Page breaks are represented with form feed characters
• Hard hyphens are represented with U+00AD SOFT HYPHEN (Note that \xc2\xad is the UTF-8 encoding of U+00AD)

You can and should feel free to replace these characters with normal spaces or whatever. Here's what veekun does:

https://github.com/veekun/spline-pokedex/blob/3554f47d722a8e36002c51796be5d762bf36fd77/splinext/pokedex/helpers.py#L100-L110

        # Page breaks are treated just like newlines.
        # Soft hyphens followed by newlines vanish.
        # Letter-hyphen-newline becomes letter-hyphen, to preserve real
        # hyphenation.
        # Any other newline becomes a space.
        html = flavor_text.replace(u'\f',       u'\n') \
                          .replace(u'\u00ad\n', u'') \
                          .replace(u'\u00ad',   u'') \
                          .replace(u' -\n',     u' - ') \
                          .replace(u'-\n',      u'-') \
                          .replace(u'\n',       u' ')

[magical]

Closing. Feel free to comment if you still have questions, or join us on IRC: https://veekun.com/chat.