Change API to use Text rather than String for readers/writers

[jgm]

This can be done without changing the representation of strings in the Pandoc data structure to Text.
(That should be done too eventually, but this change is independent of that.)

Note that in the readers, we can convert to String and use existing parsers; once the API change is made, we can change the internals to use Text at our leisure.

In the writers, too, we can just do a conversion at first, and change internals later.

But while we're making big changes in the pandoc API, it makes sense to make this one.

[jgm]

Finished for the readers as of commit 86282d6

Still TODO: writers.

[jgm]

Note that this change by itself doesn't help with memory usage:

new (on MANUAL.txt):

<<ghc: 1803307032 bytes, 3470 GCs, 6223440/21780264 avg/max bytes residency (22 samples), 61M in use, 0.000 INIT (0.003 elapsed), 0.655 MUT (0.723 elapsed), 0.450 GC (0.584 elapsed) :ghc>>

old:

<<ghc: 1777928296 bytes, 3424 GCs, 6598984/20726176 avg/max bytes residency (22 samples), 60M in use, 0.000 INIT (0.002 elapsed), 0.657 MUT (0.722 elapsed), 0.469 GC (0.608 elapsed) :ghc>>

This is no doubt because most of the memory use is from the parsers, and we're just converting the Text to a String and parsing as before. The improvement will come after we rewrite the parsers to use Text internally.

[tarleb]

Heretical question: Would it make sense to use the String type from foundation instead of Text? I don't know much about it but have heard good things. It's UTF-8 internally, while Text uses UTF-16. This could help with memory consumption and would avoid the UTF-8 → UTF-16 conversion performance penalty.

[jgm]

+++ Albert Krewinkel [Jun 13 17 00:05 ]:

Heretical question: Would it make sense to use the String type from foundation instead of Text? I don't know much about it but have heard good things. It's UTF-8 internally, while Text uses UTF-16. This could help with memory consumption and would avoid the UTF-8 → UTF-16 conversion performance penalty.

I was thinking that using Text would make most sense in the API, since users are most likely using Text for their Texts. This also allows people to use pandoc as a library without worrying about encodings. I don't think switching to foundation would be easy, and it would make the API use a less common type, so that users would more likely have to convert. Also, Text has some nice functions that foundation's String doesn't. UTF8 decoding takes hardly any time in relation to the time taken by the parser, so I don't think that's a good place to focus our efforts. I'm also skeptical that our memory issues would be much helped by switching to UTF-8 internally. Memory usage didn't go down that much when I switched the HTML reader to use Text throughout, so I think the main problem is not from the representation of the input text.