Cache symbols while tokenizing. #92

merged 1 commit into from Oct 13, 2012


None yet

2 participants


This cut the number of Regexp comparisons roughly in half on the YAML file in #84 and around 18% on a VCR file I've been testing with. In both cases the cache didn't grow very large (i.e., the set of symbols was small but repeated). If memory growth is a concern, I could reimplement as an LRU cache. But relative to the rest of the parser, it seems like a minor overhead.


Definitely don't need an LRU. The tokenizer is constructed once per document, so the cache will get GC'd eventually. Considering the rest of the stuff in play, I don't think this cache is a concern.

@tenderlove tenderlove merged commit 74ed8ae into ruby:master Oct 13, 2012

1 check failed

default The Travis build failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment