Don't construct placeholder tokenized lines #12933

merged 11 commits into from Oct 14, 2016


None yet
2 participants

maxbrunsfeld commented Oct 11, 2016

After merging #11414, we changed how we represent syntactic boundaries to use TokenizedBufferIterator instead. This approach allows us to not build placeholder lines eagerly whenever there is a buffer change or when instantiating a new TokenizedBuffer.

screen shot 2016-10-14 at 15 34 23

In reality, we still need to construct TokenizedLine instances when TokenizedBuffer.prototype.tokenizedLineForRow is called because we want to construct some synthetic null grammar boundaries so that TokenizedBufferIterator doesn't iterate till the end of the buffer (regression), and also because other packages are currently relying on that API. In practice this should be pretty cheap, because in the majority of the cases it happens only for lines that are currently on screen.

maxbrunsfeld and others added some commits Oct 11, 2016

Handle null grammars consistently, building placeholder lines on-demand
Previously we were treating the local `NullGrammar` differently from
`atom.grammars.nullGrammar`. These two grammars are conceptually the
same, as the former was created to support editors creation without a
grammar registry.

To keep backwards-compatibility, we also build placeholder lines
on-demand when calling `TokenizedBuffer.prototype.tokenizedLineForRow`.
This ensure that packages relying on the internals of `TokenizedBuffer`
to retrieve syntactic boundaries won't break when the null grammar is
used or large file mode is on.
Use `TokenizedBuffer.prototype.tokenizedLineForRow` conservatively
Since this method will now construct a placeholder line, we want to use
it only where necessary to keep memory footprint to a minimum.

as-cii added some commits Oct 12, 2016

Delete unused conditional in `tokenizeNextChunk` and `invalidateRow`
Previously, for null grammar and large file mode, we were short
circuiting the tokenization of the next chunk or the invalidation of a
row. However, that is unnecessary because there is no additional chunk
to process with null grammar or in large file mode.

@as-cii as-cii merged commit 69567ed into master Oct 14, 2016

5 checks passed

ci/circleci Your tests passed on CircleCI!
continuous-integration/appveyor/branch AppVeyor build succeeded
continuous-integration/appveyor/pr AppVeyor build succeeded
continuous-integration/travis-ci/pr The Travis CI build passed
continuous-integration/travis-ci/push The Travis CI build passed

@as-cii as-cii deleted the mb-remove-placeholder-tokenized-lines branch Oct 14, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment