Add some utils into LexAccessor and StyleContext

[zufuliu]

lexlib-0114.zip

code extract from https://github.com/zufuliu/notepad2/blob/main/scintilla/lexlib/LexAccessor.h and https://github.com/zufuliu/notepad2/blob/main/scintilla/lexlib/StyleContext.h
LexAccessor added GetCharacterAndWidth() and StyleAtEx() (to get cache style without call Flush()).
StyleContext added SeekTo(), Rewind(), BackTo() and Advance(). GetDocNextChar() and GetLineNextChar() (currently only skip ASCII whitespaces) may also useful for writing lexers.

[zufuliu]

BackTo() need updated to following to avoid unneeded flush.

void BackTo(Sci_PositionU startPos) {
	assert(startPos <= styler.GetStartSegment());
	if (startPos < styler.GetStartSegment()) {
		styler.Flush();
		styler.StartAt(startPos);
		styler.StartSegment(startPos);
	}
	SeekTo(startPos);
}

These function are useful to write advanced lexers with try parse rewind on failure approach, e.g. my enhanced lexer for GitHub Flavored Markdown, GitLab Flavored Markdown and Pandoc’s Markdown (though it still needs further work to handle indented code block).
https://github.com/zufuliu/notepad2/blob/main/scintilla/lexers/LexMarkdown.cxx

[nyamatongwe]

StyleAtEx is worthwhile but should have a name that explains the behavioural difference. Uncertain about name - maybe StyleAtReadingBuffer or BufferedStyleAt.

GetCharacterAndWidth could be useful in lexers. It might be nicer to return a struct (like Document::ExtractCharacter) or tuple instead of using an optional pointer argument. IDocument doesn't do this due to potential for ABI issues between separately built executables.

The implementation of SeekTo (and thus the other methods that call it) doesn't appear to set all the fields of StyleContext - atLineStart and chPrev is zeroed instead of setting to the previous character. Possibly others. There would be additional issues if seeking between lines - maybe this should be detected (throw?) or documented as a limitation.

IDocument::LineEnd returns the position before the line end characters so MatchLineEnd in the branch isn't behaving the same as in the main Lexilla repository. The point of MatchLineEnd is to switch styles for both single and multi-byte line ends and to eliminate situations where '\r' and '\n' are in different styles.

[zufuliu]

for StyleAtEx , what about CachedStyleAt?

It should documented SeekTo, BackTo and Rewind is intended for seeking on same line, multiline seeking is complex (line state changes).

chPrev is not needed in most time:

1. forward seeking is intended to skip some characters, which can limit the length to ensure chPrev is set after next Forward() call;
2. rewind means something failed, the lexer should call Forward() after call Rewind() or BackTo() to avoid infinite loop, which will set chPrev. if a lexer need re-parse text with another syntax which needs chPrev, following changes will help.

chPrev can only be set after change Document::GetCharacterAndWidth() to something like following to get character before position (the change it self might be useful):

if (pWidth && *pWidth < 0) {
	position = MovePositionOutsideChar(position, -1, false);
}

[nyamatongwe]

Pondered the name a bit and went for BufferStyleAt as the buffer is being written into only from the lexer. CachedStyleAt would be more appropriate if a block of styles was loaded from the document.

[nyamatongwe]

The proposed LexAccessor::GetCharacterAndWidth encourages mutable state. Returning a struct enables more use of const.

const FullCharacter ch = styler.CharacterAt(nextPos);
const auto [ch, width] = styler.CharacterAt(nextPos);

--- a/lexlib/LexAccessor.h
+++ b/lexlib/LexAccessor.h
@@ -12,6 +12,11 @@ namespace Lexilla {
 
 enum class EncodingType { eightBit, unicode, dbcs };
 
+struct FullCharacter {
+	int character = 0;
+	Sci_Position width = 1;
+};
+
 class LexAccessor {
 private:
 	Scintilla::IDocument *pAccess;
@@ -80,6 +85,11 @@ public:
 	Scintilla::IDocument *MultiByteAccess() const noexcept {
 		return pAccess;
 	}
+	FullCharacter CharacterAt(Sci_Position position) const {
+		FullCharacter fullCharacter;
+		fullCharacter.character = pAccess->GetCharacterAndWidth(position, &fullCharacter.width);
+		return fullCharacter;
+	}
 	/** Safe version of operator[], returning a defined value for invalid position. */
 	char SafeGetCharAt(Sci_Position position, char chDefault=' ') {
 		if (position < startPos || position >= endPos) {

[zufuliu]

OK then. Maybe CharacterAfter(Sci_Position position) and CharacterBefore(Sci_Position position) can also be added (with wraps GetRelativePosition and CharacterAt to avoid my proposed changes to Document::GetCharacterAndWidth()).

[zufuliu]

StyleContext-0225.zip

Updated SeekTo() to set atLineStart and chPrev; Updated BackTo() to support multiline rewind (which is need in some cases like implement Markdown multiline link text). Added comment for these methods.

[zufuliu]

There also needs better names for these method, may be rename SeekTo to MoveTo and rename BackTo to SeekTo is better, as the later can be used for multiline forward seek.
StyleContext-0225-rename.zip

[nyamatongwe]

SIZE_MAX requires cstdint or stdint.h. It often works because other headers include cstdint but it may break unusual systems.

The StyleContext constructor isn't simple, isn't performance-sensitive and would be better out-of-line in the implementation file, hiding the implementation.

Linked is a patch that does that. The patch does some of the same things as the above patches. It also makes fields const where possible, makes lineDocEnd private, initializes with field assignments where trivial, initializes using the initializer list when dependent on data.
54StyleContext.patch

[nyamatongwe]

The seeking methods are the main outstanding feature of this proposal. I think something similar could be useful but its difficult to understand when and how to use them (particularly the single line / multiple line split) so its less likely lexer developers will incorporate these techniques. Perhaps there is a subset that is strongly motivated and could be presented as a simple technique to use. An approach could be to implement the feature inside a lexer (possibly with a StyleContext subclass) then refine it until it appears finished then move it up into StyleContext.

[zufuliu]

The code is located at https://github.com/zufuliu/notepad2/blob/main/scintilla/lexlib/StyleContext.h#L218, multiline rewind is used by the Markdown lexer (with StyleContext sc; as a member)
https://github.com/zufuliu/notepad2/blob/f7b467d5ba1d5dae0bd7de6f19e88cc803528a7d/scintilla/lexers/LexMarkdown.cxx#L270-L276