nested fenced code block messes up source highlighting in the editor

[cderv]

System details

RStudio Edition : Desktop [Open Source]
RStudio Version : 2022.1.0.199
OS Version      : Windows 10 x64 (build 19043)
R Version       : R version 4.1.1 (2021-08-10)

Steps to reproduce the problem

Open a new rmarkdown document and paste the following content

---
title: "Test"
output: html_document
---

# A Header

````{verbatim}
We can output arbitrary content verbatim.
  
```{r}
1 + 1
```

The content can contain inline code like
`r pi * 5^2`, too.
````

# Another header 

Some content

Describe the problem in detail

The source editor will not correctly recognized the nested chunk and it will mess up the highlighting

Example with the code chunk not being correctly highlighted, and headers not correctly found after the chunk.

Describe the behavior you expected

This syntax is something rather new in knitr. Last version support nested chunks and some engine will benefit from this like the comment engine or the verbatim engine.

WIth the current IDE behavior, it is rather hard to write and read such content.

I would expect the highlighting of the Markdown document in the IDE to be the same as without nested chunk.

cc @yihui

[ronblum]

Reproducible with older versions of RStudio as well, going back to RStudio Desktop 1.2.5042 on MacOS 12.

[kevinushey]

I believe supporting this will require a few changes.

First, the chunk highlighter would need to become stateful, and track the number of backticks used in each chunk header / footer. That implementation lives here:

https://github.com/rstudio/rstudio/blob/main/src/gwt/src/org/rstudio/studio/client/workbench/views/source/editors/text/ace/AceBackgroundHighlighter.java

Second, the Ace highlight rules would need to become stateful as well. Those rules are activated here:

rstudio/src/gwt/acesupport/acemode/rmarkdown_highlight_rules.js

Lines 58 to 66 in 686bc16

	// Embed R highlight rules
	Utils.embedRules(
	this,
	RHighlightRules,
	"r",
	this.$reChunkStartString,
	this.$reChunkEndString,
	["start", "listblock", "allowBlock"]
	);

And the embedRules() helper lives here:

rstudio/src/gwt/acesupport/acemode/utils.js

Lines 98 to 143 in 686bc16

	this.embedRules = function(HighlightRules, EmbedRules,
	prefix, reStart, reEnd,
	startStates, endState)
	{
	if (typeof startStates === "undefined")
	startStates = ["start"];
	if (typeof endState === "undefined")
	endState = "start";
	startStates = that.asArray(startStates);
	var rules = HighlightRules.$rules;
	// some highlight modes (notably, YAML) can have rules
	// that manipulate a stack of saved states -- in the
	// context of YAML, this is done to handle e.g. multiline
	// strings (tracking the indent used for those strings).
	// we need to clear that stack whenever we switch from one
	// mode to another.
	var onMatch = function(value, state, stack, line) {
	stack.splice(0);
	return this.token;
	}
	// define the highlight rules that allow us to transition
	// into the embedded mode
	for (var i = 0; i < startStates.length; i++) {
	rules[startStates[i]].unshift({
	token: "support.function.codebegin",
	regex: reStart,
	onMatch: onMatch,
	next: prefix + "-start"
	});
	}
	// call into Ace to embed rules with given 'prefix', and
	// define the rule that's used to return to parent mode
	HighlightRules.embedRules(EmbedRules, prefix + "-", [{
	token: "support.function.codeend",
	regex: reEnd,
	onMatch: onMatch,
	next: "start"
	}]);
	};

The onMatch() rule would need to both track and check the number of backticks that were matched, and only switch states when the "correct" closing number of backticks is seen.