python chunks with empty lines aren't run

[kevinushey]

System details

RStudio Edition : 2022.10.0-daily+68
RStudio Version : Desktop [Open Source]
OS Version      : macOS Monterey 12.4
R Version       : R version 4.2.1 (2022-06-23)

Steps to reproduce the problem

Create an R Markdown document, with a chunk containing the following:

```{python}
def example():
  print("I'm part of the function definition!")

  print("I'm also part of the function definition!")

example()
```

Then, try to run it using the chunk Run button.

Describe the problem in detail

The empty line "ends" the function definition, and so the second print() statement doesn't actually become part of the code.

>>> def example():
...   print("I'm part of the function definition!")
... 
>>>   print("I'm also part of the function definition!")
I'm also part of the function definition!
>>> 
>>> example()
I'm part of the function definition!

Describe the behavior you expected

The code should be parsed and run as a single unit, so that the resulting code is:

>>> def example():
...   print("I'm part of the function definition!")
...   
...   print("I'm also part of the function definition!")
... 
>>> example()
I'm part of the function definition!
I'm also part of the function definition!

---

Session Info

R version 4.2.1 (2022-06-23)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.4

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] renv_0.15.5-43

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9          here_1.0.1          lattice_0.20-45    
 [4] png_0.1-7           rprojroot_2.0.3     digest_0.6.29      
 [7] grid_4.2.1          jsonlite_1.8.0      evaluate_0.15      
[10] rlang_1.0.4         cli_3.3.0           rstudioapi_0.13    
[13] Matrix_1.4-1        reticulate_1.25     rmarkdown_2.14     
[16] tools_4.2.1         xfun_0.31           yaml_2.3.5         
[19] fastmap_1.1.0       compiler_4.2.1      BiocManager_1.30.18
[22] htmltools_0.5.3     knitr_1.39         

[kevinushey]

We have a bunch of code that tries to pre-process console input here, to fix up indents:

rstudio/src/cpp/session/SessionConsoleInput.cpp

Lines 161 to 288 in 78304db

	void fixupPendingConsoleInput()
	{
	using namespace r::session;
	// get next input
	auto input = s_consoleInputBuffer.front();
	// nothing to do if this is a cancel
	if (input.isCancel() || input.isEof())
	return;
	// if this has no newlines, then nothing to do
	auto index = input.text.find('\n');
	if (index == std::string::npos)
	return;
	// if we're about to send code to the Python REPL, then
	// we need to fix whitespace in the code before sending
	bool pyReplActive = modules::reticulate::isReplActive();
	// pop off current input (we're going to split and re-push now)
	s_consoleInputBuffer.pop_front();
	// does this Python line start an indented block?
	// NOTE: should consider using tokenizer here
	boost::regex reBlockStart(":\\s*(?:#|$)");
	// used to detect whitespace-only lines
	boost::regex reWhitespace("^\\s*$");
	// keep track of the indentation used for the current block
	// of Python code (default to no indent)
	std::string blockIndent;
	// pending console input (we'll need to push this to the front of the queue)
	std::vector<RConsoleInput> pendingInputs;
	// split input into list of commands
	std::vector<std::string> lines = core::algorithm::split(input.text, "\n");
	for (std::size_t i = 0, n = lines.size(); i < n; i++)
	{
	// get current line
	std::string line = lines[i];
	// fix up indentation if necessary
	if (pyReplActive)
	{
	// if the line is empty, then replace it with the appropriate indent
	// (exclude last line in selection so that users submitting a whole
	// 'block' will see that block 'closed')
	if (line.empty() && i != n - 1)
	{
	line = blockIndent;
	}
	// if this line would exit the reticulate REPL, then update that state
	else if (line == "quit" || line == "exit")
	{
	blockIndent.clear();
	pyReplActive = false;
	}
	// if it looks like we're starting a new Python block,
	// then update our indent. perform a lookahead for the
	// next non-blank line, and use that line's indent
	else if (regex_utils::search(line, reBlockStart))
	{
	for (std::size_t j = i + 1; j < n; j++)
	{
	const std::string& lookahead = lines[j];
	// skip blank / whitespace-only lines, to allow
	// for cases like:
	//
	// def foo():
	//
	// x = 1
	//
	// where there might be empty whitespace between
	// the function definition and the start of its body
	if (regex_utils::match(lookahead, reWhitespace))
	continue;
	blockIndent = string_utils::extractIndent(lookahead);
	break;
	}
	}
	// if the indent for this line has _decreased_, then we've
	// closed an inner block; e.g. for something like:
	//
	// def foo():
	// def bar():
	// "bar"
	// "foo" <--
	//
	// so update the indent in that case
	else if (!regex_utils::match(line, reWhitespace))
	{
	std::string lineIndent = string_utils::extractIndent(line);
	if (lineIndent.length() < blockIndent.length())
	blockIndent = lineIndent;
	}
	}
	else
	{
	// check for a line that would enter the Python REPL
	if (line == "reticulate::repl_python()" ||
	line == "repl_python()")
	{
	pyReplActive = true;
	}
	}
	// add to buffer
	pendingInputs.push_back(
	RConsoleInput(line, input.console, input.flags));
	}
	// now push the pending inputs to the front of the queue
	for (auto it = pendingInputs.rbegin();
	it != pendingInputs.rend();
	it++)
	{
	s_consoleInputBuffer.push_front(*it);
	}
	}

It seems like this code is working as expected for regular Python scripts, but not for Python chunks in R Markdown documents. (Most likely, this is because the R Markdown chunk executor sends the code a single line at a time, whereas the standalone executor tries to process the whole input at once?)

[kevinushey]

I think the cleanest solution would be to pre-process chunk code before submitting it, to fix up blank line indentation so that it matches the user's expectation.

[kevinushey]

(Another alternative would be to save the last console input / current block indentation on the session side, and then use that to fix up console input -- but we'd need to have some way to differentiate between "blocks" of Python code versus a single empty line of Python code which might intentionally be terminating a block)

[ronblum]

Verified fixed in RStudio Desktop 2022.11.0-daily+98 on MacOS 12.5.

> reticulate::repl_python()
>>> def example():
...   print("I'm part of the function definition!")
...   
...   print("I'm also part of the function definition!")
...   
... example()
I'm part of the function definition!
I'm also part of the function definition!
>>> 
>>> quit
> 

[ronblum]

Leaving open as candidate for an automated test. I'm not sure it's worth doing so but at least to be considered. (@jonvanausdeln feel free to close if you don't think this is a good candidate.)

[mikebessuille]

can't leave this open in Elsbeth, as EG is done. I think I'll close this, as the remaining automation is already reflected in a separate ticket (859).