iterator broken when nesten inside a function

[krux02]

This code here works as expected

let input = "Hello \"Apple\" I am the \"Evil Sausage\""

block foobar:
  var readPos = 0

  iterator readInput(): char {.exportc.} =
    echo "readInput:  readPos: ", readPos, " input.len: ", input.len
    while readPos < input.len:
      readPos += 1
      yield input[readPos-1]

  proc parseString(): string {.exportc.} =
    result = ""
    for c in readInput():
      if c == '"':
        break
      else:
        result.add c

  for c in readInput():
    if c == '"':
      let str = parseString()
      echo "got string token: \"", str, "\""

The output is:

readInput:  readPos: 0
readInput:  readPos: 7
got string token: "Apple"
readInput:  readPos: 24
got string token: "Evil Sausage"

The iterator is called three times, and both string tokens, "Aplle" and "Evil Sausage" could be identified.

This version here does not work, and the only difference is, the block is changed to be a function.

let input = "Hello \"Apple\" I am the \"Evil Sausage\""

proc foobar(): void {.exportc.} =
  var readPos = 0

  iterator readInput(): char {.exportc.} =
    echo "readInput:  readPos: ", readPos, " input.len: ", input.len
    while readPos < input.len:
      readPos += 1
      yield input[readPos-1]

  proc parseString(): string {.exportc.} =
    result = ""
    for c in readInput():
      if c == '"':
        break
      else:
        result.add c

  for c in readInput():
    if c == '"':
      let str = parseString()
      echo "got string token: \"", str, "\""

foobar()

The output is:

readInput:  readPos: 0
got string token: ""
got string token: ""
got string token: ""
got string token: ""

The output is expected to be identical to the previous version. But you can see that for some reason readInput is only called once and all the other iterations are left empty. The program parses an empty string token at every " character in the input. There are four " characters in the input therefore there are four empty strings. The loop in parseString does not process a single character.

Here is the generated C code of the function parseString, you can easily see that the loop is not even generated:

N_NIMCALL(NimStringDesc*, parseString)(void) {
	NimStringDesc* result;
	result = (NimStringDesc*)0;
	result = copyString(((NimStringDesc*) &TM_klN7BP46qk1hZ9azdyZt9cMA_8));
	{
	}
	return result;
}

Here is the generated C code of the exact same parseString function but from the upper code section. This is how it should look like.

N_NIMCALL(NimStringDesc*, parseString)(void) {
	NimStringDesc* result;
	result = (NimStringDesc*)0;
	result = copyString(((NimStringDesc*) &TM_klN7BP46qk1hZ9azdyZt9cMA_4));
	{
		NIM_CHAR c;
		tyArray_Re75IspeoxXy2oCZHwcRrA T2_;
		c = (NIM_CHAR)0;
		memset((void*)T2_, 0, sizeof(T2_));
		T2_[0] = copyString(((NimStringDesc*) &TM_klN7BP46qk1hZ9azdyZt9cMA_5));
		T2_[1] = nimIntToStr(readPos_44V7z9cCDqfURJau9aZbdZnQ);
		echoBinSafe(T2_, 2);
		{
			while (1) {
				if (!(readPos_44V7z9cCDqfURJau9aZbdZnQ < (input_3QCOCQd9c9cl3sO64cYJq3iA ? input_3QCOCQd9c9cl3sO64cYJq3iA->Sup.len : 0))) goto LA4;
				readPos_44V7z9cCDqfURJau9aZbdZnQ += ((NI) 1);
				c = input_3QCOCQd9c9cl3sO64cYJq3iA->data[(NI)(readPos_44V7z9cCDqfURJau9aZbdZnQ - ((NI) 1))];
				{
					if (!((NU8)(c) == (NU8)(34))) goto LA7_;
					goto LA1;
				}
				goto LA5_;
				LA7_: ;
				{
					result = addChar(result, c);
				}
				LA5_: ;
			} LA4: ;
		}
	} LA1: ;
	return result;
}

Edit:
The exportc pragma is just to make the generated C code more readable. But it is not causing the problem or is something that I want to test here. It is debugging purpose only.

[narimiran]

Just to clarify (to myself) something:

In the first output there are readInput: readPos: 7 and readInput: readPos: 24 - meaning that iterator readInput() has started from the beginning, and not from where it has previously stopped in the while loop, right?

If that's the case, this seems like a candidate for another example in #7047.

[krux02]

@narimiran The iteratator from parse string should continue where the iterator in the outer loop left off. So it's a continuation of the same loop with different logic.

Hello "Apple" I am the "Evil Sausage"
^      ^                ^
|      |                `-- start of secord parseString: readPos: 24
|      |
|      `-- start of first parseString: readPos: 7
|
`-- outer loop start: readPos: 0