[RFC] Better iterators syntax

[narimiran]

Problem

Iterator syntax as currently implemented leaves a lot to be desired.

As seen in this forum post, there are couple of problems with current syntax:

• instantiating iterators with let x = fibo (without () and without parameters)
• calling the existing iterator with parameters which might ((1, 1) at the first call) or might not ((10, 20), (100, 999)) have anything to do with the current iterator
• wrapping iterator inside of a proc helps a bit regarding a better-looking syntax when calling the existing iterator

---

Proposal:

Use next to call the existing iterator. (Similar to other languages, e.g. Python)

iterator fibo(a, b: int): int = #1: some pragma here?
  var
    a = a
    b = b
  while true:
    yield a
    (a, b) = (b, a+b)

let  #2: instatiating iterators with their starting parameters
  x = fibo(1, 1)
  y = fibo(34, 55)

for _ in 1 .. 5:
  echo next(x)  #3: calling the existing iterator `x` - cannot pass any new parameter once the iterator is instatiated

echo next(y)  #4: first call of `y`

[bluenote10]

Related to: nim-lang/Nim#2563

What about hasNext(x)?

[data-man]

I like D's ranges.
Or range-v3 library for C++11/14/17.

[dom96]

What about hasNext(x)?

IIRC finished exists

[narimiran]

What about hasNext(x)?

This sounds like something that returns bool, and as @dom96 said - there is finished that serves probably the same/similar function.

In the example above, next calls the iterator and returns the next value in the iterator (resumes at the point after a previous yield).

[krux02]

I wrote a marco that allows a different syntax, what do you think about it. Be aware it's a proof of concept, not something final you want to work with. For example I broke the possibility to use this iterator in a simple for loop. Only the next syntax works with this version.

import macros

macro instanceIterator(iter: iterator, args: varargs[typed]): untyped =
  let formalParams = iter.getTypeInst[0]
  let tupleTy = nnkTupleTy.newTree
  let argsSym = genSym(nskVar, "tmp")
  let assignments = nnkPar.newTree()

  var argi = 0

  for i, node in formalParams:
    if i == 0:
      continue
    else:
      let identDefs = nnkIdentDefs.newTree
      for j, n in node:
        if j < node.len-2:
          let l = genSym(nskField, $n)
          identDefs.add l
          assignments.add( nnkExprColonExpr.newTree( l , args[argi] ) )
          argi += 1
        else:
          identDefs.add n
      tupleTy.add identDefs

  result = quote do:
    var `argsSym`: `tupleTy` = `assignments`
    (it: `iter`, args: `argsSym`)

proc getTupleLength(arg: NimNode): int {.compileTime.} =
  let typ = arg.getTypeInst
  result = 0
  typ.expectKind nnkTupleTy
  for identDefs in typ:
    identDefs.expectKind nnkIdentDefs
    result += identDefs.len - 2

macro forwardTupleAsArgs(call, args: typed): untyped =
  ## a function symbol `call` and a tuple symbol `args` will be
  ## transformed into `call(args[0], args[1], ...)`
  result = newCall(call)
  let numArgs = getTupleLength(args)
  for i in 0 ..< numArgs:
    result.add nnkBracketExpr.newTree(args, newLit(i))


import options

proc next[T: tuple](arg: T): Option[int] =
  let value = forwardTupleAsArgs(arg.it, arg.args)
  if not finished(arg.it):
    result = some(value)

macro newClosure(arg: untyped): untyped =
  ## This macro replaces the identifier of the iterator with a hidden
  ## symbol and uses the identifier wrappring procedure. The wrapping
  ## procedure creades an iterator object.

  let iteratorDef =
    if arg.kind == nnkStmtList:
      arg.expectLen 1
      arg[0]
    else:
      arg

  iteratorDef.expectKind nnkIteratorDef
  let newSym = genSym(nskIterator, $iteratorDef[0] & "Hidden")
  let oldSym = iteratorDef[0]
  iteratorDef[0] = newSym

  iteratorDef[4] = nnkPragma.newTree(
    newIdentNode("closure")
  )

  let formalParams = iteratorDef[3].copyNimTree
  formalParams.expectKind nnkFormalParams
  formalParams[0] = ident"untyped"

  let call = newCall( bindSym"instanceIterator", newSym )
  for i, identDefs in formalParams:
    if i == 0:
      continue
    else:
      identDefs.expectKind nnkIdentDefs
      for i in 0 ..< identDefs.len - 2:
        call.add ident($identDefs[i])

  let templateDef = nnkTemplateDef.newTree(
    oldSym, newEmptyNode(), newEmptyNode(),
    formalParams.copyNimTree, newEmptyNode(), newEmptyNode(),
    newStmtList( call )
  )

  result = newStmtList(
    iteratorDef,
    templateDef
  )

################################################################################
# from here is example code

echo "\nexample1:"
block example1:
  iterator fibo(a, b: int): int {.newClosure.} =
    var
      a = a
      b = b
    while true:
      yield a
      (a, b) = (b, a + b)

  let it = fibo(1, 1)

  echo "  ", it.next

  for _ in 1 .. 5:
    echo "  ", it.next

  echo "  ", it.next
  echo "  ", it.next

echo "\nexample2:"
block example2:

  iterator fiboFinite(a, b, c: int): int {.newClosure.} =
    var
      a = a
      b = b
    for _ in 0 ..< c:
      yield a
      (a, b) = (b, a + b)

  let it2 = fiboFinite(0,1,4)

  echo "  ", it2.next

  for _ in 1 .. 5:
    echo "  ", it2.next

  echo "  ", it2.next
  echo "  ", it2.next

This is the output:

example1:
  Some(1)
  Some(1)
  Some(2)
  Some(3)
  Some(5)
  Some(8)
  Some(13)
  Some(21)

example2:
  Some(0)
  Some(1)
  Some(1)
  Some(2)
  None[int]
  None[int]
  None[int]
  None[int]

EDIT:
It works now but it needs the latest development version of Nim. Otherwise the compiler will crash.

[nellore]

What about

proc next[T](iterable: T): untyped =
  ## Generic implementation of Python-like next()
  ## 
  ## iterable: iterable object
  ## 
  ## Result: next item from iterable or nil if iterator is exhausted
  for item in iterable:
    return item
  return nil

? For me it's solving the practical problem of cleanly advancing multiple iterators in a while true: depending on what they cough up.

[krux02]

@nellore good that it solves your purpose. But it is definitively not a general case solution, because it just doesn't compile on anything that is not a ref type. I see that you came from python and there this approach works because in python every type is like a ref type. That is not the case in Nim. Your example alreads fails at this simple example var arr = [1,2,3]; echo next(arr).

[nellore]

Oh, I see! Thanks.

[narimiran]

BUMP

[krux02]

@narimiran well you can give feedback on the macro or improve on it. For me it does exactly what you are asking for and it solves the problem to see if an iterator has ended in a clean and obvious way.

[narimiran]

you can give feedback on the macro or improve on it

I would gladly, but I can do neither of those two - I have zero experience with macros and I don't even understand the code you have written :)

Maybe somebody more experienced than me can give comment on your solution?

[krux02]

Well, you can give me feedback about the example part of the code. You don't need to comment on the macro implementation itself, just if you can do something with the result of it.

[bluenote10]

@krux02 The macro is nice, but without being able to use the iterator in a normal for loop it is not a general solution. The other reason is:

wrapping iterator inside of a proc helps a bit regarding a better-looking syntax when calling the existing iterator

What bothers me the most is the following inconsistency:

proc iterGen(someEvalOnceArg: int): (iterator (): int {.closure.}) =
  result = iterator (): int =
    yield 1
    yield 2
    yield 3

# this works fine
let it = iterGen(42)
for x in it():
  echo x

# this is an infinite loop
for x in iterGen(42)():
  echo x

I run into this infinite loop a lot, because the code intuitively should be equivalent. I was hoping that this RFC also addresses this. I see a possibility that if iterators would use next under the hood, the evaluation of for x in iterGen(42)(): can be modified so that it doesn't recreate the iterator over and over again. But that is just a guess...