Finish closures before completing futures (fix #415) #418

Menduist · 2023-07-11T09:46:49Z

The idea of this PR would be that the closure never completes the future, instead just writes to the internalValue, and the futureContinue would take care of completing the future (for now I've just applied this to the return rewrite, but could be used everywhere)
Besides fixing #415, it could also avoid a copy to the internalValue once the future is finished

However, might have some downsides

arnetheduck · 2023-07-13T06:38:30Z

chronos/asyncfutures2.nim

@@ -317,6 +317,9 @@ proc futureContinue*(fut: FutureBase) {.raises: [], gcsafe.} =
      # `await` typically) or completes / fails / is cancelled
      next = fut.internalClosure(fut)
      if fut.internalClosure.finished(): # Reached the end of the transformed proc
+        if not fut.finished:


hm, this is actually a really good point: we should not finish the future until the iterator itself has finished or an order inversion will indeed happen! it deserves to be documented.

it means that the above should not be a conditional but rather the norm - we should always arrive at this point with an unfinished future.

That in turn means that the expression return stuff needs to be rewritten as well to assign to value rather than calling complete - ditto error paths which need to set the error field but not complete the future etc..

it's actually a classic bug not to run iterators until completion and thus missing part of the code - the same happens with simple for loop iterators that have code after the last yield: if you break the for loop that code will not run..

Done, though it's messing up the location[Finish]. Not sure what's the cleanest way forward, setFinishLocation called from the macro, and a completeWithInternalValue called from futureContinue?

setFinishLocation

or just use internalLocation[...] - ie none of this should be exposed in any public functions or API so it doesn't greatly matter - we can design a better public API for these kinds of things once the dust has settled

Note that this slightly changed the Finish location (the proc line instead of the first line of the proc)
Not sure this greatly matters

arnetheduck · 2023-07-14T13:42:57Z

chronos/asyncfutures2.nim

@@ -317,6 +317,8 @@ proc futureContinue*(fut: FutureBase) {.raises: [], gcsafe.} =
      # `await` typically) or completes / fails / is cancelled
      next = fut.internalClosure(fut)
      if fut.internalClosure.finished(): # Reached the end of the transformed proc
+        #TODO hacky


this needs a comment explaining why it's important that the future remains unfinished until the iterator is done (ie to run finally code fully, or really, any code after the last yield) - it's not actually hacky, it follows the way the compiler generates code for result - ie return xxx is typically rewritten as result = xxx; return - it's nuts but here we are.

chronos/asyncmacro2.nim

arnetheduck · 2023-07-14T13:45:14Z

funny enough, this PR should also improve performance because we will no longer generate a temporary variable that must be zero:ed then copied to the future

Menduist · 2023-07-24T11:03:55Z

This branch is failing on nimbus:

/home/tavon/Status/nimbus-eth2/beacon_chain/el/el_manager.nim(724, 18) template/generic instantiation of `err` from here
/home/tavon/Status/nimbus-eth2/beacon_chain/el/el_manager.nim(717, 42) Error: cannot instantiate Future [type declared in /home/tavon/Status/nimbus-eth2/vendor/nim-chronos/chronos/futures.nim(68, 3)]
got: <T>
but expected: <T>

Some generic fun to fix

EDIT: caused by nim-lang/Nim#22645

cheatfate · 2023-07-25T10:16:38Z

chronos/asyncmacro2.nim

-              quote do: {.pop.},
-            )
+            quote do:
+              template result: auto {.used.} = `castFutureSym`.internalValue


Whatever you want to do here, please avoid quote do statements because this statements makes all our stack traces to point into asyncmacro2.nim file instead of pointing to actual source.

cheatfate · 2023-07-25T10:18:42Z

chronos/asyncmacro2.nim

-    let
-      res = newNimNode(nnkStmtList, node)
-    res.add completeWithNode(fut, baseType, processBody(node[0], fut, baseType))
+    let res = newNimNode(nnkStmtList, node)


Please add the comment here with generated source code.

arnetheduck · 2023-08-08T15:07:46Z

this should have a test that shows the previously broken code snippet

arnetheduck · 2023-09-05T09:08:39Z

chronos/asyncmacro2.nim

        )
      ),
      nnkElseExpr.newTree(
-        newCall(ident "complete", fut, node)
+        newAssignment(ident "result", node)


this should directly use internalValue rather than going via the template (more templates = more problems)

and processBody..

This proc is only used with procBodyBlck which is already went through processBody

more templates = more problems

That's unfortunately true, since using internalValue instead of result fixed some compilation issue in nimbus
Looks like something that might bite us later

arnetheduck · 2023-09-05T09:09:30Z

chronos/asyncmacro2.nim

+        nnkElifExpr.newTree(
+          nnkInfix.newTree(
+            ident "isnot", baseType, ident "void"),
+          newAssignment(ident "result", node[0])


ditto, avoid result - also, node[0] needs to pass through processBody

for the test suite:

return if 1 == 1: return 42 else: 33 assert returnedValue == 42

ie nested returns are valid code

chronos/asyncmacro2.nim

Co-authored-by: Jacek Sieka <jacek@status.im>

arnetheduck · 2023-09-06T08:40:32Z

chronos/asyncfutures2.nim

@@ -165,6 +165,12 @@ template complete*(future: Future[void]) =
  ## Completes a void ``future``.
  complete(future, getSrcLocation())

+proc completeWithInternalData(future: FutureBase) =


Suggested change

proc completeWithInternalData(future: FutureBase) =

proc completeWithInternalValue(future: FutureBase) =

arnetheduck · 2023-09-06T08:42:43Z

chronos/asyncmacro2.nim

-    )
-
-proc completeWithNode(fut, baseType, node: NimNode): NimNode {.compileTime.} =
+proc completeWithNode(node, resultNode: NimNode): NimNode {.compileTime.} =


assignOrProcessNode

Suggested change

proc completeWithNode(node, resultNode: NimNode): NimNode {.compileTime.} =

proc assignOrProcessNode(node, resultNode: NimNode): NimNode {.compileTime.} =

cheatfate · 2023-09-07T09:46:10Z

chronos/asyncmacro2.nim

+          nnkPragma.newTree(
+            nnkExprColonExpr.newTree(
+              newIdentNode("error"),
+              newLit("no async return type declared"))


This is confusing error message for everybody, even i do not understand what it mean.

proc x: void = return 42

testit.nim(2, 3) Error: no return type declared

this is what the compiler normally prints - not great but... maybe "cannot return value in function returning void"

also, the call to error should include node so that line number is included

The node doesn't seem necessary in this case, it already points to the right line (probably since it's just creating the pragma, that will pop the error later)

And lesson learned, shouldn't copy the compiler error messages 😛

Compiler provides no return type declared your code provides no async return type declared.
Where you found any mention of async return type and what the difference with return type is unclear.

A lot of compiler error messages are "cryptic" and sometimes when you generate code using macro, compiler could generate absolutely inadequate error message which could be understand only by macro code author. In this case generating error message is author's duty, because at the end error message will be read by library user, and it should not be cryptic, it should has meaning in context.

cheatfate · 2023-09-07T09:48:05Z

tests/testmacro.nim

+      try:
+        return
+      finally:
+        await stepsAsync(1)


I want to see more than one await in finally. and also i want to see how it works in exception handler, e.g. more than one await. Also we need to test defer functionality.

arnetheduck · 2023-09-07T10:21:19Z

chronos/asyncmacro2.nim


    # don't do anything with forward bodies (empty)
    if procBody.kind != nnkEmpty:
      let
        # fix #13899, `defer` should not escape its original scope
        procBodyBlck = nnkBlockStmt.newTree(newEmptyNode(), procBody)

+        # workaround https://github.com/nim-lang/Nim/issues/22645
+        internalFutureTypeSym = genSym(nskType, "InternalFutureType")
+        castFutureSym2 = nnkCast.newTree(internalFutureTypeSym, internalFutureSym)


expanded code comment

Menduist · 2023-09-07T14:56:18Z

We found a failure with this PR, and no workaround yet:

  proc testReturner: Future[int] {.async.} =
    template returner =
      result = 5
    returner

I'll look a bit more for a workaround, otherwise the solution will be to retain the temporary result variable.

arnetheduck · 2023-10-16T09:55:47Z

Superseded by #449

Tentative fix for #415

358fc53

Menduist mentioned this pull request Jul 11, 2023

Better support for nim-result #217

Closed

arnetheduck reviewed Jul 13, 2023

View reviewed changes

remove all completes from asyncmacro

e829afd

arnetheduck reviewed Jul 14, 2023

View reviewed changes

chronos/asyncmacro2.nim Outdated Show resolved Hide resolved

Menduist added 2 commits July 24, 2023 12:49

Fill the internalLocation

845b1a1

improve comment

3bb4e76

fix

a1c41ee

cheatfate reviewed Jul 25, 2023

View reviewed changes

arnetheduck mentioned this pull request Aug 8, 2023

Unroll defers and remove breaks. #440

Merged

arnetheduck reviewed Sep 5, 2023

View reviewed changes

Menduist added 3 commits September 5, 2023 15:19

Merge remote-tracking branch 'origin/master' into fixes

af65ad2

now compiles with nimbus

f959575

add tests

382a96f

Menduist changed the title ~~Tentative fix for #415~~ Finish closures before completing futures (fix #415) Sep 5, 2023

Menduist marked this pull request as ready for review September 5, 2023 13:26

arnetheduck reviewed Sep 5, 2023

View reviewed changes

chronos/asyncmacro2.nim Outdated Show resolved Hide resolved

Update chronos/asyncmacro2.nim

68fd98f

Co-authored-by: Jacek Sieka <jacek@status.im>

arnetheduck reviewed Sep 6, 2023

View reviewed changes

cheatfate reviewed Sep 7, 2023

View reviewed changes

arnetheduck reviewed Sep 7, 2023

View reviewed changes

address reviews

8cf81ad

Menduist marked this pull request as draft September 7, 2023 14:56

Menduist mentioned this pull request Sep 26, 2023

Complete futures in closure finally (fix #415) #449

Merged

arnetheduck mentioned this pull request Oct 16, 2023

Write to Future.internalValue directly in macro-translated code #452

Open

arnetheduck closed this Oct 16, 2023

arnetheduck deleted the fixes branch November 17, 2023 13:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finish closures before completing futures (fix #415) #418

Finish closures before completing futures (fix #415) #418

Menduist commented Jul 11, 2023 •

edited

arnetheduck Jul 13, 2023

arnetheduck Jul 13, 2023

Menduist Jul 13, 2023

arnetheduck Jul 21, 2023

Menduist Jul 24, 2023

Menduist Jul 24, 2023

arnetheduck Jul 14, 2023

Menduist Jul 24, 2023

arnetheduck commented Jul 14, 2023

Menduist commented Jul 24, 2023 •

edited

cheatfate Jul 25, 2023

cheatfate Jul 25, 2023

arnetheduck commented Aug 8, 2023

arnetheduck Sep 5, 2023

arnetheduck Sep 5, 2023

Menduist Sep 5, 2023

arnetheduck Sep 5, 2023

Menduist Sep 5, 2023 •

edited

arnetheduck Sep 5, 2023

arnetheduck Sep 5, 2023 •

edited

arnetheduck Sep 6, 2023

arnetheduck Sep 6, 2023

cheatfate Sep 7, 2023

arnetheduck Sep 7, 2023 •

edited

Menduist Sep 7, 2023

Menduist Sep 7, 2023

cheatfate Sep 7, 2023

cheatfate Sep 7, 2023

cheatfate Sep 7, 2023

arnetheduck Sep 7, 2023

Menduist commented Sep 7, 2023

arnetheduck commented Oct 16, 2023

	proc completeWithInternalData(future: FutureBase) =
	proc completeWithInternalValue(future: FutureBase) =

	proc completeWithNode(node, resultNode: NimNode): NimNode {.compileTime.} =
	proc assignOrProcessNode(node, resultNode: NimNode): NimNode {.compileTime.} =

Finish closures before completing futures (fix #415) #418

Finish closures before completing futures (fix #415) #418

Conversation

Menduist commented Jul 11, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnetheduck commented Jul 14, 2023

Menduist commented Jul 24, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnetheduck commented Aug 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Menduist Sep 5, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnetheduck Sep 5, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnetheduck Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Menduist commented Sep 7, 2023

arnetheduck commented Oct 16, 2023

Menduist commented Jul 11, 2023 •

edited

Menduist commented Jul 24, 2023 •

edited

Menduist Sep 5, 2023 •

edited

arnetheduck Sep 5, 2023 •

edited

arnetheduck Sep 7, 2023 •

edited