Typedesc forwarding: macros lose typedesc information

[bluenote10]

Forwarding a typedesc works in general for procs, i.e., if a proc has a broad typedesc argument T: typedesc it can call narrower procs that have a specific T: typedesc[<some-type>] argument. The same does not work for macros:

import macros

proc f(T: typedesc[SomeInteger]) =
  echo "called f with some integer"

proc f(T: typedesc[string]) =
  echo "called f with string"

proc aProc(T: typedesc) =
  f(T)

macro aMacro(T: typedesc) =
  f(T)

# aProc properly forwards the typedesc to f
aProc(int)
aProc(string)

# aMacro loses the exact typedesc and tries to call f(T: typedesc)
aMacro(int)
aMacro(string)

Error:

test.nim(13, 4) Error: type mismatch: got (typedesc)
but expected one of: 
proc f(T: typedesc[string])
proc f(T: typedesc[int])

[jangko]

I don't think this is a bug.
inside a macro, both 'int' and 'string' have the same type that is 'typedesc'.
'int', 'string', 'float', etc are values of a 'typedesc', same as 1,2,3,4 are values of an 'int'.
what Nim lacks is #6785

[bluenote10]

I had a brief conversation with @Araq about that on IRC. He mentioned that this may be caused by some special casing in overloading resolution for macros, and it sounded like it is solvable. You're right, I only noticed this issue searching for a workaround to #6785.

[krux02]

This issue has been fixed differently. The error message is now the following:

/tmp/scratch.nim(13, 4) Error: type mismatch: got <NimNode>
but expected one of: 
proc f(T: typedesc[SomeInteger])
  first type mismatch at position: 1
  required type for T: typedesc[SomeInteger]
  but expression 'T' is of type: NimNode
proc f(T: typedesc[string])
  first type mismatch at position: 1
  required type for T: typedesc[string]
  but expression 'T' is of type: NimNode

As typedesc arguments are now passed in as NimNodes.

[bluenote10]

Hm, and how can we make such code work? The code looks it should compile...

The fix has turned the problem into #6785 which I still don't understand why it had to be closed without a solution. It would be nice to extend the documentation why such code is conceptionally impossibly and how to work-around the issue.

[krux02]

I think what you want to solve your problem is a macro API for type relations, the is keyword that works on NimNode. Currently all we have is sameType and that does not work for typeClasses such as SomeInteger:

macro aMacro(T: typedesc) =
  let typ = getTypeInst(T)[1]
  if sameType(typ, bindSym"SomeInteger"):
    echo "type is SomeInteger" # never called
  if sameType(typ, bindSym"string"):
    echo "type is string"

If that is what you want, please make a separate feature request for it. The title issues Typedesc forwarding: macros lose typedesc information has been solved.

[bluenote10]

Not quite, the use case was to call any existing function that takes a typedesc argument from within a macro. I run into this issue a lot, I design a macro to take a typedesc argument and at some point I need to call an existing typedesc function from the macro world, mainly standard library stuff like:

• proc name*(t: typedesc): string (e.g. for error messages)
• proc arity*(t: typedesc): int
• proc default*(T: typedesc): T
• proc high*(T: typedesc[SomeFloat]): T / proc low*(T: typedesc[SomeFloat]): T
• proc sizeof*(x: typedesc): int
• proc alignof*(x: typedesc): int
• proc rand*[T: SomeInteger](t: typedesc[T]): T
• proc none*(T: typedesc): Option[T]

The title issues Typedesc forwarding: macros lose typedesc information has been solved.

After the conversion to NimNode the argument is no longer usable as a typedesc, which was what I meant by "loses typedesc information".

[krux02]

For name you can use repr, should work pretty well for error messages. arity is easy to reimplement using macros. For sizeof/alignof there is getSize/getAlign/getOffset. For anything that returns T ... I don't know.

[Araq]

I think there is no problem here, sorry. In the macro you have NimNodes, you can inspect the type via the getType family of procs. You can construct calls to rand quite easily but there is no reason to call rand in your macro with a typedesc. Think about it, it would mean we should produce generic specializations for generic macros. While that might be useful, it's not a feature that we currently have.

[zah]

I have explained the original vision for typedesc parameters here:
nim-lang/RFCs#148 (comment)

Indeed, one of my goals was to make all typetraits usable both inside and outside of macros. We can work towards this goal by re-implementing the existing built-in type traits in terms of macros that work with the AST nodes returned by getType.

Certain type traits might be defined as templates. Consider this example:

template ElemType*[T](x: typedesc[openarray[T]]): untyped = T
template ElemType*(x: typedesc[string]): untyped = char

A perfect compiler will allow me to use such templates inside a macro in the following way:

macro foo(T: typedesc): untyped =
   var E = ElemType(T)
   echo E.treeRepr

This can work if the compiler is able to automatically insert getAst on the first line of the macro, but there are additional complications, because getAst will have to learn how to resolve and pick the right overload depending on the "run-time" type given to the macro.

This is controversial, because resolving overloads depends on the current scope, so we'll have to define that macros are evaluated in a particular scope. For me, this is a good development, because it will eventually allow us to introduce magics such as determineType (or typecheck), which will greatly expand the types of DSLs that can be created in Nim. Please note that we've already set precedent by introducing a late-bound bindSym that takes into account the scope where the macro is expanded.

The original design was also handling the is operator in a special way. The is operator is quite different, because it's internally defined to work over the compiler's PType values (not over NimNodes). To restore the previously available functionality, I think the best approach is for the VM to keep the types of nodes stored in the regular node.typ field and then the is operator can be defined to use that field. For added type safety, we can introduce a type NimTypeNode = distinct NimNode which will be returned from magics such as getType and determineType and the is operator can be defined only for NimTypeNode.

[Araq]

For added type safety, we can introduce a type NimTypeNode = distinct NimNode which will be returned from magics such as getType and determineType and the is operator can be defined only for NimTypeNode.

That is a good idea in general.

[bluenote10]

You can construct calls to rand quite easily but there is no reason to call rand in your macro with a typedesc.

Why not? I can't remember my original use cases but they all had to do with moving work from runtime to compile time. In the rand example: When generating large random samples, it would be a reasonable optimization to directly produce an AST [5, 2, 4, 9, 2, ...] instead of [rand(10), rand(10), rand(10), rand(10), rand(10), ...] (which becomes more significant when the work gets non-trivial like using rejection sampling). This particular example can probably be solved by const -- just illustrating the motivation.

But having done some more experiments I see that this is simply not how generic macros work.

[krux02]

@bluenote10

In the rand example: When generating large random samples ...

You should do this at initialization time of the program. Anything else will bloat your executable with unnecessary data and it will kill your compilation time as not only do your random numbers need to be calculated by the NimVm which is already not very fast, they also need to be hauled through the C compiler.

Apart from this, I don't see how this is related to typedesc.

, it would be a reasonable optimization to directly produce an AST [5, 2, 4, 9, 2, ...] instead of [rand(10), rand(10), rand(10), rand(10), rand(10), ...]

(which becomes more significant when the work gets non-trivial like using rejection sampling). This particular example can probably be solved by const -- just illustrating the motivation.

But having done some more experiments I see that this is simply not how generic macros work.