enable FFI at CT

[timotheecour]

use cases

• there are so many, I don't think we need to justify the need. Just one example: enabling nre (or re) at CT (which isn't possible without FFI at CT because it uses PRCRE lib cf irc /cc @flaviut )

discussion

EDIT

in https://forum.nim-lang.org/t/4249#26456 @Araq mentioned he had a working patch that allowed using FFI at CT for arbitrary importc procs, but it wasn't merged because it wasn't clear what to do with cases like these:

void a(int* x; int L);

Does the a write into x? If it does, we need to write back the changes performed to the blob x to the symbolic representation the VM uses.

However we can definitely think of solutions for this case, eg:

• pragma annotations to distinguish this
• disallow specific cases like this for now, but allow other cases that don't have this issue

links

EDIT

• keeps popping up in IRC, eg: https://gitter.im/nim-lang/Nim?at=5bc3e7a3ef4afc4f2875952b
• https://forum.nim-lang.org/t/4249#26456 (Why can't the Nim VM use FFI?)
• Add 'read' to naming scheme #61 (comment) comment by @zah

Integration of libffi in the evaluation engine that will allow calling arbitrary C functions

• https://github.com/Araq/libffi libffi wrapper for Nim.
• comment from @Araq https://forum.nim-lang.org/t/1743#10936

The VM used to support the FFI (via libffi) and I can dig out this code again, but [...]

• http://www.chiark.greenend.org.uk/doc/libffi-dev/html/The-Basics.html

[joshgoebel]

Generating code dynamically from the filesystem is a little hard when you can't do anything other than read statically known file locations at compile time. Ie, walkFiles and walkDirs are off limits because they don't seem to work at compile time either.

[timotheecour]

well there's staticExec("find ...") as workaround in some cases but indeed it's far from ideal (eg platform specific, and shelling out is usually not as good as calling API's); in other cases though, as noted, there's currently no good workaround

[joshgoebel]

For anyone else running into this it's work nothing that walkDir seems to work just fine (from the VM) and let you iterate over the files in a given folder... so this solves my problem. I guess the big difference is it doesn't support POSIX globbing of the input string?

[haltcase]

@yyyc514 walkDir unfortunately does not work on Windows in the VM. Not sure if there's an issue for it but it has come up before. Linked message calls out walkFiles but both iterators use the same implementation (walkCommon - more discussion in IRC).

[joshgoebel]

Grrrr, guess I'll just forget about windows for now and revisit this later. :)

[dom96]

FFI at compile-time is hard. Araq attempted it using libffi and gave up multiple times. If you're up for the challenge then by all means go for it, it would be an awesome feature (it would enable a proper REPL as well).

Btw please search for related issues, I'm pretty sure there are plenty...

[krux02]

Regular expressions are such a versatile tool that it is worth discussing if it should be a compiler vm feature. Then there is still no FFI but fast regular expressions at compile time.

As dom96 pointed out, this problem is hard. Please try to come up with a solution if your really want this issue to be fixed.

Because this issue will be opened again anyway, I will keep this issue open, but there is no active work to support calling into C function pointers from nimvm. I would like to mention though that Araq is working on a compiler plugin system that is not based on nimvm, here this problem would not raise again.

[timotheecour]

@dom96

Btw please search for related issues, I'm pretty sure there are plenty...

I did, before submitting this issue (as I always do btw, although mistakes happen), and no github issue tracks enabling FFI at CT. Eg of searched keywords: libffi and importc. The only relevant thing is a 1 line comment by @zah in this closed (unrelated) issue #61 :

Integration of libffi in the evaluation engine that will allow calling arbitrary C functions

the only other thing that relates is a forum post: https://forum.nim-lang.org/t/4249#26456 (Why can't the Nim VM use FFI?) but I want to track this in gitbhub otherwise it'll never materialize.

@Araq said:

I wrote the code to do that, twice. It always "kinda works" but the devil is in the details
The VM used to support the FFI (via libffi) and I can dig out this code again,

I would love if @Araq could share a pointer to these attempts, to help with future work on this

Consider this problem: void a(int* x; int L); Does the a write into x?

I can see a number of practical ways to improve status quo:

• enable FFI at CT for the (limited yet still useful) cases where there's no pointers in input params (or pointer to const only)
• use pragma annotations to specify whether variables are readonly (yes, that assumes we understand the C function contract)

[dom96]

Regular expressions are such a versatile tool that it is worth discussing if it should be a compiler vm feature. Then there is still no FFI but fast regular expressions at compile time.

A much better solution is to implement a regex engine in pure Nim, and that is what the regex package already does pretty well. We just need to adopt it into the stdlib.

[Clyybber]

@krux02 @dom96 How can regexes replace the functionality of a compile time FFI?
EDIT: Sry, misunderstood @krux02 's proposal

[dom96]

@krux02 @dom96 How can regexes replace the functionality of a compile time FFI?

They can't. That is not at all what we're suggesting.

[timotheecour]

=> PR #10150