[Nim] Nim implementation #7362

danlapid · 2022-06-22T14:45:13Z

So this is a mostly working implementation for flat buffers in nim.
Based on the existing languages (mostly - swift, python, lua) and on https://github.com/RecruitMain707/NimFlatbuffers
Relevant issue: #5855

This is incomplete, I would like to add:

more tests
one file support
object api
Documentation

It's actually my first PR so I am opening it even before it's finished just to make sure what I'm doing is appropriate and okay with everyone.
If I'm missing anything please tell me.
Thanks!

google-cla · 2022-06-22T14:45:18Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

CasperN · 2022-06-23T19:05:39Z

Wow, thanks for the huge effort!

CasperN

Hi, I only put in a few comments so we can start iterating, but I will continue reviewing/adding comments

CasperN · 2022-06-23T19:21:14Z

src/idl_gen_nim.cpp

+const char *SelfDataPos = "self.tab.Pos";
+const char *SelfDataBytes = "self.tab.Bytes";
+
+class NimGenerator : public BaseGenerator {


Can you build NimGenerator off of the BfbsGenerator? We're trying to reduce technical debt which in large part is our dependence on the hard-to-understand-god-object that is our "Parser". Its similar but the type system has been run through our intermediate format.

Would that be like the bfbs_gen_lua?
I don’t mind switching over,
Do I need to have both or just transfer over to bfbs and remove the old code once it’s done?

Thanks! Just transfer over to bfbs and remove the old code once its done

CasperN · 2022-06-23T19:23:21Z

src/idl_gen_nim.cpp

+}
+
+Namer::Config NimDefaultConfig() {
+  return { /*types=*/Case::kKeep,


Please don't use Case::kKeep and use whatever is in the nim style guide.

I introduced it to help transition code generators that did not handle casing to Namer while also not affecting gencode.

CasperN · 2022-06-23T19:26:44Z

src/idl_gen_nim.cpp

+  }
+
+  // Begin a class declaration.
+  void BeginClass(const StructDef &struct_def, std::string *code_ptr) {


please just pass in std::string& code here and elsewhere.

The "out parameters must be pointers" thing was an old rule in the Google style guide that has since been overturned.

CasperN · 2022-06-24T13:41:46Z

nim/flatbuffers/src/builder.nim

+   result.bytes.setLen(size)
+   result.minalign = 1
+   result.head = size.uoffset
+   #result.vtables.setLen(16)# = newSeq[uoffset](16)


nit: commented out code

CasperN · 2022-06-24T13:44:31Z

src/idl_gen_nim.cpp

+           /*variable=*/Case::kLowerCamel,
+           /*variants=*/Case::kKeep,
+           /*enum_variant_seperator=*/".",
+           /*escape_keywords=*/Namer::Config::Escape::BeforeConvertingCase,


You probably want to escape keywords after converting case

CasperN · 2022-06-24T13:44:54Z

src/idl_gen_nim.cpp

+           /*variants=*/Case::kKeep,
+           /*enum_variant_seperator=*/".",
+           /*escape_keywords=*/Namer::Config::Escape::BeforeConvertingCase,
+           /*namespaces=*/Case::kKeep,  // Packages in python.


this is not python

CasperN · 2022-06-24T13:51:24Z

src/idl_gen_nim.cpp

+const char *Export = "*";
+const char *SelfData = "self.tab";
+const char *SelfDataPos = "self.tab.Pos";
+const char *SelfDataBytes = "self.tab.Bytes";


nit: I'm not sure if it's helpful to have these constants, like why not inline all of these?

CasperN · 2022-06-24T13:53:38Z

tests/FlatBuffers.Test.Nim/tests/optional_scalars/test.nim

+
+    # Creates a flatbuffer with optional values.
+    check(optionals.justI8 == 0)
+    # check(optionals.maybeF32() is None)


these probably shouldn't be commented out

CasperN · 2022-06-24T13:54:57Z

tests/FlatBuffers.Test.Nim/tests/mygame/test.nim

+  check(monster.pos.test3.a == 5)
+  check(monster.pos.test3.b == 6)
+  check(monster.testType == Any.Monster.uint8)
+  # let monster2 = Monster(tab: monster.test.tab)


nit: commented out code

CasperN · 2022-06-24T13:55:16Z

tests/FlatBuffers.Test.Nim/tests/mygame/test.nim

+        121, 0, 0, 5, 0, 0, 0, 70, 114, 111, 100, 111, 0, 0, 0])
+
+
+    # var test_mutating_bool: TestMutatingBool


nit: commented out code

CasperN · 2022-06-24T13:55:57Z

tests/FlatBuffers.Test.Nim/NimTest.sh

+rm -r ${nim_dir}/generated
+rm -r ${nim_dir}/tests/*/test
+
+# nim r tests/test_mutating_bool.nim


nit: commented out code (I'm guessing its because these features aren't done yet?)

danlapid · 2022-06-24T14:08:19Z

So I’m almost finished with object api
After that I’ll transfer everything to bfbs during which I will clean up the whole code, remove comments and weird code bits.
I will ping you once I’m done so as not to continue wasting your time meanwhile :)
@ @CasperN

…s which are not supported in the language

danlapid · 2022-06-25T08:49:04Z

@CasperN
So, I've come up a little short, it seems there's no support for circular dependencies in the language
Examples like:

union Any { M1: Monster, M2: Monster, M3: Monster }

table Monster {
  any:Any;
}

When implementing the ObjectAPI then AnyT would refer to MonsterT and MonsterT would refer to AnyT

A possible solution I've found would be to force onefile
If the code is declared in a single file then I can declare all types under single "type" statement which works for odd reasons:

type
  Any* {.pure.} = enum
    NONE = 0.uint8,
    M1 = 1.uint8,
    M2 = 2.uint8,
    M3 = 3.uint8,

  AnyT* {.union.} = ref object
    asM1*: MonsterT
    asM2*: MonsterT
    asM3*: MonsterT

  MonsterT* = ref object
    anyType*: Any
    any*: Option[AnyT]

  Monster* = object of FlatObj

Problem is, then we'll have to implement namespaces as underscores making MyGame.Example.Monster -> MyGame_Example_Monster for every usage of it.
Otherwise we'll possibly have circular dependencies between the "one file"s of every namespace

I'd love to know your opinion.
Thanks

danlapid · 2022-06-25T13:48:26Z

Alternatively I could implement "namespaces" in one file with backticks thus the on file will look like:

type
  `Namespace1.Any`* {.pure.} = enum
    NONE = 0.uint8,
    M1 = 1.uint8,
    M2 = 2.uint8,
    M3 = 3.uint8,

  `Namespace1.AnyT`* {.union.} = ref object
    asM1*: `Namespace2.MonsterT`
    asM2*: `Namespace2.MonsterT`
    asM3*: `Namespace2.MonsterT`


  `Namespace2.MonsterT`* = ref object
    anyType*: `Namespace1.Any`
    any*: Option[`Namespace1.AnyT`]

  `Namespace2.Monster`* = object of FlatObj

And the usage will look like:

import RecursiveImports_generated
var fbb = newBuilder(0)
fbb.MonsterStart()
let root = fbb.MonsterEnd()
fbb.Finish(root)

var defaults: `Namespace2.Monster`
defaults.GetRootAs(fbb.FinishedBytes(), 0)
var defaultst: `Namespace2.MonsterT`
defaultst.InitFromBuf(fbb.FinishedBytes(), 0)

Again, not sure what to think as to what's better
Just to clarify, all of this is relevant only to the object_based_api which causes all these recursive imports
We could allow the multiple files when there's no object_based_api and only force one file for object_based_api

CasperN · 2022-06-30T14:29:58Z

Hm, I am not a nim expert so I cant say what is most ergonomic for users of the language.

Does nim support tricks like defining everything in one file then importing it into a namespace hierarchy that reexports everything?

It's interesting that only the object API forces these recursive imports, the normal API definitely needs to know the types of all the fields....... maybe its a distinction between returning a type vs having a type as field?

dbaileychess · 2022-08-06T05:15:43Z

@danlapid Just checking in on this work. Would love to have it.

danlapid · 2022-08-06T05:29:21Z

I haven’t had time to work on it lately
As is it works great either without object api or with object api but no recursive references.
Obviously I would love to get it to work flawlessly always but considering nim doesn’t really support type forward declaration nor recursive imports I currently don’t have a good idea on how to get this done.
The leading option would be a single file with all of the implementation and then re-exporting directory tree but it still requires a bit more work I need to finish on another branch.

CasperN · 2022-08-08T14:37:11Z

I would prefer having nim supported with those caveats, no need to get things perfect in the first iteration

dbaileychess · 2022-08-15T16:52:00Z

I am looking forward to this, but will want to postpone any merge until after 2.0.7 is release.

dbaileychess · 2022-09-12T17:42:07Z

@danlapid I hate to be this person, but i think I only want to accept new implementations if they follow the BFBS code generation. See how this is done with the Lua codegenerator.

This probably will make your code easier to write, but since you already spent a lot of time writing idl_nim_gen.cc it may take a while to port over.

danlapid added 3 commits June 21, 2022 18:15

first draft of nim implementation, no tests yet

b0012a3

tests files starting

70bf4da

some tests exist and they are passing

7361285

github-actions bot added c++ CI Continuous Integration codegen Involving generating code from schema swift labels Jun 22, 2022

danlapid added 8 commits June 22, 2022 17:49

buildifier

14a9f35

more tests

0ded275

switch to namer

f2f481f

accidentaly deleted file

3802451

switch to testament

4bc6574

tests working with testament as expected

6c93df6

fully qualified names for imports

73c07ff

different union casting option

e182dc3

CasperN reviewed Jun 23, 2022

View reviewed changes

CasperN reviewed Jun 24, 2022

View reviewed changes

danlapid added 2 commits June 24, 2022 23:58

Object api implemented, no tests are running yet

41b7837

test are mostly passing besides all the glaring recursive dependencie…

5406869

…s which are not supported in the language

danlapid added 2 commits June 29, 2022 23:14

a bit of cleanups

b4148ae

remove comments

072def7

danlapid mentioned this pull request Sep 15, 2022

[Nim] Bfbs Nim Generator #7534

Merged

danlapid closed this Sep 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Nim] Nim implementation #7362

[Nim] Nim implementation #7362

danlapid commented Jun 22, 2022

google-cla bot commented Jun 22, 2022

CasperN commented Jun 23, 2022

CasperN left a comment

CasperN Jun 23, 2022

danlapid Jun 23, 2022

CasperN Jun 24, 2022

CasperN Jun 23, 2022

CasperN Jun 23, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

CasperN Jun 24, 2022

danlapid commented Jun 24, 2022

danlapid commented Jun 25, 2022

danlapid commented Jun 25, 2022

CasperN commented Jun 30, 2022

dbaileychess commented Aug 6, 2022

danlapid commented Aug 6, 2022

CasperN commented Aug 8, 2022

dbaileychess commented Aug 15, 2022

dbaileychess commented Sep 12, 2022

		121, 0, 0, 5, 0, 0, 0, 70, 114, 111, 100, 111, 0, 0, 0])


		# var test_mutating_bool: TestMutatingBool

[Nim] Nim implementation #7362

[Nim] Nim implementation #7362

Conversation

danlapid commented Jun 22, 2022

google-cla bot commented Jun 22, 2022

CasperN commented Jun 23, 2022

CasperN left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danlapid commented Jun 24, 2022

danlapid commented Jun 25, 2022

danlapid commented Jun 25, 2022

CasperN commented Jun 30, 2022

dbaileychess commented Aug 6, 2022

danlapid commented Aug 6, 2022

CasperN commented Aug 8, 2022

dbaileychess commented Aug 15, 2022

dbaileychess commented Sep 12, 2022