Syntax Ambiguities & Parsing Complexity - Moving Forward

[joe-conigliaro]

I have started this discussion as there are whispers of a compiler rewrite some time in the near future.
There are syntax issues which I feel need to be addressed before a rewrite occurs. They need to be addressed either way.
Please mention any ambiguities or syntax issues you know of which are not listed here so I can add them.

Attributes

Attributes work fine just about everywhere, but there are some cases where there are ambiguities.
The simplest fix for the issues below would be to change the attributes syntax, for example to: @[attribute_a], which would allow the attribute syntax to remain the same everywhere without any special cases.
The current suggested syntax change is:

@[attribute_a: value]
struct MyStruct{
	func_a fn() ! @[attribute_a]
	field_a string = 'foo' @[attribute_a: value]
	field_b int = 111 @[attribute_a; attribute_b]
	field_c int = field_c_default() @[attribute_a: value]
}

OR

@attribute_a(value)
struct MyStruct{
	func_a fn() ! @attribute_a
	field_a string = 'foo' @attribute_a(value)
	field_b int = 111 @attribute_a @attribute_b
	field_c int = field_c_default() @attribute_a(value)
}

This change requires removing the @keyword feature, or changing it's syntax, for example to #keyword.

Struct Declaration field with fn type

struct StructA {
         // this is a problem
	func_a fn() ! [attribute_a]
	// this is fine, but still not simple to parse
	func_b fn() ![len_a]u8
}

You can see that func_a fn() ! [attribute_a] looks like the start of a fixed array result type. This can be parsed, but it adds a level of complexity which is not justified considering this issue can be removed completely.

Struct Declaration field with default value

struct StructA {
	field_a string = 'foo' [attribute_a]
	field_b int = 111 [attribute_a; attribute_b]
	field_c int = field_c_default() [attribute_a]
}

The cases above could be index expressions, or look like the start of one. There are ways to parse this and determine what is what, however its not ideal at all. Once again the complexity needed is not justified when this issue can be removed completely.

Struct Initialization in Condition Expression

if StructA{} == StructA{} { ... }
if err == IError{Eof{}} { ... }

These are a little tricky to parse, mostly if StructA{} == StructA{} {, I'm not sure what the solution for this could be. We may just need to deal with doing some trickery. I have listed it here anyway.

Compile time If

This may ruffle some feathers.
It is the least important issue (if you can call it that) I have simply noted it here because it adds some unneeded parsing complexity, and possibly goes against syntax unification and separation of concerns.

I would like you to consider this statement/rule:

• An expression prefixed with $ becomes compile time, for example a call $some_fn(). This is nice and simple.

Assuming the statement above is true, it doesn't make sense for $ to find its way inside of the if expression itself and becomes a part of its branches. It seems out of the scope of the If expression (on a syntax level). Consider the simplicity of this AST:

comptime_if := ComptimeExpr{Expr(IfExpr{...})}
comptime_call := ComptimeExpr{Expr(CallExpr{...})}

This is nit-picking on my behalf, however there is clarity to it.

So what I'm suggesting is to go from:

	$if linux {
		println('linux')
	}
	$else $if windows {
		println('windows')
	}
	$else {
		println('other')
	}

to this:

	$if linux {
		println('linux')
	}
	else if windows {
		println('windows')
	}
	else {
		println('other')
	}

Index Expression vs Generic Call

It may seem like a difficult problem to determine the difference between an index expression and a generic call in some cases:

foo.array_of_fns_or_generic_fn[type_or_index]()
array_of_fns_or_generic_fn[type_or_index](args)

In actual fact the complexity to achieve this is very low. Parsing is simple, cases which can be determined in the parser will be, anything else will be determined after type checking. Since these are basic core features and the complexity of handling them is so low, I don't see any need to reconsider the syntax.

<< Operator - Array push vs Left shift

There is ambiguity as this operator has two uses, however the ambiguity only exists on a visual level, not a logical one. The complexity to parse and then later determine what operation we are doing is very low, although it must be done after type checking. We simply check the type of the left hand side expression, if we have an array then we know it's a push operation.

[JalonSolov]

How about just

$ {
	if linux {
		println('linux')
	}
	else if windows {
		println('windows')
	}
	else {
		println('other')
	}
}

Shorter syntax in most cases (though slightly longer for single $if or $for).

[joe-conigliaro]

How about just

$ {
	if linux {
		println('linux')
	}
	else if windows {
		println('windows')
	}
	else {
		println('other')
	}
}

Shorter syntax in most cases (though slightly longer for single $if or $for).

Thanks for your comments.

That is another option, however it would be nice not to introduce any special cases, the syntax should be the same everywhere, unless all comptime expressions are ${expression}, for example ${some_fn()}.

This issue is not so important compared with the other issues. However there is reasoning behind it.

[JalonSolov]

That's what I meant... ${expression} would be comptime code all the time... unless it's inside quotes, in which case it is string interpolation. That would be a new ambiguity, though.

However, it also allows for adding new comptime statements without needing a special syntax for them.

[joe-conigliaro]

Ahh yes I didnt read it properly :).

$expr/stmt or ${expr/stmt} will work just as easily to make any expression or statement comptime, so its just a matter of syntax preference. Neither one is easier to accomplish.

If there was such thing as arbitrary comptime blocks then${} would make more sense, but I cant think of a purpose for that currently.

[JalonSolov]

It has been requested more than once, but it is up to you/Alex whether or not it gets implemented. :-)

[nbrugger-tgm]

This wouldn't work as far as i can see. How would you do this

$if linux {
     if file_exists("/dev/swap") {
         print("ready");
     }
} $else {
    print("ready");
} 

that could not be done then anymore since ${} would convert the inner if to a compile time if even though it is a runtime if

[JalonSolov]

For arrays, I'd like to see #17514 :-)

[joe-conigliaro]

Thanks @JalonSolov I will take a look

[JalonSolov]

And https://discord.com/channels/592103645835821068/592106336838352923/1084149368313090079 to get rid of _default.

[joe-conigliaro]

I will take a look at this also, thanks for your input @JalonSolov ✊

[i582]

Thanks for starting this topic, I added a couple of cents from myself, maybe I'll remember something else later.

Ambiguity and other problems

Generics

Generics and array access

struct Foo {
	funcs []fn ()
}

foo := Foo{
	funcs: [fn() {
		println('hello')
	}]
}
index := 0
foo.funcs[index]()
//       ^^^^^^^^^ array access or method call?

and

import datatypes

struct Foo {}

fn (f Foo) method[T]() {
	println('method with ${T.name}')
}

foo := Foo{}
foo.method[datatypes.Stack[int]]()
//        ^^^^^^^^^^^^^^^^^^^^^^^^ array access or method call?

To solve this problem, the parser must be case-sensitive and assume that an identifier with a small
letter cannot be a type, so this is array element access.

However, if you use a fully qualified module name for a type, then the parser needs to look 2 tokens
ahead to make sure it is a type (or not, if it is a constant reference).

Generics and field attributes

struct Boo[T] {}

struct Foo[T] {
	boo Boo[T]
	//  ^^^^^^ generic Boo

	boo2 Boo [T]
	//   ^^^ ^^^ attribute
	//   just Boo
}

foo := Foo[int]{}

The @[] syntax for attributes will resolve this space dependency.

JSON decode

f := json.decode(Foo, j) or { panic(err) }
//               ^^^ type

The presence of special functions, where the type is a parameter, greatly complicates parsing, and
taking into account the presence of generic types, it also introduces inconsistency into the
language.

The json2 module should solve this problem, although I would prefer the current module to work
with generic types as well.
It is likely that vfmt will be able to automatically convert the code to the new style.

<< operator

Now the << operator is used to denote a left bit shift and as
operator for adding a new element to an array.

My experience of writing a BNF parser shows that even though these two cases can be distinguished,
bitwise shift can only be in expressions, and adding an element to an array only as a separate
statement, but this adds ambiguity to the language.

I would prefer this operator to be used only for bit shifting.

[joe-conigliaro]

Thanks for your input @i582 I will have a good read over this.

I have solved these cases without using case sensitivity foo.funcs[index]() and foo.method[datatypes.Stack[int]]() In my solution the parser will determine every case it possibly can, and the ones it cannot are determined at a later stage after type checking. The complexity to parse this and then determine what we have later is very low.

I think this is a perfectly acceptable solution for this case, since these are base syntax / language features (Call & Index). I don't think its worth adding any special syntax for such a basic feature. This is just my initial thoughts.

[sparkylein]

Methods in structs look like this:

fn (my myType) hello () { method body...}

so, in case Generics why not do:

fn [T] hello () { method body...}

Applying this to the examples in the documentation:

struct [T] Repo {
    db DB
}

and for a method:

fn [T] new_repo(db DB) [T]Repo {
    return [T]Repo{db: db}
}

BTW: this:

foo.funcs[index]()
//       ^^^^^^^^^ array access or method call?

could also be a call of a function stored in a map (stored as voidptr).

[programandala-net]

Petr Makhnev escribió/skribis/wrote/scrit (2023-03-19T22:20:13-0700):

Now the `<<` operator is used to denote a left bit shift and as operator for adding a new element to an array. I would prefer this operator to be used only for bit shifting.

+1 That unrelated second usage of `<<` was suprising when I first saw it. Reusing, say, the `+` operator as a statement would have a similar effect on regularity and legibility.

…

-- Marcos Cruz http://programandala.net

[medvednikov]

I would just ban if StructA{} == StructA{} {. This is bad code, no reason to allow this.

[] for attribute have to go. @attribute and @attribute(value) are great. Similar to Python. Would have to remove @keyword, but I don't think there's any need in that. Will make the language a bit smaller, which is nice.

Capital letters for types is fine and should stay.

$if {} else {} looks a bit cleaner, so I'm fine with it.

[medvednikov]

@danieldaeschle what's the difference between say @app.route("/") def hello_world(): and V's way to handle routes via attributes?

[danieldaeschle]

The way it works. The syntax might be similar but app.route is a method on the App class. Decorators are just functions, that's why they have a function syntax with an @ before.

Attributes (or in Java they're called annotations) are handled differently (through reflection or at compile-time).

[spytheman]

vfmt already forces each attribute on a newline, so it won't be an issue

no, it merges them instead:

[medvednikov]

right, it does the opposite
I always preferred having them on different lines

it uses separate lines only if the attribute has arguments

[JalonSolov]

It sorts them if they were already separate...

$ v fmt -diff foo.v
--- /home/jalon/foo.v   2023-03-21 12:43:17.618153690 -0400
+++ /tmp/v_1000/vfmt_01GW2GVK1B2RMDF042H2RWN22M_foo.v   2023-03-21 12:43:23.048074451 -0400
@@ -1,3 +1,3 @@
-[unsafe]
 [export: 'cname']
+[unsafe]
 fn foo() {}
[jalon@7950x ~]$ 

but indeed splits them if there is an argument

[jalon@7950x ~]$ v fmt -diff foo.v
--- /home/jalon/foo.v   2023-03-21 12:44:23.507192106 -0400
+++ /tmp/v_1000/vfmt_01GW2GXFD1F0R4GKQMY47KG07A_foo.v   2023-03-21 12:44:24.860505687 -0400
@@ -1,2 +1,3 @@
-[export: 'cname'; unsafe]
+[export: 'cname']
+[unsafe]
 fn foo() {}
[jalon@7950x ~]$ 

which is really inconsitent... it combines them if they don't have arguments, but splits them if they do?

And... it only puts the ones with arguments on their own line. The others it leaves together

[jalon@7950x ~]$ v fmt -diff foo.v
--- /home/jalon/foo.v   2023-03-21 12:45:37.886106475 -0400
+++ /tmp/v_1000/vfmt_01GW2GZR0SDW8KAKVBYYEHHD9E_foo.v   2023-03-21 12:45:39.222753631 -0400
@@ -1,2 +1,3 @@
-[export: 'cname'; unsafe; direct_array_access]
+[export: 'cname']
+[direct_array_access; unsafe]
 fn foo() {}
[jalon@7950x ~]$ 

[medvednikov]

json.decode(Foo, j) will be replaced with json.decode[Foo](j)

<< will stay. It's a very frequently used operation, much more frequent than say %. So makes sense to have an operator for it. Using += would also be ambiguous.

[Yaisis]

It is possible as in dlang:

s = s ~ a
s ~= a

With a special operator, you can concatenate multiple arrays into a single expression:

s = s1 ~ s2 ~ s3

[medvednikov]

With the new attributes we can have something like

@attribute
struct Route {
    path       string
    method HttpMethod
}

@Route{'/index', .get}
fn index() {
}

[i582]

Or with parens:

@[attribute]
struct Route {
    path       string
    method HttpMethod
}

@[Route{'/index', .get}]
fn index() {
}

[joe-conigliaro]

Indeed this way would work perfectly also

[danieldaeschle]

This is the C# way and would help.with comptime. You don't have to parse a string.

[JalonSolov]

Question: Why add the [] if @ denotes an attribute?

The attributes go to end of line, whether they are function attributes

@Route{'/index/, .get}
fn index() {
}

struct attributes

@heap
struct Foo {
}

field attributes

struct Bar {
    my_field string @json: 'myField'
}

or whatever. There's no need for the extra [] to differentiate the attributes from the rest.

[JalonSolov]

Yes, this could "collide" with using @reserved name, but... I'd rather see that go away and use _ instead of @, leaving @ just for attributes.

[JalonSolov]

Another point of note: If we are going to have aliases at all, they should always act like actual aliases. We have had too many problems where you couldn't use an operator or a method on an alias that was available on the original.

This should never be the case. An alias should be exactly that... an alternative name for the same thing as the original. So all operators, all methods, etc., etc. should work exactly the same on the alias as it does on the original.

The only difference between an alias and it's original besides the name that is typed (obviously) is any extra methods added to the alias would only be available on the alias, not the original.

[danieldaeschle]

If we are going to have aliases at al

What do you mean by that? Is it under consideration to remove them?

[JalonSolov]

No, it is that they aren't implemented as I described now, or we wouldn't keep getting reports about something not working on an alias that does on the original thing.

[JalonSolov]

An alternative would be to completely remove aliases, but I'm not recommending that. I'm recommending making them work the way they should. At the moment, it is ambiguous in that they should work exactly like the original, but they don't.

[danieldaeschle]

I understand

[JalonSolov]

This is an example of what I was talking about: #17818

His BOOL alias should act exactly like bool, so it should be usable the same way as well.

[JalonSolov]

A syntax ambiguity in strings... you can get the runes out of a string (.runes()) method, but you can't add a rune to a string without converting the rune to a string, first. In other words:

s += ` `

fails, while

s += ` `.str()

works. I think the first way should work, as well, and avoid the extra string conversion.