[Parser] Implement parsing rules for functions

[akshanshbhatt]

❯ cat examples/test_1.py
def main0():
    x: i32
    x = 5
    print(x)

❯ src/bin/lpython --show-ast --new-parser examples/test_1.py
syntax error: Token 'def' is unexpected here
 --> examples/test_1.py:1:1
  |
1 | def main0():
  | ^^^


Note: if any of the above error or warning messages are not clear or are lacking
context please report it to us (we consider that a bug that needs to be fixed).

The new parser has not (yet) implemented the rules for function headers. I tried implementing it but got build errors -

❯ ./build0.sh
+ ci/version.sh
++ git describe --tags --dirty
+ version=v0.2.0-282-ga63bdc777-dirty
+ version=0.2.0-282-ga63bdc777-dirty
+ echo 0.2.0-282-ga63bdc777-dirty
+ python grammar/asdl_py.py
Assuming default values of Python.asdl and python_ast.py
+ python grammar/asdl_cpp.py grammar/Python.asdl src/lpython/python_ast.h
+ python grammar/asdl_cpp.py src/libasr/ASR.asdl src/libasr/asr.h
+ cd src/lpython/parser
+ re2c -W -b tokenizer.re -o tokenizer.cpp
+ cd src/lpython/parser
+ bison -Wall -d -r all parser.yy
parser.yy: error: shift/reduce conflicts: 1 found, 0 expected

or segmentation faults (in case it did compile) -

❯ ./src/bin/lpython --show-ast --new-parser examples/expr2.py
Traceback (most recent call last):
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 752
    return emit_ast(arg_file, runtime_library_dir, compiler_options);
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 122
    al, runtime_library_dir, infile, diagnostics, compiler_options.new_parser);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 95
    Result<LPython::AST::Module_t*> res = parse(al, input, diagnostics);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 18
    Parser p(al, diagnostics);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 51
    m_tokenizer.set_string(inp);
  File "parser.tab.cc", line 1726
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/python_ast.h", line 40
    LFORTRAN_ASSERT(is_a<T>(*f));
  Binary file "/usr/lib/system/libsystem_platform.dylib", local address: 0x1803414c3
Segfault: Signal SIGSEGV (segmentation fault) received

For now, I have commented out the template code. @Thirumalai-Shaktivel, can you please guide me?

[Thirumalai-Shaktivel]

Yup, Thanks @akshanshbhatt for showing your interest towards implementing parser.

[Thirumalai-Shaktivel]

❯ ./build0.sh
+ ci/version.sh
++ git describe --tags --dirty
+ version=v0.2.0-282-ga63bdc777-dirty
+ version=0.2.0-282-ga63bdc777-dirty
+ echo 0.2.0-282-ga63bdc777-dirty
+ python grammar/asdl_py.py
Assuming default values of Python.asdl and python_ast.py
+ python grammar/asdl_cpp.py grammar/Python.asdl src/lpython/python_ast.h
+ python grammar/asdl_cpp.py src/libasr/ASR.asdl src/libasr/asr.h
+ cd src/lpython/parser
+ re2c -W -b tokenizer.re -o tokenizer.cpp
+ cd src/lpython/parser
+ bison -Wall -d -r all parser.yy
parser.yy: error: shift/reduce conflicts: 1 found, 0 expected

Here at the last line, there is one grammatical conflict!
So, while defining the grammar; make sure there are no conflicts.

[Thirumalai-Shaktivel]

❯ ./src/bin/lpython --show-ast --new-parser examples/expr2.py
Traceback (most recent call last):
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 752
    return emit_ast(arg_file, runtime_library_dir, compiler_options);
  File "/Users/akshansh/Documents/GitHub/lpython/src/bin/lpython.cpp", line 122
    al, runtime_library_dir, infile, diagnostics, compiler_options.new_parser);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 95
    Result<LPython::AST::Module_t*> res = parse(al, input, diagnostics);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 18
    Parser p(al, diagnostics);
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/parser/parser.cpp", line 51
    m_tokenizer.set_string(inp);
  File "parser.tab.cc", line 1726
  File "/Users/akshansh/Documents/GitHub/lpython/src/lpython/python_ast.h", line 40
    LFORTRAN_ASSERT(is_a<T>(*f));
  Binary file "/usr/lib/system/libsystem_platform.dylib", local address: 0x1803414c3
Segfault: Signal SIGSEGV (segmentation fault) received

After defining the grammar, For ex:

function_def
    : KW_DEF id "(" arg_list ")" ":" statements {  }

If you get the above error, then it is a success (now the grammar defining part is complete)! 
Then we need to expose the values to the AST by defining the macros.

[Thirumalai-Shaktivel]

We will Define the grammar step by step. So at first, we will parse

def main():
	print()

Here def is KW_DEF (which is a keyword)
main is TK_NAME ie., id (identifier)
after : there is a new line we can use sep
print() appears in statement
There is an one INDENT and DEDENT
By integrating we get:

function_def
    : KW_DEF id "(" ")" ":" sep TK_INDENT statement TK_DEDENT {  }

[Thirumalai-Shaktivel]

To get the tokens, you can do:

$ lpython --show-tokens examples/expr2.py
(KEYWORD "def") 0:2
(TOKEN "identifier" main) 4:7
(TOKEN "(") 8:8
(TOKEN ")") 9:9
(TOKEN ":") 10:10
(NEWLINE) 11:11
(TOKEN "indent") 12:12
(TOKEN "identifier" print) 13:17
(TOKEN "(") 18:18
(TOKEN ")") 19:19
(NEWLINE) 20:20
(EOF) 21:21

[Thirumalai-Shaktivel]

For motivation please analyze: #349 #383
https://gitlab.com/lfortran/lfortran/-/blob/master/src/lfortran/parser/parser.yy
https://gitlab.com/lfortran/lfortran/-/blob/master/src/lfortran/parser/semantics.h

[Thirumalai-Shaktivel]

Hi @akshanshbhatt, function defintion is a little difficult to parse, why don't you try while or for statements

[Thirumalai-Shaktivel]

Here is the example program for while statement:

i:i32 = 1
while i < 6:
    print(i)
    i += 1

i = 1
while i < 6:
    print(i)
    if i == 3:
        break
    i += 1

i = 1
while i < 6:
    print(i)
    i += 1
else:
    print("i is no longer less than 6")

[Thirumalai-Shaktivel]

Grammar: https://docs.python.org/3/reference/compound_stmts.html#the-while-statement

[akshanshbhatt]

Hi @akshanshbhatt, function defintion is a little difficult to parse, why don't you try while or for statements

Yeah, sure! I'll first try to implement the grammar rules for looping statements. I'll ask you if I face any issues.

[Thirumalai-Shaktivel]

@akshanshbhatt -- let us know if you need any help with this PR.

[akshanshbhatt]

@akshanshbhatt -- let us know if you need any help with this PR.

Sure, @Thirumalai-Shaktivel, I'll open a new PR. I've been a bit occupied lately with my end-semester examinations. I will complete this PR as soon as I get some time.

[Thirumalai-Shaktivel]

Sure, Take your time (also, All the best for your remaining exams)!