Go support

[meme]

Is there any planned support for Go?

Since it is quite similar syntactically to C, I have just modified all that is necessary to use the simple example from Go to illustrate what I envision:

/*!re2c re2c:flags:i = 1; */

/*!max:re2c*/
/*!re2c
    digit  = [0-9];
    number = digit+;
*/

package main

type Cursor struct {
	data []byte
	state int
}

type YYCTYPE byte
type YYMARKERTYPE *Cursor

func YYPEEK(marker YYMARKERTYPE) YYCTYPE {
	return YYCTYPE(marker.data[marker.state])
}

func YYSKIP(marker YYMARKERTYPE) {
	marker.state++
}

func Lex(YYCURSOR YYMARKERTYPE) int {
	/*!re2c
	re2c:define:YYCTYPE = YYCTYPE;
	re2c:yyfill:enable = 0;

	* { return 1 }

	number {
		fmt.Println("number\n")
		return 0
	}
	*/
}

func main() {
	state := &Cursor{
		data:  []byte("1024 ;]"),
		state: 0,
	}

	fmt.Println(Lex(state))
	fmt.Println(Lex(state))
}

re2c --input custom -W example.r2

/* Generated by re2c 1.3 on Sun Mar 22 12:43:56 2020 */

package main

import "fmt"

type Cursor struct {
	data  []byte
	state int
}

type YYCTYPE byte
type YYMARKERTYPE *Cursor

func YYPEEK(marker YYMARKERTYPE) YYCTYPE {
	return YYCTYPE(marker.data[marker.state])
}

func YYSKIP(marker YYMARKERTYPE) {
	marker.state++
}

func Lex(YYCURSOR YYMARKERTYPE) int {

	{
		var yych YYCTYPE
		yych = YYPEEK(YYCURSOR)
		switch yych {
		case '0':
			fallthrough
		case '1':
			fallthrough
		case '2':
			fallthrough
		case '3':
			fallthrough
		case '4':
			fallthrough
		case '5':
			fallthrough
		case '6':
			fallthrough
		case '7':
			fallthrough
		case '8':
			fallthrough
		case '9':
			goto yy4
		default:
			goto yy2
		}
	yy2:
		YYSKIP(YYCURSOR)
		{
			return 1
		}
	yy4:
		YYSKIP(YYCURSOR)
		yych = YYPEEK(YYCURSOR)
		switch yych {
		case '0':
			fallthrough
		case '1':
			fallthrough
		case '2':
			fallthrough
		case '3':
			fallthrough
		case '4':
			fallthrough
		case '5':
			fallthrough
		case '6':
			fallthrough
		case '7':
			fallthrough
		case '8':
			fallthrough
		case '9':
			goto yy4
		default:
			goto yy6
		}
	yy6:
		{
			fmt.Println("number\n")
			return 0
		}
	}

}

func main() {
	state := &Cursor{
		data:  []byte("1024 ;]"),
		state: 0,
	}

	fmt.Println(Lex(state))
	fmt.Println(Lex(state))
}

The patch is as follows (modifications only to what was deemed necessary):

git diff

diff --git a/src/codegen/emit_action.cc b/src/codegen/emit_action.cc
index 3d51707b..be5ec626 100644
--- a/src/codegen/emit_action.cc
+++ b/src/codegen/emit_action.cc
@@ -327,7 +327,7 @@ void gen_goto(code_lines_t &code, const State *from, const State *to
 
     if (to) {
         code.push_back("goto " + opts->labelPrefix
-            + to_string(to->label) + ";\n");
+            + to_string(to->label) + "\n");
     }
 }
 
diff --git a/src/codegen/emit_dfa.cc b/src/codegen/emit_dfa.cc
index b829b534..e41a2b74 100644
--- a/src/codegen/emit_dfa.cc
+++ b/src/codegen/emit_dfa.cc
@@ -229,7 +229,7 @@ void DFA::emit(Output & output, uint32_t& ind, bool isLastCond, bool& bPrologBra
             {
                 if (opts->bEmitYYCh)
                 {
-                    output.wind(ind).wstring(opts->yyctype).ws(" ").wstring(opts->yych).ws(";\n");
+                    output.wind(ind).ws("var ").wstring(opts->yych).ws(" ").wstring(opts->yyctype).ws("\n");
                 }
                 output.wdelay_yyaccept_init (ind);
             }
diff --git a/src/codegen/go_emit.cc b/src/codegen/go_emit.cc
index c608b506..88f5792c 100644
--- a/src/codegen/go_emit.cc
+++ b/src/codegen/go_emit.cc
@@ -61,7 +61,7 @@ void Case::emit (Output & o, uint32_t ind) const
             bool last_case = i == ranges.size () - 1 && b == ranges[i].second - 1;
             if (!last_case)
             {
-                o.ws("\n");
+                o.ws(" fallthrough\n");
             }
         }
     }
@@ -69,7 +69,7 @@ void Case::emit (Output & o, uint32_t ind) const
 
 void Cases::emit(Output &o, uint32_t ind, const DFA &dfa, const State *from) const
 {
-    o.wind(ind).ws("switch (").wstring(o.block().opts->yych).ws(") {\n");
+    o.wind(ind).ws("switch ").wstring(o.block().opts->yych).ws(" {\n");
 
     for (uint32_t i = 1; i < cases_size; ++i) {
         const Case &c = cases[i];
diff --git a/src/codegen/input_api.cc b/src/codegen/input_api.cc
index 6760711d..a4ae5ce9 100644
--- a/src/codegen/input_api.cc
+++ b/src/codegen/input_api.cc
@@ -12,7 +12,7 @@ std::string output_expr_peek(const opt_t *opts)
 {
     return opts->input_api == INPUT_DEFAULT
         ? "*" + opts->yycursor
-        : opts->yypeek + " ()";
+        : opts->yypeek + "(" + opts->yycursor + ")";
 }
 
 std::string output_restore(uint32_t ind, const opt_t *opts)
@@ -47,7 +47,7 @@ void output_peek(std::ostream &o, uint32_t ind, const opt_t *opts)
 {
     o << indent(ind, opts->indString) << opts->yych << " = " << yych_conv(opts);
     if (opts->input_api == INPUT_CUSTOM) {
-        o << opts->yypeek << " ()";
+        o << opts->yypeek << "(" + opts->yycursor + ")";
     } else {
         o << "*" << opts->yycursor;
     }
@@ -58,7 +58,7 @@ void output_skip(std::ostream &o, uint32_t ind, const opt_t *opts)
 {
     o << indent(ind, opts->indString);
     if (opts->input_api == INPUT_CUSTOM) {
-        o << opts->yyskip << " ()";
+        o << opts->yyskip << "(" + opts->yycursor + ")";
     } else {
         o << "++" << opts->yycursor;
     }

---

I am wondering if you would be open to further work on the Go output, to be used with the -go compiler flag, for example.

[skvadrik]

I'm certainly interested in adding Go support (although no work in that direction has been done yet). Thanks for the patch, I will have a closer look in a few days.

[skvadrik]

For this example, it is almost possible to use re2c in its current form.

The only necessary modification is the addition of fallthrough. The changes to YYPEEK and YYSKIP are not needed because they can be closures (see the code below). The change to yych is not needed because re2c has a configuration re2c:yych:emit = 0;. The modified example looks like this:

/*!re2c re2c:flags:i = 1; */

package main

import "fmt"

type YYCTYPE byte
type Cursor struct {
        data  []YYCTYPE
        offset int
}

func peek(cursor *Cursor) func() YYCTYPE {
        c := cursor
        return func() YYCTYPE {
                return c.data[c.offset]
        }
}

func skip(cursor *Cursor) func() {
        c := cursor
        return func() {
                c.offset++
        }
}

func Lex(YYCURSOR *Cursor) int {
        YYPEEK := peek(YYCURSOR)
        YYSKIP := skip(YYCURSOR)
        var yych YYCTYPE

        /*!re2c
        re2c:yych:emit = 0;
        re2c:yyfill:enable = 0;

        digit  = [0-9];
        number = digit+;

        * { return 1 }

        number {
                fmt.Println("number\n")
                return 0
        }
        */
}

func main() {
        cursor := &Cursor{
                data:  []YYCTYPE("1024 ;]"),
                offset: 0,
        }

        fmt.Println(Lex(cursor))
        fmt.Println(Lex(cursor))
}

Run like this (the invocation of sed adds fallthrough):

re2c 1.re -W -o 1.go --input custom && \
    sed -i -E "s/^(\tcase '[0-9A-F]+':)$/\1\tfallthrough/g" 1.go && \
    go build 1.go && \
    ./1
number

0
1

[skvadrik]

More generally, the difficulty with adding Go (or any other language) support to re2c is in anticipating all possible scenarios in which re2c can be used. The example above only uses a small subset of re2c features (e.g. it's not using YYFILL). So I need to experiment with other features.

But I'm surprised that so little change is needed to make this example work. :)

[meme]

What are the next steps, does the current set of examples cover the feature set of re2c? If so, would it make sense to translate them all to Go and use those as a test bench for the Go compiler back-end?

Further, it seems that the codegen subsystem of the project is not factored for multiple targets beyond both TARGET_CODE and TARGET_DOT, so I ask: do you foresee the addition of more targets to re2c? Then it might be worth while organizing that to facilitate more back-ends and working on the Go back-end using that refactored system.

Lastly, there's one more idea I want to pitch: ideally, I'd like to see Go projects that use re2c use the go generate functionality, something like:

//go:generate re2c -o lexer.go lexer.re

It would still require re2c to be installed, but I think it has the opportunity to fit really nicely into the Go ecosystem.

[skvadrik]

What are the next steps, does the current set of examples cover the feature set of re2c? If so, would it make sense to translate them all to Go and use those as a test bench for the Go compiler back-end?

Yes, something like that. In many cases re2c has a few alternative ways of doing things, like using default or generic API, using or not using EOF rule, etc. For a start, it suffices to choose one that is easier to adapt, and the alternative options should be be forbidden for the Go backend.

Further, it seems that the codegen subsystem of the project is not factored for multiple targets beyond both TARGET_CODE and TARGET_DOT, so I ask: do you foresee the addition of more targets to re2c?

Yes, it's been on my wish list for a while. I did some preliminary work, but the codegen subsystem is still in a mess. It needs restructuring to separate the construction of a C program from its rendering (conversion to text). The current representation is a list of program fragments, some of which are rendered immediately, and some are placeholders for data that is accumulated over the whole program and can only be rendered at the end of codegen phase. Restructuring the code is not a strict prerequisite for Go support: it can be done in a step by step manner, starting with some rudimentary support and gradually making it better. As long as it's not released, it's not part of the API and can be changed.

ideally, I'd like to see Go projects that use re2c use the go generate functionality

Sounds good. Does Go use compiler plugins for 3rd party tools like re2c?

[skvadrik]

I started an experimental branch golang-backend and added the above example (slightly modified) as a test: b7facb2.

[meme]

1. Agreed, but I'll have to leave that to you because I am much less comfortable with the more advanced features of re2c
2. I also think it is worth just hacking it in and then gradually making it better
3. Apologies if I was not clear, go generate runs the argument as a command during go build, so no modification to Go is required, you simply need re2c in your path, and then anyone that go gets or go builds the project will be good to go

Experimental branch looks good. What can I do to help, or are you planning on implementing the necessary changes for the feature request?

[skvadrik]

Agreed, but I'll have to leave that to you because I am much less comfortable with the more advanced features of re2c

Sure, no problem with that. I'm interested in doing this. (I just try to keep the discussion in the open. )

Apologies if I was not clear, go generate runs the argument as a command during go build [...]

Ah, that's how it works. I think it would make sense to have a tool re2go (which will be just re2c with Go backend enabled by default).

What can I do to help, or are you planning on implementing the necessary changes for the feature request?

I am. You can help by trying to use the golang-backend branch on your real-world example and commenting on what's missing. But! In case you wanted to do a bit of re2c hacking and implement Go support yourself (at least in its basic form), I can step aside and review your pull requests. Just let me know.

[meme]

Sounds good to me! I'll opt to help by using the Go branch in the project I had intended to use re2c for. Thanks for your interest in the request.

[meme]

Now that Go support has landed in master, should this be closed?

[skvadrik]

@meme I merged the experimental branch because it contained useful work that should be on master, and because I didn't want it to diverge too far. But i didn't finish working on Go support yet. I have done the most difficult preliminary work (full restructuring of the code generation subsystem), but a lot of follow-up work still remains. There are only a few Go tests yet and no documentation. Lets keep the bug open for now.

[skvadrik]

Go support is fully working in master now. I will release 2.0 soon and update documentation on the website.

[meme]

Incredible! Thank you for your work. Will take a look at Go integration, please let me know if the docs need attention w.r.t. using //go:generate & other Go-isms

[skvadrik]

@meme Thanks! The Go examples are here if you'd like to take a look. The source files have .re extension and the output files have .go extension, which is currently a requirement of the re2c testing script. But with a trivial renaming and patching of the go:generate comment they can be processed with go generate and go test.