cgo: add support for negative numeric constants in tokenizer

[cornelk]

Closes #770

before:

# net
/usr/include/netdb.h:617:23: unexpected token ILLEGAL
/usr/include/netdb.h:618:22: unexpected token ILLEGAL
/usr/include/netdb.h:624:23: unexpected token ILLEGAL

failing lines were like:
# define EAI_NONAME -2 /* NAME or SERVICE is unknown. */

[aykevl]

I think this will cause trouble in the future when expressions are implemented. For example: how should 5 -2 be parsed? Obviously as three tokens (5 - 2) to get an expression but with this change it would be parsed as 5 -2, and two consecutive numbers aren't valid C (like in 5 4).

At the moment, the cgo package doesn't support parsing C arithmetic expressions but it should eventually learn to do this. For example, the Nordic SoftDevice contains code like this for error messages:

#define NRF_ERROR_SDM_BASE_NUM  (0x1000)
#define NRF_ERROR_SDM_INCORRECT_CLENR0                  (NRF_ERROR_SDM_BASE_NUM + 2)

[aykevl]

I should add that your work is certainly appreciated, just that this particular PR may not be the best solution for negative numeric constants.

[typeless]

I did an experiment using @cznic's https://gitlab.com/cznic/cc to parse the #defines. (Where the author uses it to translate the entire SQLite codebase.)
It almost works except that the cc package doesn't have a stddef.h shipped like a normal C compiler.
That can be solved by importing a stddef.h from somewhere else or just creating a dummy one for it.
However, I am not sure if it is the way worth pursuing as it still cannot solve my problem immediately (for the lack of say, symbols defined in the os package).

PS. I wanted to use TinyGo to compile CLI utilities runnin on embeded Linux devices.

[aykevl]

If we're going to use a C parser, it'll be the same we use everywhere: libclang. It seems like a bad idea to me to use two different C parsers for different purposes.

That said, I really think the right way forward is not using a C parser at all but a hand-rolled parser that only parses the most common types of C expressions and convert those expressions to Go expressions, just like we do already: we parse the C string in the #define to a Go AST node and let the Go compiler compile that expression further. Maybe there are cases where that's not enough but I think it will work in the vast majority of #defines.

However, I am not sure if it is the way worth pursuing as it still cannot solve my problem immediately (for the lack of say, symbols defined in the os package).

PS. I wanted to use TinyGo to compile CLI utilities runnin on embeded Linux devices.

Can you raise a new issue for this?

[typeless]

@aykevl

/home/mural/src/golang.org/go1.16.3/src/net/fd_unix.go:202:12: NewFile not declared by package os
/home/mural/src/golang.org/go1.16.3/src/net/pipe.go:156:16: ErrDeadlineExceeded not declared by package os
/home/mural/src/golang.org/go1.16.3/src/net/pipe.go:169:16: ErrDeadlineExceeded not declared by package os
/home/mural/src/golang.org/go1.16.3/src/net/pipe.go:188:16: ErrDeadlineExceeded not declared by package os
/home/mural/src/golang.org/go1.16.3/src/net/pipe.go:204:17: ErrDeadlineExceeded not declared by package os
/home/mural/src/golang.org/go1.16.3/src/net/cgo_resnew.go:21:17: cannot initialize 2 variables with 1 values
/home/mural/src/golang.org/go1.16.3/src/net/cgo_unix.go:36:52: undeclared name: C.GoString
/home/mural/src/golang.org/go1.16.3/src/net/cgo_unix.go:107:17: cannot initialize 2 variables with 1 values
/home/mural/src/golang.org/go1.16.3/src/net/cgo_unix.go:161:17: cannot initialize 2 variables with 1 values
/home/mural/src/golang.org/go1.16.3/src/net/cgo_unix.go:190:11: undeclared name: C.GoString

The above is what I got when compiling my app using TinyGo + cc + stddef.h from GCC 9.
#1888 looks like the same issue about the missing ErrDeadlineExceeded.
I think most of these, if not all, can be fixed by replacing the net package (#1602).

The remaining error messages like undeclared name: C.GoString and cannot initialize 2 variables with 1 values are more likely on my part though.