Switch to GNU Bison

[ISSOtm]

We've been sticking to POSIX Yacc for a while, for compatibility with some target systems, especially the BSDs. However, most systems actually use GNU Bison behind the scenes, and using some of their extensions would be quite beneficial. (Note: I'm not sure if all of these are GNU extensions, if I am wrong I'll strike-through my mistakes)

• Nicer error messages ("Syntax error" message is way too obtuse #385)
• Error recovery, so that RGBASM doesn't abort on the first syntax error
• freeing memory when a token is deleted, useful if we want to try reducing memory leaks

This wouldn't change much for us (our CI seems to exclusively use Bison already), it'd mostly be a downstream problem, though probably not a big one. Is it worth it?

[AntonioND]

I think this is a pretty good idea. Anything that improves error handling and messages is welcome.

EDIT: Also, I don't think that any system that has yacc doesn't also support bison.

[ISSOtm]

I don't think any system cannot have Bison, but I'm pretty sure some systems don't have it by default, like the BSDs have GNU Make as a separate gmake. Similarly to using CMake, it only makes RGBDS less lightweight in terms of dependencies; I don't think it's a big deal, though, since we're not introducing anything outlandish.

If we're agreeing on the move, then should we do it in 0.4.2, or wait for 0.4.3?

[JL2210]

I think that we shouldn't change dependencies between a pre-release and an actual release, so 0.4.3 seems good.

[ISSOtm]

I don't consider 0.4.2-pre an actual release, so I don't think changing deps would be a problem, but I'll respect that decision.

Further, the changelog is actually decently-sized, the lexer changes are large enough as-is, and the target release month of November is nearing, so it probably won't hurt to limit 0.4.2's scope. Marking this for 0.4.3, then.

[ISSOtm]

I have been looking at Bison's docs in preparation to the change, and one worthwhile subject to look at is which parser table type we want. I'd say IELR is best, since we mostly care about performance, but then we'd have to enable LAC to get proper error messages, which sounds slow...

It's also worth considering enabling raw token kinds, as that'd remove one level of indirection.

[Rangi42]

Error recovery, so that RGBASM doesn't abort on the first syntax error

This could be done with:

 line		: label T_NEWLINE
 		| label cpu_command T_NEWLINE
 		| label macro T_NEWLINE
 		| label simple_pseudoop T_NEWLINE
 		| pseudoop T_NEWLINE
 		| conditional /* May not necessarily be followed by a newline, see below */
+		| error T_NEWLINE /* Continue parsing the next line on a syntax error */
 ;

It might lead to more confusing messages though, as a syntax error in one line could break many subsequent lines (like how a missing semicolon in C/C++ can give lots of unrelated-looking errors instead of one "semicolon not found").

Edit: The current label-macro-arg test errors are affected this way, by continuing a macro after a syntax error in sizeof_\1something = 1:

 ERROR: label-macro-arg.asm(38) -> label-macro-arg.asm::test_char(25):
     syntax error, unexpected '='
 while expanding symbol "VAR_DEF"
-error: Assembly aborted (1 errors)!
+ERROR: label-macro-arg.asm(38) -> label-macro-arg.asm::test_char(29):
+    Interpolated symbol "sizeof_.something" does not exist
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(25):
+    Label "sizeof_" created outside of a SECTION
+while expanding symbol "VAR_DEF"
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(25):
+    Macro "something" not defined
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(26):
+    'sizeof_' already defined at label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(25)
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(26):
+    Macro "something" not defined
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(29):
+    Invalid format spec 'sizeof_'
+ERROR: label-macro-arg.asm(39) -> label-macro-arg.asm::test_char(29):
+    Interpolated symbol "something" does not exist
+error: Assembly aborted (8 errors)!