Refactor tag/value parser for the new data model

[nicoweidner]

The tag/value parser should be refactored and adapted to the new data model, similar to how it was done for json in #305. The old code can be found at https://github.com/spdx/tools-python/tree/b8fde35e141daf082b8b0fe32f875da06e6316e6/spdx/parsers (see the tagvalue modules as well as the lexer package), though it was pretty hard to understand - hopefully we can improve in that regard.

@meretp already did a quick initial investigation and came across https://github.com/webmaven/python-parsing-tools. It may be worth evaluating e.g. pyparsing as an alternative parser library if it makes the code clearer and easier to understand.

[zvr]

The list of parsing tools mentioned is many years old (not updated) and includes pointers to abandoned projects.

Actually the existing/old parsing code was build on ply, which makes it very easy, if one has experience with traditional parsing tools lex/flex and yacc/bison. Since David does not make installable modules for the new releases of ply, consider using the very similar sly module.

[nicoweidner]

Well, "if one has experience with traditional parsing tools lex/flex and yacc/bison" is a potential issue here, but may depend on who works on the parser. I personally find the fact that one configures the parser/lexer via docstrings and has to follow magic naming rules for methods horrendous API. I was hoping that some other library would make it easier to understand what's going on without requiring users to have special knowledge.

It was also remarked by one or two other contributors that they gave up on implementing their changes for tag/value (and rdf, for that matter) because they didn't understand how the code works.

All that said, I don't have experience with the other libraries that were mentioned, so I don't have a favorite. Thanks for the pointer to sly - maybe that improves the API already!

[zvr]

The alternative to docstrings would be to have external files describing the grammar, much like the traditional tools do. I actually find the solution ingenious, but I understand it may not be for the uninitiated.
The sly interface only slightly changes this syntax: instead of docstrings for the grammar productions you use method annotations.

If the maintainer(s) of this code is not familiar with LALR parsing, probably writing a recursive descent parser for tag-value could be a better approach.

[meretp]

Given the fact that using a new package and writing a complete new logic is rather time-consuming, I decided to implement the tag-value parser with ply. To avoid the use of docstrings to configure the lexer the package already provides the decorator @TOKEN (for all lexer methods). There is no default decorator for the parser but I'll probably write a decorator to use instead of docstrings as suggested by the developer. I think this approach would be the best compromise to avoid the current docstrings, refactor the parser according the new structure and not having to rewrite everything.

[zvr]

As I wrote above, the author's sly package is configured via decorators, instead of docstrings that ply is using.

Since docstrings are going to be used for the syntax/grammar part, why not use them for lexical/tokenizing part as well, rather than mix the two methods?

[meretp]

I don't want to mix the methods, I'm planning to use decorators for both parts. And I tried to use sly in the first place but it is (apart from the decorators) a bit different to implement and it seemed to me that it is easier to reuse part of the current code than rewrite everything, so I decided to use ply.

[meretp]

fixed by #502