Add back XSD file (or other definitely parseable schema)

[marcusmueller]

In reaction to #167, the XSD file was removed – understandably, I might add, since maintaining that separately from a manual validator is a hassle.

However, that was a bad move, in my humble opinion. Now, the docs deviate from the output of validate (#278), and neither indicates whether the document is actually valid for any downstream parsers that aren't this software in itself.

Also, the majority of the issues on this tracker are request for new field or property types. Thus, as an external observer it seems that the grunt of the maintenance work is to maintain the scheme, not the parser for the same.

And I greatly appreciate this – this is mostly about defining a metadata format that different consumers need to parse and multiple tools should be able to produce; the less clear it is to me why the validator is not doing a syntactic check based on e.g. a DTD (which libxml2 does out of the box) or XSD.

That would allow me to look up things that aren't well-defined in the docs (which is understandable - documentation is for humans, but strict metadata schemes require rules, which are hard to put in still human-consumable text), for example, whether or not I can have multiple tags of some type (contact URLs), whether or not <project_group/> should actually be a content-free tag, as the writing <tagname/> suggests in the docs, or contain some CDATA and so on. I just tried to write one metainfo file, and was met with multiple issues. I can look up how other files handle this – but who tells me these files weren't simply written before appstreamcli validate got a check that checks for what they shouldn't be doing?

Please, instead of having a 2104 line as_validator.c strive to put validation back into the hands of a tool that was designed for more than one project. Also, clearly version that file with releases, please.

[ximion]

I am definitely not planning this anytime soon. There is so much more to validate than just the bare form of the XML that also depends on a lot of context (like certain tags needing to be present only if the component is of a certain type, lots of string formatting validation, URL checking, image checking, ...), and I much rather spend my limited time on improving the validator rather than keeping an extremely complex XSD schema up-to-date.
I would accept a PR for adding a schema back though, if someone committed to maintaining it ;-)

Thus, as an external observer it seems that the grunt of the maintenance work is to maintain the scheme, not the parser for the same.

It's actually a fairly even split - the AppStream libraries do some fairly complex parsing for legacy support, but also do way more than that (which is usually where the actual work starts, like in the data generation or caching code).

[marcusmueller]

Hi! Just wanted to a) show some appreciation for the work you do, and as a FOSS maintainer, tell you that it's very OK to assign your limited resources to the best of your abilities and assessments of merit, but b) sadly say that a metadata data format that doesn't specify valid metadata is not viable, so my own investiment in providing appstream metadata gets limited by my assessment of their long-term usefulness. Been bitten by deprecations and new invalidity two times already, and not having a scheme to check against means the file format is less than useful.

[pktiuk]

@marcusmueller

not having a scheme to check against means the file format is less than useful.

You can still use appstreamcli validate

[marcusmueller]

@pktiuk please refer to the opening description of this issue to understand why that has proven, by now on multiple different changes, to be undesirable and unstable.

I am sorry, but this is a metadata format. It needs to be formally specified, not in the shape of C code.

[marcusmueller]

Again, I fully appreciate the work load and the need for forward development, but:

This is a file format that distros encourage projects to use. In fact, they have contributors going through Foss projects issue trackers and opening bugs, if the age restriction metadata isn't right. Obviously, I added unit tests that directly call appstreamcli validate. These break, without warning, because the format is not properly frozen in versions that you can specify that you adhere to and check against.

Saying "we have a tool that checks whether a file is consistent with the things we check in that particular version of the tool, which is neither the full formal file format, nor restricted to erroring under strict violation only" simply shifts effort to hundreds of project maintainers.

So, yeah, as producers of this format, these projects need stable releases of the format with a formal spec. This isn't some file format with a single producer and a single consumer, like, say, gimp project files, this is supposed to be the backbone of Foss software description.

If a reliable spec is not yet possible, then I'll consider the file format in alpha state and will have to kindly let our (very awesome, by the way) Debian maintainer know that if it breaks again, or if there is requests for changes or additions in metadata, it becomes a distro packaging problem, not an upstream problem, to provide appstream metadata and keep it up to date.