META6 specification only exists "de facto" #236
Comments
|
I would say that it would probably benefit in the first place from a machine readable specification (probably JSON Schema,) with some annotation describing the roles and responsibilities of parsers, testers, installers, the core and so forth. I'm guessing that extension could be achieved with JSON prefixes (e.g. |
|
However, we should also check that meaning is what it should be, and resolve ambiguities, but yes, JSON Schema would be a nice first step. |
|
I'm not going to share the whole generated schema right now because there are a couple of things that aren't expressed well in a simple schema, but it looks like people have already been extending it themselves without causing any great harm to the tools, here is the complete list of top level properties found in all the extent META files: abstract :-D |
|
Interestingly it seems that at least seven of the META files that are fetched by the |
|
Okay leaving out the ones that appear to be Perl 5 meta files we have: The |
|
The I think however that I can get a schema that will validate most instances. |
|
... and then we will need to publish that schema in some official way so
that we can validate META6.json validators...
|
|
Well, it would be stick it in it's own repository with tagged versions and so forth, probably the tags would align with some |
|
We should be wary of specifying things raku does not itself use. For instance it would seem a bit odd to me for the roast to specify what source-url, an ecosystem specific field, is supposed to mean. Raku doesn’t even have a way to parse dependency specifications from depends (nor does it use them currently), so how can we reasonably spec that? Having a schema and tooling seems perfectly reasonable. Speccing everything in the roast seems short sighted. |
|
Personally, I'm not leaning toward speccing this in roast - I agree that is largely ecosystem specific, but it should be specified somewhere. |
|
The |
|
I was rather thinking about a roast-like repo, maybe a specific one, maybe
we could re-use the Raku/ecosystem and add them somewhere.
Definitely, *not* *the* roast.
|
|
It would be great if |
Sure, if your modules are in the github ecosystem then you can just update in your repository, if they're in CPAN you may need to up the version and re-upload. 👍 |
|
fwiw I wrote a web service with a very naive and very incomplete validation using OpenAPI which might be useful for prototyping or bootstrapping such a validation service -- https://github.com/ugexe/Perl6-App--OpenAPI--META6Validator |
The problem is that these things aren't completely specified anyway, that's what we're trying to get at here. Though |
|
This is my first cut at a schema that will validate at least some of the META6 in the wild: The I'll stick some annotations in later. |
|
`build` is `builder`-specific, i.e. it's a place designated for information
that the module specified as `builder` will use. Currently the only `builder`
implementation is `Distribution::Builder::MakeFromJSON`
All information in `build` and `depends` is subject to `System::Query`
collapsing, i.e. at any point you can specify an object with a `by-
something.property` key and the whole object gets replaced with one of the
appropriate values. The "somethings" in the key may be `env`, `env-exists` for
checking environment variables or `distro`, `kernel`, `backend` for Raku's
`$*DISTRO`, `$*KERNEL`, `$*BACKEND` objects respectively.
E.g. the whole `depends` section may depend on the distro name. Or just the
version of one of the dependencies may depend on the VM backend.
The `depends` section may also contain alternatives via an object with an
`any` key:
"depends": [
"Foo",
{"any": ["Bar", "Baz"]}
]
```
This (and much more) is specified in S22:
https://github.com/Raku/old-design-docs/blob/master/S22-package-format.pod
I don't understand why this is called "mostly undocumented"?
The `build-depends` and `test-depends` keys are deprecated. The `depends` key
itself is not required (after all a module may just not have any
dependencies). Same for the `resources` key. I would argue that this is even
true for the `provides` key. A distro may just contain resources and no
modules. Or even more realistic, it can be just a collection of dependencies
like `Task::Galaxy`.
|
Apparently, because those documents needed some updating. Also, it's not anywhere else, and anyway design documents are, literally:
The documentation in docs.raku.org about META6.json is not an specification, anyway.
Well, that might also be underdocumented. In this issue (from exactly one month ago), you can read:
So, maybe, just maybe, we would need a little specification here? |
|
Apparently, because [those documents needed some updating]
I was writing specifically about "The object version of `requires` seems to be
similarly mostly undocumented". My commit did not change anything about that.
The object version of `requires` has been documented for years.
Also, it's not anywhere else, and anyway design documents are, literally:
> may be out of date... See docs.raku.org for documentation, or the official
> test suite.
"may be out of date" does not mean "definitively out of date".
The documentation in docs.raku.org about META6.json is not an specification,
anyway.
I disagree. I do know that when I wrote it I meant it to be a specification.
> test-depends isn't deprecated, and I'm not sure why anyone would assume it
> is.
So, maybe, just maybe, we would need a little specification here?
Huh...ok, this needs discussion then. I don't see why we'd need test-depends
and build-depends when the same can already be specified in depends. But maybe
there's a good reason that we discussed back then and I simply forgot.
|
Largely the |
For instance, to a tool that helps us validate a META6.json including a JSON schema and its outcomes. |
This part makes providing an exhaustive schema for a META6 and any static validation against such a schema rather tricky as it is the result of the collapsing that needs to be validated because, as I understand it, the collapsing can apply to any part of the JSON to produce a value. I think a validator would need a schema with annotations to indicate which parts are subject to collapsing, pre-process those parts with |
|
I really had to look up what you mean by System::Query "collapsing". Apparently it refers to this module. Again, we really need a specification with annotations for META6.json, so that we can enter deprecation cycles for some keys or whatever is needed to carry this forward. |
An attempt at implementation has shows that using lists for dependency alternatives causes confusion and is not self-documenting enough as the depends section itself consists of a list. Using an object with an any key is more clear and lends itself to future expansion. Collapsing of system specific values has always been meant to also cover dependencies. This is now stated explicitly and the details on what's available described. The build-depends and test-depends keys are deprecated. They should not have been part of the v1 META spec. Leaving them in was just an oversight.
|
FWIW, it may be worthwhile actually writing a specification for what |
This has a further implication that https://github.com/jonathanstowe/META6 may struggle to do all of the things that people are using it for if it is to parse a |
Yeah, but for me this was the key piece of information that I was missing, once that was clear it all fell into place :-D |
|
While I think about this, some thought might want to go into documenting a "roadmap" and versioning scheme for the META6. I'm thinking the versions (
Obviously not exhaustive. Handling of the versions will have different implications for installers, validators, generators and other processors. |
|
That |
|
Oh yeah, looking at the My current line of thinking is to create a schema for each version, document the versions explicitly and lay out the requirements of the different classes of META6 processors viz handling of the versions (which in itself may require a separate roadmap as to when support for a version may be dropped for e.g. an installer.) I'm having a think about about how best to determine where to draw some of the version lines against what is presented in the wild, I'll get to making something over the weekend. PR, not so sure, I'm thinking this should all go in its own repository. |
|
I really think it would be much better to have it in a specific repo, with
specs, schemas and possible validating tests.
Also a calendar for meta-versions, aligned (or not) with Raku releases.
|
|
So regarding consensus I think it's good for "let's sort this out", probably want to get more input from the various vested interests (tools authors, module authors etc,) on the actual details of the specification as the versioning firms up, I'm not even sure I know all the places where the META6 is used by tools, modules and so forth beyond those that depend on the module |
|
Consider -- Raku implementations will be expected to consume a specific subset of META6.json data, so those parts of the spec are definitely owned by raku (and is even tested in the roast via CURFS loading distributions through their META6.json). Are the proposed meta-version changes expected to have any meaning to raku (rhetorical: I would suspect the ones mentioned in here would not)? If not then how can raku change the parts of the META6 spec it does own in a versioned manner without having to be aware of all these (what I'll call) ecosystem-not-raku specs? It almost seems like there needs to be two different meta versions... one for the 'core' and one (or more) for anything else outside of what raku would consume. |
|
This ticket talks about creating a specification for the META6.json file. The |
AlexDaniel
added
language
|
This spec is *not* part of the language. Maybe part of the ecosystem, but
not the language.
|
|
Good point! For reference: #45 |
AlexDaniel
added
fallback
Yet, META6 is so tightly bound to Raku that speccing the format alongside with the language would be the right thing to do. Raku must define a minimal META6 set which is sufficient for, say, a module to be available for installation by different Raku implementations. Also, considering possible META6 extensions by 3rd party tools, it is better to take care of possible key name clashes beforehand as later it may pose serious risk of a tool failing on a module due to it using a key meant for another tool. Bad and incompatible tooling around a language would do bad for its reputation. So, not really speccing in the meaning of having tests, but a way to register new non-core keys should exists. BTW, I'd really be happy to see extension keys using namespaces. Perhaps this should be considered a requirement. Also, I think of changing files in And perhaps the last consideration I have: META6 could be not the last entity which doesn't really belong to speccing yet tightly bound to the language. I can't really foresee what exactly could be standardized, but can't exclude the possibility itself. With all the above in mind, I think a good thing to do would be to introduce The way I see the final META6 spec is a paper with primary section defining the base set of keys sufficient for a project to be installed by any Raku implementation or packaging tool (as mentioned in the beginning of this comment). Then additional sections describe extensions supported by tools. A tool author may request adding new keys to the spec. This would make them reserved and protected from future re-use by other tools or by Raku language itself. Though if namespace use is made a requirement for extensions then it's only namespaces that would require registration. JSON schema could be provided alongside with the paper or as a section of it. Unless I missed something in the preceeding extensive discussion, the proposed solution should get most of the issues about META6 standard resolved. |
|
The sufficient keys are simply name, auth, api, version, and provides. Literally nothing else is needed to install a distribution in raku. |
|
Also I don’t agree that tooling authors should be able to make extensions to the spec. Extensions should work totally independent of any raku authority, else why is it an extension at all? |
|
Yet, META6 is so tightly bound to Raku that speccing the format alongside
with the language would be the right thing to do. Raku must define a
minimal META6 set which is sufficient for, say, a module to be available
for installation by different Raku implementations.
I'm absolutely for specing the contents of the META6.json file as part of the
language. I consider more standardization an entirely good thing in this area.
The serialization format is flexible enough to store arbitrary data, even Raku
code, if one thinks that's necessary, so by allowing custom keys, we should be
flexible enough to be future proof.
Indeed, the discussion is kinda moot, as our spec tests already contain tests
for META6.json files and installation of distributions containing them.
Also, I think of changing files in `old-design-docs` as of a bad idea. That
repo is now better be kept for pure historical purpose including cases when
some language design decision is based upon an old spec or some
argumentation inside it. I understand @niner reasoning about fixing META6
descirption while he has time for it. Unfortunately, under time pressure it
is often really only two choices of either evading doing something or doing
it a wrong way. But for now I'd like to have this fixed.
I do not agree. At some point we started focusing more on the code and the
design documentation fell a bit behind. Someone noticed this and added a
banner to the website that rendered these design docs, so readers would be
less confused by differences between the design docs and the spec tests (and
the implementation). Then we moved some infrastructure as part of the name
change and someone picked the name "old-design-docs" for the design documents.
I have not seen any consensus or deliberate decision on abandoning the design
documents. It's just people having taken steps to avoid confusion and making
assumptions. At no point in time have these documents matched the
specification (i.e. roast) and therefore they have never matched any of the
implementations we've had. They could not have since by their very nature,
they had to be ahead of the other two.
These documents are useful. They would be even more useful if they were more
up to date. I just cannot see any reason for deliberately making them less up
to date. Who would be helped by that?
Regarding the updates I did to S22, if I'd waited for the other place to
appear where I could have written up the specification, I'd still wait today
and would have still not been able to write. There simply isn't one. And
anyway, what would be improved by taking the same document and putting it into
an addition place? How would the availability of different versions of the
same document reduce confusion?
|
AlexDaniel
added
language
What about
It wouldn't oblige any tool developer to to anything. But if they wish to prevent future conflicts then formally made request to register an extension would most certainly be appreciated. Similar situation with namespaces except that registering a namespace (again, purely at will of a developer) eases everybody's life in many ways. |
I wouldn't agree that keeping them up to date would make a lot of sense with respect to the resources used for the task. My view that the repo name is very much correctly reflects what value the papers have to the moment.
I don't propose to have different versions of the same document. I propose to take the original and use it as a draft for a new paper. The final version of it should be very much different from the source simply due to the different purpose of it. With regard to the synopsis content, to the best of my knowledge, it's only a subset of S22 which is currently supported. Perhaps it is yet to be decided if the rest of it should be implemented and consequently included into the final spec. |
yes
I don't see it as a problem needs to be solved with that type centralization. We wouldn't e.g. allow users to register namespaces in GLOBAL to give an absurd example. An author of tools consuming non-standard fields from the META6.json would likely be aware they shouldn't call their special field a common name (which are documented), and that revelation to an unbeknownst tool author wouldn't seem like a big deal to anyone -- "put an underscore in front of it". You can't even register a module namespace right now... discussing a process for users to register a META6.json field in itself just feels like scope creep at this point in time. |
|
Realizing now that we still have a "perl" key in META6, which is used by most distribution, along with a "raku" key some new distributions use. |
The META6 format was defined in S22, and is arguably not part of the language. There's a single test for S22, but it does not really check an interpreter of that format, just that the format can be used correctly.
The interpreters of this format are, in general, external utilities
As far as I understand, there're no tests that cover all posibilities in S22, not even "currently adopted" possibilities.
zefis the standard de facto, so the test we have now is "if it goes with zef, we're good". But not having a common set of (roast or roast-like) tests is an obstacle for evolution, should we need to extend it somehow (as was suggested here).There are also a number of ambiguities that are not solved; mainly with the "depends" key and the "source-url" key. Build dependencies can appear in the depends → requires → key and in the build-depends key. Is there any precedence? Should we take both into account?
The text was updated successfully, but these errors were encountered: