ENH: add layout metadata compiler and examples #5
Conversation
There was a problem hiding this comment.
Sorry for the extremely late review, this really slipped under my radar. As requested, I read over compile-examples.py and I like it. I wonder if it's worth building this into in-toto? AFAICS, the larger part of the script takes care of traversing the metadata, in order to sort fields and truncate long values, which largely aligns with the feature request in in-toto/in-toto#18. What do you think?
If we do want to merge this here, there are a couple of things that need to be fixed:
- The script is not compatible with the current metadata specification (nor are the ~30K added lines of metadata)
- It shouldn't be required that the layout is called
root.layout - Links are loaded by globbing for
*.linkfiles and showed in the order glob returns them. Should we not rather load them as they are defined in layout, as e.g.in_toto.verifylib.load_links_for_layoutdoes? - Link file globbing did not work for me (I didn't troubleshoot though)
- Sublayout's are not handled
- IMO it comes unexpected that the displayed
materialsandproductsare random samples if there are more than 9 of them - Adding ellipses as last item (to show that a
dictis truncated), does not guarantee that it is indeed printed as last item. Python does not guarantee to keep the order of dict items as they were inserted. - A bare minimum of documentation would be nice
Let me know if I should help.
|
Btw. here's a list of required metadata schema changes:
|
|
Hi!
I'm not entirely sure if this is what's to be addressed on this side. I intentionally avoided using in-toto as a dependency (or any templating library for that matter) so as to keep this script-y. I believe that that issue has been floating around and having different meanings every time we revisit it, I'm afraid. As for the point issues:
Yes, unfortunately time has gone by, we may want to update it to conform to the latest spec.
Probably not, but considering we're the ones that will be running this to update our examples in the docs I don't see why we should care much about user interface right away (this could be also ticketized)
This requires a dependency on in-toto or smarter parsing of json objects, which I tried to avoid. Again, we could make the tool smarter if we'd like by adding more deps/code, but we may want to think about it for just a docs repo.
I suspect it's because the keyid prefix,
No, as we don't have any examples on the docs repo that use sublayouts. We can always add support for this as the need arises.
I can't personally think of any other way to keep things succint, but I'm open to suggestions. I do agree this is not a perfect solution to overly verbose metadata.
This is true, and it's something I bailed on working on back then. We could use a frozendict or serialize and then append on the printout after-the-fact (which would be messy).
Agreed. Let's decide on whether this goes here and work accordingly. |
- metablock
- all top level fields are now under the `signed` field, and the
`signatures` field is a sibling of the `signed` field
- layout
- `expected_command` and `run` must be lists
- `expires` must not have milliseconds
- `material_matchrules` are now `expected_materials`
- `product_matchrules` are now `expected_products`
- `MATCH` rule syntax has change
- `threshold` is a mandatory field
- link
- `_type` value is lowercase
- `return_value` is now part of `byproducts`
- `environment` is a mandatory field
Add function that recursively traverses a passed python object, e.g. in-toto metadata, allowing to truncate long strings, lists and dicts, and to reorder dict keys, using OrderedDict.
Update and merge template populating functions, make them use the newly added metadata beaufifier (truncate and order) and rename to create_markdown_summary.
Update markdown summaries for debian, polypasswordhasher and seattle sample in-tot metadata.
Update sample metadata summaries for pph, seattle and debian supply chain metadata, using latest version of Santiago's metadata compiler script (see in-toto/specification#5). This commit also adds jekyll frontmatter to auto convert markdown to html and make the metadata sample pages part of the website layout.
|
@SantiagoTorres, I updated the sample layout and link metadata to meet the latest version of the spec, cleaned-up the compiler script, and used it to create new versions of the markdown-formatted summaries, which I have already published on our website (see pph, seattle and debian). Let me know what you think. |
|
Btw. 088277b is my stab at in-toto/in-toto#18. IMO especially the dict field ordering for pretty printing is very useful. |
examples/compile-examples.py
Outdated
| if len(obj) > kw["max_str_len"]: | ||
| obj = obj[:kw["max_str_len"] - 3] + "..." | ||
|
|
||
| # Truncate list and recurse into _beautify for each item |
There was a problem hiding this comment.
Good eyes! Thanks for the review. :)
Hello, this commit adds the metadata compiler and a couple of metadata samples (debian grep and seattle).
Don't review the link metadata files, just the readme and the compiler please.