Support and use YAML blackboxes

[martijnbastiaan]

Added support for YAML blackboxes. Clash will now pickup on files with a
.primitives.yaml extension. While we recommend upgrading your primitive files
to the new format, old style primitives are still supported.

Depending on whether you use Stack or Cabal you can execute the following to
upgrade your primitive files:

cabal run clash-lib:v16-upgrade-primitives -- --help

Stack users can use:

stack exec --package clash-lib v16-upgrade-primitives -- --help

Both commands assume you're running them in an up-to-date
(starter) project environment.

[alex-mckenna]

+14,491 −16,124

Someone's gunning for top contributor 2022

[martijnbastiaan]

I wish there was a [no-count-diff] :P

[leonschoorl]

This maybe hard or impossible with the high-level YAML api.
But I feel it would be make the generated .primitives.yaml much nicer to read if we could get the conversion tool to put the type field, before the template field.

[alex-mckenna]

I doubt easy. Objects in aeson are just stored as maps then traversed over. I think you're at the mercy of how the map is ordered internally

[martijnbastiaan]

Yeah.. that'd be the Ord instance of Text :(

[leonschoorl]

You're adding the Clash_CoSim_CoSimInstances.primitives.yaml, but the corresponding Clash_CoSim_CoSimInstances.primitives was never in git.
It's generated and .gitignored
I think we need to do the same here and update the generation code.

[martijnbastiaan]

But I feel it would be make the generated .primitives.yaml much nicer to read if we could get the conversion tool to put the type field, before the template field.

@leonschoorl Python's dicts are insertion ordered, so I've abused that to generate YAML files with the pretty order:

#!/usr/bin/env python3
import yaml
import sys

# https://github.com/yaml/pyyaml/issues/240#issuecomment-1018712495
def str_presenter(dumper, data):
    if len(data.splitlines()) > 1:  # check for multiline string
        return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
    return dumper.represent_scalar('tag:yaml.org,2002:str', data)

yaml.add_representer(str, str_presenter)
yaml.representer.SafeRepresenter.add_representer(str, str_presenter) # to use with safe_dum

def reorder_dicts(o, order=("name", "kind", "type", "template")):
  if isinstance(o, list):
    return list(map(reorder_dicts, o))
  if isinstance(o, dict):
    # Reorder dict
    new_dict = {}
    for s in order:
      if s in o:
        new_dict[s] = o.pop(s)
    new_dict.update(o)

    # Traverse all values in dict and reorder _them_
    reorderd = {k: reorder_dicts(v) for k, v in new_dict.items()}
    return reorderd
  if isinstance(o, (str, int, bool)):
    return o
  else:
    raise TypeError(f"Unexpected type: {type(o)}")

def reorder_dicts_in_file(path):
  with open(path) as fp:
    prims = yaml.safe_load(fp)

  prims = reorder_dicts(prims)
  prims = yaml.dump(prims, sort_keys=False, line_break="\n")
  with open(path, "w") as fp:
    fp.write(prims)

if __name__ == '__main__':
  import glob

  for path in glob.glob(sys.argv[1], recursive=True):
    reorder_dicts_in_file(path)

Aeson should probably use insertion ordered maps too..

[martijnbastiaan]

I think we need to do the same here and update the generation code.

Done

[leonschoorl]

I think we still need:

• documentation
  • changelog
  • some documentation on (how to run) the upgrade tool
  • the examples in Clash.Tutorial still use the old format
• maybe a test that uses the old format, so we don't accidentally break it

[martijnbastiaan]

@leonschoorl All done.

[alex-mckenna]

Yeah, seems fine. I didn't spot any, but since all the primitives are being changed it might be worth trying to see if any have obviously wrong type comments, i.e. using Undefined instead of NFDataX

[alex-mckenna]

Bounds for yaml since it's not a boot library

[alex-mckenna]

A masterpiece

[alex-mckenna]

+16,061 −17,015

It just gets chonkier 👀

[martijnbastiaan]

:(

[alex-mckenna]

As for me, I like big diffs and I cannot lie. -- Martijn Bastiaan