Support for YAML

[yesnoio]

Hoping to propose a port of https://pyyaml.org/wiki/PyYAML to a CircuitPython-compatible module.

https://yaml.org

[ladyada]

i think this would be best done as a pure python library - rather than built into the circuitpython core. is it something you'd like to hack on?

[yesnoio]

Sure! Do you happen to have link(s) to preferred documentation on making this a Python library that could be easily imported into an environment using CircuitPython? Can CircuitPython use library or package management software like pip?

[ladyada]

we do! https://learn.adafruit.com/creating-and-sharing-a-circuitpython-library
since this isnt hardware dependant, we recommend you can write your code for cpython3 (using nothing that circuitpython doesn't have access to) for superfast dev, then 'port' it to circuitpython which should be just any clean up

[yesnoio]

Awesome, thx! Will take a closer look! Hope to have progress to report in the coming weeks...

[tammymakesthings]

@yesnoio I don't have a lot of time until the beginning of September, but I'm interested in helping with this effort.

[klardotsh]

I'm preeeeeeetty sure it got ripped out of (or at least disabled in) CircuitPython, but MicroPython ships ujson as a C module. YAML is technically a superset of JSON, so if this was implemented as a C core module, you've got something of a start to base off of: http://docs.micropython.org/en/latest/library/ujson.html https://github.com/micropython/micropython/blob/master/extmod/modujson.c

That said.... YAML is a finnicky, (in my opinion overly) flexible beast and trying to port PyYAML despite all the platform differences is probably the route of fewer headaches, if your board is fast enough to handle it in pure Python.

[njourdane]

i think this would be best done as a pure python library - rather than built into the circuitpython core. is it something you'd like to hack on?
-- @ladyada

Full Yaml specification is more complex to implement than it seems, as it includes many little-known features, such as anchors and refs, flow style, etc.

A light yaml spec subset is probably more relevant for onboard electronics, as it's faster to execute.

In particular, strictyaml is a good candidate, and its implementation is written in pure Python.

I would dig in that direction if I was interested for Yaml support in micro-python / circuit-python.

[njourdane]

An other alternative is poyo, a lightweight YAML Parser for Python. The project status is currently uncertain and it has some serious issues, but it is way more lighter than PyYaml or StrictYaml, so let's give a try. :)

So today I tried to port it to MicroPython, following the instructions here, and unfortunately it was unsuccessful. It's MicroPython but similar issues would also append in CircuitPython.

For any purpose, I'm going to give more details:

1. First I compiled the Unix port of MicroPython from sources;
2. micropython -m upip install -p modules poyo -> got error: Error installing 'poyo': , packages may be partially installed;
3. cloned the Poyo repo, then always following instructions I edited the setup.py and added import sdist_upip and cmdclass={'sdist': sdist_upip.sdist};
4. micropython -m upip install build -> got same error: Error installing 'build': , packages may be partially installed;
5. lets get more details: micropython -c 'import poyo' -> File "poyo/__init__.py", line 3, in <module> | ImportError: no module named 'logging'
6. ok let's manually install logging: micropython -m upip install -p . micropython-logging
7. micropython -c 'import poyo' -> File "poyo/parser.py", line 3, in <module> | ImportError: no module named 'functools'
8. ok let's manually install functools: micropython -m upip install -p . micropython-functools
9. micropython -c 'import poyo' -> File "poyo/patterns.py", line 53, in <module> | AttributeError: 'module' object has no attribute 'MULTILINE';
10. ok, so what is the value of the constant in CPython? python -c 'import re; print(re.MULTILINE.value)' -> 8`
11. ok so I add _RE_MULTILINE = 8 in patterns.py and replace all re.MULTILINE occurences by _RE_MULTILINE;
12. micropython -c 'import poyo' -> File "poyo/patterns.py", line 58, in <module> | ValueError: error in regex
13. line 58 is LIST = re.compile(_LIST, re.MULTILINE), so there is a problem with the _LIST regex, which is a concatenation of several regexes - lets try the first one (_INDENT = r"(?P<indent>^ *)");
14. micropython -c 'import re; re.compile(r"(?P<indent>^ *)")' -> same problem spotted: File "<stdin>", line 1, in <module> | ValueError: error in regex.

So here is the problem: this very light yaml parser contain regexes that are not currently well supported by the micropython.re module (see not supported section).

So basically, the blocking point to have a yaml parser is the lack of good regex support by MicroPython/CircuitPython.

[rimwolf-redux]

Another alternative is, well, Python. The parser is already built-in, it's got data structures with maps, lists, ints, floats, strings, binary strings, ..., and it supports comments.

[njourdane]

Yes!

Or use an external tool to convert yaml to json, load this json on the board, then parse it with the built-in json parser.

[tannewt]

Hey folks, it isn't helpful to suggest non-YAML alternatives here. That's missing the point of the issue.

[njourdane]

I deeply explained in my previous comment that a yaml parser would be very difficult to have in MicroPython/CircuitPython at the current stage of the firmware.

So I think it is helpful to suggest non-YAML alternatives here.