implement py2hy

[gilch]

A lot of issues (e.g. #850, #848, #855, etc.) indicate that Hy can't yet do everything Python can. Now, we've noticed and started fixing those, but it makes me wonder what else we're missing. Even an easy language like Python has a fairly complex grammar compared to Lisp.

Our current approach of just waiting to notice things and then fixing them is risky. What fraction of those issues break backwards compatibility when corrected? And how many more haven't we noticed? We've resolved to do a grand language cleanup this time, but in the future when there's more Hy code to break we'd be even more reluctant to try something like this again. We'd probably end up with clutter and kludges.

A better plan would be to go over the Python Language Reference with a fine-toothed comb, to make sure we've got everything. Anyone attempting this would surely learn Python well, but this sounds tedious to me.

Therefore, I propose a third option: implement py2hy, not just as a side project, but as an integral part of the Hy project itself. This doesn't seem nearly as boring. Once started, we can leverage existing Python code and unit tests to exercise Hy itself. We would begin to notice problems and omissions much more quickly.

Round-trip translations using hy2py could quickly point out inefficiencies like #842.

py2hy would also benefit beginners coming from Python, who know what they want to say in Python, but not how to say it in Hy, particularly when run through hydiomatic afterwards. (It would be nice to have these capabilities in the repl.) They could work through existing Python textbooks with Hy instead. (So could we, for that matter, and maybe rewrite some of the Creative Commons ones too, instead of starting from scratch).

py2hy would also give us the power to automatically translate existing Python into Hy and then refactor with macros instead of boilerplate, instead of just importing Python as we do now.

We'd probably end up going through AST. I'm not sure how difficult this would be, but @paultag seems to think this is doable in #90.

[refi64]

There's an easy way: add something like:

(python-code print(1, 2, 3))

Python code needs to follow matching brackets and such anyway.

[algernon]

One of the use cases of py2hy, namely to discover areas where Hy is more limiting than Python, is something I'd care less about: one can always write that part in python and import it. I don't think of Hy as Python with a different syntax, I think of it as a language that happens to compile to Python AST, and has a reasonably good interop with Python. If you treat it as a different language, the need for py2hy diminishes.

It'd still be a useful tool, and combined with hydiomatic and some decent pretty printer & auto-formatter, one could do incredible things. But unlike paultag, I wouldn't be so optimistic about Python AST->Hy being easy.

[woodrush]

I'm currently working on this now at https://github.com/woodrush/py2hy. The main idea is to make the Python AST into S-expression form, then treat Python AST keywords as Hy macros, and let Hy recursively macroexpand the raw Python AST S-expression. There is some sample output I've generated which seems fairly promising right now.

I wrote a parser for the Python AST specs that creates a template code to be filled in to create py2hy. The project is still work in progress but I believe it could be implemented this way. Current potentially large issues I've noticed is how to work around return, which would at the time need lots of AST workarounds. Otherwise, I believe it is fairly straightforward to implement it this way.

Once I get this finished, I plan to try to write HyHy, a Hy analogue of PyPy in Hy, and see how far it could go. I think it would serve as a clean definition of how Hy transforms its AST to Python AST and vice versa.

[gilch]

See #739 about the return statement. We might end up adding one.

[woodrush]

I did some work on py2hy, and it's mostly done now.

The current state is like this:

• It is capable of rewriting the Hy core to Hy, to create a running version of "HyHy".
• The return statement is implemented via a workaround similar to Tritlo's comment.
• I've started writing a pytest for Python>=3.3. It runs on tox but doesn't have a full pytest yet.
• The code it outputs is still ugly as mentioned by @gilch in another issue, mainly because of the return workaround, and it doesn't do some nice optimizations yet.

Upon integrating this to the Hy core, I was thinking we would benefit more if we made it a separate repository in the hylang toolset like hylang/hydiomatic. Some reasons why are,

• While hy2py leverages on the Hy core (the core is by definition a Hy to Python converter), py2hy runs on an entirely separate main function from the Hy core. This makes the development of hy2py and py2hy a different task, and therefore creates an extra burden when updating the Hy core.
• The codebase is large. The current py2hy is over 1000 lines, and has several helper modules (the pretty printer and the AST specs parser). Since the test would highly likely be as large as itself, that would make the estimated final size to be around 2000 lines.
• The tests aren't finished yet, and it would take another load of time to finish writing it, so it needs some more effort to be a mature portion of the Hy core. Yet, it successfully runs HyHy and all of its pytests, so it would be practically useful for debugging Hy itself, as @gilch mentioned in the first comment. py2hy could be debugged concurrently with Hy, and as said earlier, that would make more sense if done as a separate task from the Hy core.

What are your thoughts?

[Kodiologist]

That sounds fine to me. Also, if you're going to continue working intensively on py2hy, it will be more convenient for it to be in another repository where you can push at your leisure, instead of here where you need two approvals for every change.

Note that I recently implemented return, although it's not merged yet (#1380).

[woodrush]

Thanks! I plan to continue working on it, so yes, it sounds be better to keep it in the current repo for a while.

While that's going on, would it be OK for me to push this on PyPI? I haven't pushed to PyPI before but I was wondering if it wouldn't cause unwanted conflicts with Hy.

Note that I recently implemented return, although it's not merged yet (#1380).

Yes I saw it! Many thanks for the PR. I'll put that in py2hy once it's merged. It would make everything very clean and simple.

[Kodiologist]

While that's going on, would it be OK for me to push this on PyPI? I haven't pushed to PyPI before but I was wondering if it wouldn't cause unwanted conflicts with Hy.

Sure, that shouldn't be a problem.

I would advise you to make sure that the bytecode for your Hy code is either included in the source package (and the wheel, if you make one) or generated at installation time. Bytecode makes a big speed difference, so you don't want the user to install the code in a place where only the superuser can write without the corresponding bytecode.

[woodrush]

Awesome! Thank you very much! I'll post again when I get that done.

so you don't want the user to install the code in a place where only the superuser can write without the corresponding bytecode.

Indeed! I finally got why Hy uses the Install class in setup.py. Thanks for the advice. I'll make sure I do that in py2hy as well.

[Kodiologist]

Since this feature request is being fulfilled as a separate project, at least for now, I guess we should close this issue.