The great core shakeup

[Kodiologist]

Hy has a big and disorganized set of core functions and macros that get automatically injected into __builtins__. This is an improvement over the old status quo, in which they were automatically imported, but still messy. I propose some radical changes to clean things up. I'll do it over the course of several PRs, and we can bikeshed about the details (e.g., the fates of individual functions and macros) when we get there.

• Remove machinery for injecting functions into __builtins__.
  • Thus, the only names that will be user-visible without the user importing or requireing stuff will be hy, core macros, and gensyms and other temporary names.
• Rename model types to remove "Hy".
• Make quote use qualified names to pull in model objects, so '5 will compile to hy.models.Integer(5) instead of Integer(5).
• Reorganize core macros:
  • Rename if* to if.
  • Rename annotate* to annotate.
  • Delete fn* in favor of defn.
  • Delete with* and with/a* in favor of with and with/a,
• Rename hy.core.shadow to something more logical like hy.pyops and encourage users to star-import it when they want first-class Python operators, as in (import [hy.pyops [*]]).
• Make these functions available qualified under hy, as e.g. hy.eval: eval mangle unmangle gensym macroexpand macroexpand-1 read read-str disassemble repr (formerly hy-repr) repr-register (formerly hy-repr-register).
• Make a new repository under hylang, hyrule, to hold a Hy library with a variety of functions and macros that are useful for Hy (and sometimes Python) but not required to implement Hy.
  • Hyrule's contribution guidelines and core team will be the same as Hy's, but it can be less stable than Hy itself and perhaps it will have a different release schedule.
  • The organization of Hyrule is to be determined, but splitting things up into modules is likely better than one big module.
  • Include all of extra and contrib, except for hy-repr.
  • Other macros (formerly core) to be included:
    • Control structures: unless do-n list-n lif
    • Meta-macros: with-gensyms defmacro/g!
    • Parenthesis sugar: as-> -> doto ->>
    • Others: assoc of defmain #@ comment cfor
  • Other functions (formerly core) to be included:
    • Iterables: distinct butlast drop-last rest flatten coll?
    • Others: inc dec xor parse-args constantly
• Deal with the remaining core macros:
  • Delete if (use if*, now called if, or cond instead).
  • Delete if-not (use (if (not …) …) or (unless …) instead).
  • Delete lif-not (use (lif (not …) …) instead).
  • Delete macro-error.
• Deal with the remaining core functions:
  • Delete all re-imports and aliases from Python standard libraries: fraction take-while drop-while remove reduce accumulate count cycle repeat *map chain compress group-by islice tee combinations multicombinations permutations product zip-longest.
    • We'll still need an alias of fraction for compiling fraction literals, but we don't need to advertise it to users.
  • Delete all functions that are redundant with the Python library Toolz. Toolz has frequently been discussed in relation to Hy, but for some unfathomable reason, we've reimplemented a lot of parts of it instead of just telling Hy users to import Toolz like a normal library. These functions are: comp complement identity iterate juxt merge-with first last interleave interpose nth partition take take-nth second drop.
  • Delete most of the predicate functions: empty? even? every? float? integer? integer-char? iterable? tuple? zero? some string? symbol? numeric? odd? neg? pos? none? iterator? keyword? list?
  • Also delete:
    • instance? - Use Python's isinstance instead.
    • keyword - Use hy.models.Keyword instead.
    • repeatedly - Use toolz.iterate instead.
  • Make calling-module and calling-module-name internal to Hy.

[allison-casey]

this is... a lot. It'll take a bit for me to really parse what you're proposing.

[Kodiologist]

TL;DR: Delete a bunch of core stuff that's pointless in various ways, spin off most of the rest into a new library, and do some cleanup of special forms.

[peaceamongworlds]

There are definitely parts of this that I strongly agree with, such as renaming the model types, re-evaluating if everything in core is still needed and used, and suggesting something like Toolz or cytoolz.

However, with a lot of the other stuff, I don't really have a strong opinion, and it will depend on the specific implementation details.

[Kodiologist]

That's fair.

[scauligi]

We should set up a document to keep track of the latest proposed changes as we discuss/refine/bikeshed them. Either updating the first comment here or a page in the wiki, perhaps?

We should also address #1723 with this, as part of the bikeshedding.

There are also some bugs with how with (can't mix assigning and assign-less forms properly) and lfor (scoping issues when it creates a temp function) that I've come across, I can bring them up here or in their own issue(s) for more visibility?

[scauligi]

If we get a Discord or other space set up (#1778) that would also be a great place to hash out the details of this plan!

[Kodiologist]

I was just going to edit the first comment.

I can bring them up here or in their own issue(s) for more visibility?

Those sound like bugs, and should have their own issues.

[allison-casey]

With regard to #1778, github now has discussions we can enable for this repo that have threads and everything. I think that would be a good place to start diving into this discussion.

[allison-casey]

Went ahead and enabled discussions for Hy! Go check it out when you have the chance. It's still pretty new so we'll be figuring it out as we go, but it seems pretty nice so far.

[Kodiologist]

Regarding SkyHy, I was also thinking of Hyrule as an appropriately cute name, although maybe that's better reserved for a library about rules, or Zelda games. I'm inclined to use "Hylian" as a word for "Hy programmer", in any case.

[allison-casey]

if with/a is becoming special, defn/a should absolutely as well.

And for SkyHy, i think we should follow the toolz and funcy organization of having submodules dedicated to specific categories, but everything available under the root level module if people want to just import/require everything like you can with toolz

[Kodiologist]

But defn/a is easy to write as a macro. We already have it as one, in fact.

[allison-casey]

its a core feature of async programming in python and I don't think its acceptable to make someone either write a macro or require it from a different package like skyhy just to do a core language feature. its an unnecessary hurdle to usability.

[Kodiologist]

You know that fn/a is special and will remain so, right?

[allison-casey]

yes

[allison-casey]

was the plan for if* -> if rename to bring in the elif functionality of the if macro into the special form version of if?

[Kodiologist]

No, it's just a rename. You can still use cond for linearized else-if ladders.

[allison-casey]

cond doesnt work the same as the current if macro so good to know

[allison-casey]

also, i think macro-error has already been made internal and undocumented as of either #2071 or #1979

[Kodiologist]

Seems to still be there to me:

hy 1.0a1+107.g93794e91 using CPython(default) 3.8.6 on Linux
=> (macro-error 5)
…
TypeError: macro_error() missing 1 required positional argument: 'reason'

[allison-casey]

ahhh, i see. its not in the documentation anywhere. It's just getting slurped up during bootstrap. Can we just get rid of it? We're already moving almost all the macros out of hy and the rest are getting moved to special forms anyways. Or just move it to skyhy 🤷‍♀️

[Kodiologist]

Yes, I think it would be better to do without it; one should be able to raise errors in a macro the same way as elsewhere in Python.

[allison-casey]

as @scauligi mentioned in #2050 now that defn is a special form i'm gonna put my weight behind their suggestion of moving the return annotation before the function name since its more consistent and lets us get rid of this weird exception to annotation formats

; Function return annotations have different ordering
(defn add1 ^int [^int x] (+ x 1))
(fn ^int [^int y] (+ y 2))

#2077 is already doing a lot so i'd probably just make it its own PR after that gets merged

[Kodiologist]

No objection here.

[Kodiologist]

This branch https://github.com/Kodiologist/hy/tree/hyrule is what I've done so far to remove Hyrule bits from Hy. At this point it's in pretty good shape except for the documentation. Next I'll make Hyrule itself, starting from a largely unaltered subset of a commit from Hy, and once I have Hy and Hyrule working together well, I'll open a PR to merge my hyrule branch of Hy.