Merge pull request #2125 from scauligi/scoping

yet another let implementation

NEWS.rst

@@ -23,11 +23,17 @@ Other Breaking Changes
 
   * Functions: `butlast`, `coll?`, `constantly`, `dec`, `destructure`, `distinct`, `drop-last`, `end-sequence`, `flatten`, `inc`, `macroexpand-all`, `parse-args`, `pformat`, `postwalk`, `pp`, `pprint`, `prewalk`, `readable?`, `recursive?`, `rest`, `saferepr`, `walk`
   * Classes: `PrettyPrinter`, `Sequence`
-  * Macros: `#%`, `#:`, `->`, `->>`, `ap-dotimes`, `ap-each`, `ap-each-while`, `ap-filter`, `ap-first`, `ap-if`, `ap-last`, `ap-map`, `ap-map-when`, `ap-reduce`, `ap-reject`, `as->`, `assoc`, `cfor`, `comment`, `defmacro!`, `defmacro/g!`, `defmain`, `defn+`, `defn/a+`, `defseq`, `dict=:`, `do-n`, `doto`, `fn+`, `fn/a+`, `ifp`, `let`, `let+`, `lif`, `list-n`, `loop`, `ncut`, `of`, `profile/calls`, `profile/cpu`, `seq`, `setv+`, `smacrolet`, `unless`, `with-gensyms`
+  * Macros: `#%`, `#:`, `->`, `->>`, `ap-dotimes`, `ap-each`, `ap-each-while`, `ap-filter`, `ap-first`, `ap-if`, `ap-last`, `ap-map`, `ap-map-when`, `ap-reduce`, `ap-reject`, `as->`, `assoc`, `cfor`, `comment`, `defmacro!`, `defmacro/g!`, `defmain`, `defn+`, `defn/a+`, `defseq`, `dict=:`, `do-n`, `doto`, `fn+`, `fn/a+`, `ifp`, `let+`, `lif`, `list-n`, `loop`, `ncut`, `of`, `profile/calls`, `profile/cpu`, `seq`, `setv+`, `smacrolet`, `unless`, `with-gensyms`
 
 * The constructors of `String` and `FString` now check that the input
   would be syntactically legal.
 * `hy.extra.reserved` has been renamed to `hy.reserved`.
+* Hy scoping rules more closely follow Python scoping in certain edge cases.
+* `let` is now a core macro. Semantics of `let` have changed in certain
+  edge cases: Any assignment or assignment-like operations (`with`, `match`,
+  etc.) will *assign* to the let-bound name, while any definintions or
+  definition-like operations (`defn`, `defclass`, `import`) will *shadow* the
+  let-bound name (and will also continue to be defined after the `let`-scope ends).
 
 Bug Fixes
 ------------------------------
@@ -40,6 +46,9 @@ Bug Fixes
 * Fixed a bug with self-requiring files on Windows.
 * The `repr` and `str` of string models now include `brackets` if
   necessary.
+* Complex comprehensions are now always treated as inline expressions for
+  variable scoping, rather than creating a new block scope. Specifically,
+  assignments within comprehensions are now always visible to the surrounding code.
 
 New Features
 ------------------------------

docs/api.rst

@@ -725,12 +725,12 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
    - ``:if CONDITION``, which is equivalent to ``:do (if (not CONDITION)
      (continue))``.
 
-   For ``lfor``, ``sfor``, ``gfor``, and ``dfor``, variables are scoped as
-   if the comprehension form were its own function, so variables defined by
-   an iteration clause or ``:setv`` are not visible outside the form. In
-   fact, these forms are implemented as generator functions whenever they
-   contain Python statements, with the attendant consequences for calling
-   ``return``. By contrast, ``for`` shares the caller's scope.
+   For ``lfor``, ``sfor``, ``gfor``, and ``dfor``,  variables defined by
+   an iteration clause or ``:setv`` are not visible outside the form.
+   However, variables defined within the body, such as via a ``setx``
+   expression, will be visible outside the form.
+   By contrast, iteration and ``:setv`` clauses for ``for`` share the
+   caller's scope and are visible outside the form.
 
 .. hy:function:: (dfor [binding iterable #* body])
 
@@ -817,6 +817,67 @@ base names, such that ``hy.core.macros.foo`` can be called as just ``foo``.
        ...  (print x "is greater than 0"))
        3 is greater than 0
 
+
+.. hy:function:: (let [bindings #* body])
+
+   ``let`` creates lexically-scoped names for local variables. This form takes a
+   list of binding pairs followed by a *body* which gets executed. A let-bound
+   name ceases to refer to that local outside the ``let`` form, but arguments in
+   nested functions and bindings in nested ``let`` forms can shadow these names.
+
+
+   :strong:`Examples`
+
+   ::
+
+       => (let [x 5   ; creates new local bound names 'x and 'y
+                y 6]
+       ...  (print x y)
+       ...  (let [x 7]  ; new local and name binding that shadows 'x
+       ...    (print x y))
+       ...  (print x y))  ; 'x refers to the first local again
+       5 6
+       7 6
+       5 6
+
+   ``let`` can also bind names using
+   Python's `extended iterable unpacking`_ syntax to destructure iterables::
+
+       => (let [[head #* tail] (, 0 1 2)]
+       ...   [head tail])
+       [0 [1 2]]
+
+   Basic assignments (e.g. ``setv``, ``+=``) will update the local
+   variable named by a let binding when they assign to a let-bound name.
+   But assignments via ``import`` are always hoisted to normal Python
+   scope, and likewise, ``defn`` or ``defclass`` will assign the
+   function or class in the Python scope, even if it shares the name of
+   a let binding. To avoid this hoisting, use
+   ``importlib.import_module``, ``fn``, or ``type`` (or whatever
+   metaclass) instead.
+
+   Like the ``let*`` of many other Lisps, ``let`` executes the variable
+   assignments one-by-one, in the order written::
+
+       => (let [x 5
+       ...       y (+ x 1)]
+       ...   (print x y))
+       5 6
+
+       => (let [x 1
+       ...      x (fn [] x)]
+       ...   (x))
+       1
+
+   Note that let-bound variables continue to exist in the surrounding
+   Python scope. As such, ``let``-bound objects may not be eligible for
+   garbage collection as soon as the ``let`` ends. To ensure there are
+   no references to ``let``-bound objects as soon as possible, use
+   ``del`` at the end of the ``let``, or wrap the ``let`` in a function.
+
+   .. _extended iterable unpacking: https://www.python.org/dev/peps/pep-3132/#specification
+
+
 .. hy:function:: (match [subject #* cases])
 
    The ``match`` form creates a :ref:`match statement <py3_10:match>`. It

docs/cheatsheet.json

@@ -76,6 +76,7 @@
           "sfor",
           "setv",
           "setx",
+          "let",
           "match",
           "defclass",
           "del",

docs/whyhy.rst

@@ -132,7 +132,7 @@ the same time, Hy goes to some lengths to allow you to do typical Lisp things
 that aren't straightforward in Python. For example, Hy provides the
 aforementioned mixing of statements and expressions, :ref:`name mangling
 <mangling>` that transparently converts symbols with names like ``valid?`` to
-Python-legal identifiers, and a :hy:func:`let <hy.contrib.walk.let>` macro to provide block-level scoping
+Python-legal identifiers, and a :hy:func:`let` macro to provide block-level scoping
 in place of Python's usual function-level scoping.
 
 Overall, Hy, like Common Lisp, is intended to be an unopinionated big-tent

hy/compiler.py

@@ -1,4 +1,4 @@
-import ast, copy, importlib, inspect, keyword, pkgutil
+import ast, copy, importlib, inspect, keyword
 import traceback, types
 
 from funcparserlib.parser import NoParseError, many
@@ -10,6 +10,7 @@
 from hy.errors import (HyCompileError, HyLanguageError, HySyntaxError)
 from hy.lex import mangle
 from hy.macros import macroexpand
+from hy.scoping import ScopeGlobal
 
 
 hy_ast_compile_flags = 0
@@ -69,21 +70,42 @@ def _dec(fn):
 # Provide asty.Foo(x, ...) as shorthand for
 # ast.Foo(..., lineno=x.start_line, col_offset=x.start_column) or
 # ast.Foo(..., lineno=x.lineno, col_offset=x.col_offset)
+# Also provides asty.parse(x, ...) which recursively
+# copies x's position data onto the parse result.
 class Asty:
+    POS_ATTRS = {
+        'lineno': 'start_line',
+        'col_offset': 'start_column',
+        'end_lineno': 'end_line',
+        'end_col_offset': 'end_column',
+    }
+
+    @staticmethod
+    def _get_pos(node):
+        return {
+            attr: getattr(node, hy_attr,
+                          getattr(node, attr, None))
+            for attr, hy_attr in Asty.POS_ATTRS.items()
+        }
+
+    @staticmethod
+    def _replace_pos(node, pos):
+        for attr, value in pos.items():
+            if hasattr(node, attr):
+                setattr(node, attr, value)
+        for child in ast.iter_child_nodes(node):
+            Asty._replace_pos(child, pos)
+
+    def parse(self, x, *args, **kwargs):
+        res = ast.parse(*args, **kwargs)
+        Asty._replace_pos(res, Asty._get_pos(x))
+        return res
+
     def __getattr__(self, name):
-        setattr(Asty, name, staticmethod(lambda x, **kwargs: getattr(ast, name)(
-            lineno=getattr(
-                x, 'start_line', getattr(x, 'lineno', None)),
-            col_offset=getattr(
-                x, 'start_column', getattr(x, 'col_offset', None)),
-            end_lineno=getattr(
-                x, 'end_line', getattr(x, 'end_lineno', None)
-            ),
-            end_col_offset=getattr(
-                x, 'end_column', getattr(x, 'end_col_offset', None)
-            ),
-            **kwargs)))
+        setattr(Asty, name, staticmethod(
+            lambda x, **kwargs: getattr(ast, name)(**Asty._get_pos(x), **kwargs)))
         return getattr(Asty, name)
+
 asty = Asty()
 
 
@@ -195,7 +217,7 @@ def expr_as_stmt(self):
             return Result() + asty.Expr(self.expr, value=self.expr)
         return Result()
 
-    def rename(self, new_name):
+    def rename(self, compiler, new_name):
         """Rename the Result's temporary variables to a `new_name`.
 
         We know how to handle ast.Names and ast.FunctionDefs.
@@ -204,7 +226,7 @@ def rename(self, new_name):
         for var in self.temp_variables:
             if isinstance(var, ast.Name):
                 var.id = new_name
-                var.arg = new_name
+                compiler.scope.assign(var)
             elif isinstance(var, (ast.FunctionDef, ast.AsyncFunctionDef)):
                 var.name = new_name
             else:
@@ -302,9 +324,11 @@ def __init__(self, module, filename=None, source=None):
         # compilation.
         self.module.__dict__.setdefault('__macros__', {})
 
-    def get_anon_var(self):
+        self.scope = ScopeGlobal(self)
+
+    def get_anon_var(self, base="_hy_anon_var"):
         self.anon_var_count += 1
-        return "_hy_anon_var_%s" % self.anon_var_count
+        return f"{base}_{self.anon_var_count}"
 
     def compile_atom(self, atom):
         # Compilation methods may mutate the atom, so copy it first.
@@ -422,6 +446,8 @@ def _storeize(self, expr, name, func=None):
             new_name = typ(elts=new_elts)
         elif isinstance(name, ast.Name):
             new_name = ast.Name(id=name.id)
+            if func == ast.Store:
+                self.scope.assign(new_name)
         elif isinstance(name, ast.Subscript):
             new_name = ast.Subscript(value=name.value, slice=name.slice)
         elif isinstance(name, ast.Attribute):
@@ -549,7 +575,8 @@ def compile_symbol(self, symbol):
             return asty.Constant(symbol, value =
                 ast.literal_eval(mangle(symbol)))
 
-        return asty.Name(symbol, id=mangle(symbol), ctx=ast.Load())
+        return self.scope.access(asty.Name(
+            symbol, id=mangle(symbol), ctx=ast.Load()))
 
     @builds_model(Keyword)
     def compile_keyword(self, obj):
@@ -783,7 +810,8 @@ def hy_compile(
         # Import hy for compile time, but save the compiled AST.
         stdlib_ast = compiler.compile(mkexpr("eval-and-compile", mkexpr("import", "hy")))
 
-    result = compiler.compile(tree)
+    with compiler.scope:
+        result = compiler.compile(tree)
     expr = result.force_expr
 
     if not get_expr: