Strange behavior or #docs / __macros__ and __tags__

[scauligi]

Minimum viable test case, must be run from its own file (not from the REPL):

;; (deftag bogus [-] -)  ; Uncomment to get help for `None`
;; (macroexpand '#@ (foo (defn bar [])))  ; Uncomment to get help for `#@`
#doc @

Ok so this one is a little weird but I think I know why it's happening.
TL;DR Hy patches in __macros__ and __tags__ to a module only on an as-needed basis and only during compilation. As a result, you can end up with Hy modules that don't have these defined.

Observed symptoms:

• If you run the above code, the #doc @ fails with '__tags__' is not defined.
• If you uncomment the (deftag), the module now defines a __tags__, but it's only populated with the tag bogus — thus the #doc @ brings up the help page for None.
• If you uncomment the (macroexpand), then Hy finally brings the @ tag macro into __tags__, and you get the proper help page.

Further weirdness!
Create two files like so:

;; file requireme.hy
(defn bogus [-] -)
#@(bogus (defn foo []))

;; file runme.hy
(require requireme)
#doc @

When you run runme.hy, you get the help page for None.
Without changing anything, run it again — now it fails with '__tags__' is not defined!

On the first run the compiler patches in __tags__ and __macros__ for the compilation process, but every subsequent run skips compilation and loads the pyc files from the pycache — and no code was ever produced to actually create a __tags__ in runme.hy.

So far I only foresee this as a weird edge case of #doc and don't expect this leading to any real bugs, but it does produce confusing behavior for anyone doing introspection on Hy modules.

Possible solutions:

• Always add __macros__ and __tags__ to every Hy module at runtime, possibly via the import hy statement, and pre-populate with the core macros/tags.
• Document that #doc has strange behavior if you use it outside the REPL (and also fix the test-doc test, that's how I stumbled across this).
• Something else?

[allison-casey]

I've had tests fail intermittently with '__tags__' is not defined! as well and only fixable by rm -r-ing all the __pycache__ folders. So this may actually be less of an edge case than your think. I'm not sure what the best solution is. I think this one's gonna take some digging

[allison-casey]

With the tag macro simplification, running pytest twice no longer fails with '__tags__' is not defined, but:

      (doc "#@")
      (setv [out err] (.readouterr capsys))
>     (assert (in "with-decorator tag macro" out))
E     AssertionError

in tests/native_tests/core.hy will have to dig a bit deeper it seems.

[scauligi]

Yeah the problem is the disconnect between macros getting patched in at compile time vs macros that get compiled into the module's .pyc file.

A related issue that illustrates this:

;; module_trap.hy
(defn toot_fn [s] (+ "fn TOOT " s))
(defmacro toot_mac [s] `(+ "mac TOOT " ~s))

(print (toot_fn "self"))
(print (toot_mac "self"))

;; confused.hy
(require module-trap)
(import module-trap)

(print (module-trap.toot_fn "imported"))
(print (module-trap.toot_mac "imported"))

$ hy confused.hy

fn TOOT self
mac TOOT self
fn TOOT imported
mac TOOT imported

$ sed -i 's/TOOT/bloot/' module_trap.hy
$ hy confused.hy

fn bloot self
mac bloot self
fn bloot imported
mac TOOT imported    # <-- !!

$ touch confused.hy
$ hy confused.hy

fn bloot self
mac bloot self
fn bloot imported
mac bloot imported

Updating the module doesn't affect macros in confused.hy until it gets recompiled (from being newer than the generated .pyc).

[scauligi]

Closed with #1971.