not exporting symbols based on *package* breaks the hu.dwim codebase

[attila-lendvai]

commit 39d458f (from PR #7) breaks the hu.dwim codebase (i reverted it).

it fails to export the NAME-OF symbol from hu.dwim.walker, due to which an originally single generic method got split into two, the other one being a private/isolated NAME-OF internal to hu.dwim.meta-model.

or to put it another way: our codebase expects that symbols get exported, regardless of what is their home package [EDIT: i.e. the CL export semantics].

[Ambrevar]

I've thought about it a little.

One way to fix it would be simply to replace (package-name *package*) with (symbol-package sym), this way export would export the symbol from the right package, that is, possibly not the one the user is dealing with.

That said, this means that defining a class can change the export of a symbol from a foreign package. This seems to be a broken design, but as I understand this is what hu.dwim.walker expects. Can't we fix hu.dwim-walker instead?

[attila-lendvai]

that would still break the assumptions in our codebase.

here's how i am (we were) thinking about packages and export/import: when we :use a package, then all the exported symbols get imported into our package, regardless of what is the original home package of the symbol. this is also how CL packages work (AFAIR): the home package of a symbol does not influence the import/export semantics in any way.

when i ask defclass* to export something, then i expect it to export the symbol in question from the package that the definition is in. one lib mutating the package of another is a very bad idea. it would mean that whether lib A is loaded or not (or when it is loaded -- even worse!) would influence the behavior of the rest of the system -- that's a recipe for undebuggable, potentially nondeterministic bugs.

there was a time when we used asdf-system-connections, but the bugs drove us crazy and we got rid of it. we even switched to explicit lib-a+lib-b.asd ASDF systems to explicitly describe/handle/request cross-system interactions, and those interactions were only additive, rarely mutating; and in those rare cases with big fat warnings in the codebase, sometimes even at load time.

IMO, there's nothing to be "fixed" in hu.dwim.walker. it exports a name-of symbol, and the rest of the codebase, i.e. whoever uses walker, expects to share the same name-of symbol with every other such libs (and define e.g. generic methods on it).

this is a fundamental design decision/assumption. "fixing" that would mean a very time consuming audit of the entire codebase... and as i remember, about half of our debugging time/effort in CL was spent on package issues. i remember i ended up even patching SBCL to emit warnings/errors with fully qualified symbol names...

we even developed a joke/instinct that whenever something didn't make any sense, we concluded that it must be a package issue.

does this make sense?

[Ambrevar]

I agree with your assessment, but then there is still a bug I believe:

when i ask defclass* to export something, then i expect it to export the symbol in question from the package that the *definition* is in. one lib mutating the package of another is a very bad idea

Currently (without my patch), defclass-star tries to reexport symbols (i.e. mutate) in foreign packages. This is what my patch was meant to fix. See the recipe in #7 for a concrete example. The problem may lie in what you mean with "symbol from the package", because "cl-user:foo" is not a symbol from "foo-package" and there is no way to export it in "foo-package". (Indeed, "foo-package::foo" is a different symbol.) Can you share a minimal example that exhibits how hu.dwim.walker fails with the patch from #7 ?

[attila-lendvai]

i finally took the effort to understand your example, sorry for the lag.

i've pushed a new (failing) test that formally captures your example: 2698bd9

Can you share a minimal example that exhibits how hu.dwim.walker fails
with the patch from #7 ?

not easily. but it's something like this: your change filters everything in *symbols-to-export*, and due to that the hu.dwim.walker:name-of accessor doesn't get exported anymore from one of the packages that :use :hu.dwim.walker. walker exports name-of, a user package defines a userpkg::name slot, whose name-of accessor gets resolved to hu.dwim.walker:name-of (because it's present in userpkg), but exporting is skipped with your change.

The problem may lie in what you mean with "symbol from the package", because "cl-user:foo" is not a symbol from "foo-package" and there is no way to export it in "foo-package". (Indeed, "foo-package::foo" is a different symbol.)

my gut reaction is to make sure that the symbol is present in package before attempting to export it by first importing it if needed. i'm not sure about its implications though.

hrm, actually... what if all remains as it is now? and we just say that it's the users' responsibility to set up their packages so that this doesn't happen? i'm leaning towards this, because this is the solution with the least probability of surprises, and i don't think it requires much extra effort in a lot of use-cases.

what do you think?

[attila-lendvai]

i've pushed a commit that reverts it.

closing it, but feel free to reopen if the current state is still not satisfying.

[attila-lendvai]

my reasoning: in my experience package related bugs are part of the class of bugs that eat up most of my time as a CL programmer. my general strategy when it comes to packages is to try to keep surprises to the minimum. things should rather fail early and loud than introduce a subtle bug that takes ages to catch.