import foo {.all.}: import private symbols

[timotheecour]

supersedes #11865 which was ready for ~2 years but which I had to re-implement from scratch because #16550 changed too many things.

This represents a lot of effort, I hope this PR can be merged soon to avoid rebasing/bitrot/re-implementing many times.

• closes [superseded] import foo {.all.} #11865
• see linked discussion, rationale in [superseded] import foo {.all.} #11865
• closes Enable cross-module initialization of private fields RFCs#145
• closes https://forum.nim-lang.org/t/4658
• closes https://forum.nim-lang.org/t/4293
• closes https://forum.nim-lang.org/t/4296
• closes https://forum.nim-lang.org/t/7193
• fixes Nim should allow getField(someObj, someField) to access private fields; use case: custom serialization/deserialization RFCs#106 via scopable privateAccess

rfc

• closes RFC import foo {.all.} RFCs#299

details

• proc privateAccess*(t: typedesc) enables access to private fields of t in current scope
• import foo {.all.} enables access to private symbols of foo, plus all existing import syntaxes are supported. Private fields of symbols from foo are not exposed; privateAccess can be used for that

this gives maximum flexiblity, allowing by-passing of symbol/field visibility in a given scope without affecting other clients of the imported module / type, and without any of the gotchas that come with include.

note to reviewer

• there's a lot of files in this PR, but they're mostly tests

future work

• avoid implicit c variable with dual meaning, see getC, onUse, onDef, onDefResolveForward in compiler/modulegraphs.nim
• integrate styleCheckDef in onDef/onDefResolveForward
• improve the error message to "symbol module.X exists but is private".
• evaluate whether a single table approach works better (potentially would simplify code a bit, but symbol lookup performance in the common case of import foo (not import foo {.all.}) should be evaluated
• use isDefined(nimDebug) for import foo {.all.}: import private symbols #17706 (comment) (=> -d:nimDebug: calls doAssert false instead of quit in compiler code #17739)

[Araq]

This represents a lot of effort, I hope this PR can be merged soon to avoid rebasing/bitrot/re-implementing many times.

I'm behind this language extension and will help out with the implementation, if required.

[Araq]

This lacks IC support. ;-)

[Araq]

Too ugly... Can we find a better solution?

[timotheecour]

this is pre-existing and I've strictly improved things in this PR via getC by avoiding this:

    when compiles(c.c.graph):
      if c.c.graph.onUsage != nil: c.c.graph.onUsage(c.c.graph, s, info)
    else:
      if c.graph.onUsage != nil: c.graph.onUsage(c.graph, s, info)

which had compiles and was not DRY

But I've added a future work item to further improve this.

[Araq]

It was only code for "nimfind" which is dead code now that we got "nim check --defusages"

[timotheecour]

@Araq i see you've removed nimfind in #17737 (good or bad, I'm not sure), but please don't remove onDef + friends, they're very useful during debugging (and i wrote a competitor tool of nimfind that added more features, which relied on those onDef, onUsage etc). It's very useful to have 1 place defined that catches all definition/usage events.

[Araq]

I prefer to distinguish between nkSym and nkSymDef instead but it would break the macro system until we have an AST to AST translation step. :-/

[Araq]

Also ... what if every top level symbol is in the "exported" scope and then there is another filtering step? This could also be used to improve the error message to "symbol module.X exists but is private". I think this would simplify the logic quite a bit.

[timotheecour]

PTAL

(btw, in code comments (like #17706 (comment)) instead of at PR level (like #17706 (comment)) are easier to tackle, so that threading is maintained)

Also ... what if every top level symbol is in the "exported" scope and then there is another filtering step? [...] I think this would simplify the logic quite a bit.

I thought about this but let's please defer this to future work after evaluating that it doesn't hurt CT performance (because the common case is import foo, not import foo {.all.}, and symbol lookup in a small table should be faster than in a larger table).

This could also be used to improve the error message to "symbol module.X exists but is private".

ya, that's another benefit i had listed in #11865; but note that this can be done even with the 2-tables approach

i've added those 2 items in future work section

[timotheecour]

PTAL

[Araq]

Merging this for now but the feature itself might not be the best idea. It must be documented as "experimental".