wrapnils: allows optional chaining of field access and indexing when LHS i snil

[timotheecour]

(supersedes #13014)
This PR introduces a proc maybe for optional chaining, which has similar counterparts in other languages (eg, it's ubiquitous in swift, see https://docs.swift.org/swift-book/LanguageGuide/OptionalChaining.html)

very often, when debugging (or other cases), we need to special case field access to make sure a ref/ptr object is not nil, leading to clumsy code, eg:

type PSym = ref TSym
proc fun(s: PSym, iter: int) =
  var msg: string
  if s != nil:
    msg.add "name:", s.name.s & "kind: " s.kind
  else:
    msg.add "s is nil"
  echo msg & "iter:" & $iter

this PR allows much simpler code instead, that allows intermediate LHS values to be nil (in which case it returns default for the field) in field access and indexing:

type PSym = ref TSym
proc fun(s: PSym, iter: int) =
  echo (isnil: s == nil, name: s.maybe.foo[12].bar.name[], kind: s.maybe.kind[], iter: iter)

EDIT: example showing behavior on procs

/cc @Araq IMO this is the desired, common case, behavior

# this works even if n.typ == nil; in which case `typeToString` is NOT called
echo (dest.kind, n.typ == nil, n.typ.maybe.typeToString[])

note1:

supersedes #13014 ; advantages:

• requires no parser change (whereas .? and ?. had issues with foo.?bar.baz etc, which would require parser/lexer changes)
• works with [] indexing (hard to do with .?; any of these ?[], [?], []? would require parser/lexer changes)

note2:

I designed it to be recursive (in the sense that operations on a Maybe return a Maybe, and at the end of the chain you can add [] to get back the value); IMO it's the most commonly useful behavior; we could introduce a non-recursive variant (which hence wouldn't need [] at the end) in further PR and it's trivial to do so (that was in fact my original version, but new version is better)

Eg for long chains, maybe is needed only once, at the 1st place where some value can be nil:

# say foo isn't nil but foo.bar is maybe nil:
foo.bar.maybe.x1.x2.x3[] # everything after foo.bar can be nil

# the non-recursive variant would've been:
foo.bar.maybe.x1.maybe.x2.maybe.x3

obviously user has full control eg to terminate chain early:

echo foo.maybe.x1[].x2

note3:

• despite superficial similarities, this is different from std/options; std/options just can't be used for this use case
• I chose to put it in a new module instead of std/sugar (no real harm to having more modules especially bc of lack of cyclic imports it's actually better)

future work

• revisit un-exporting of wrapnil, unwrap, isValid
  EDIT => wrapnils: add ??. which returns an Option #16931
• followup: major improvements to std/wrapnils: optimal codegen, case objects, lvalue semantics #18435 (optimal codegen and support for case objects)

[Araq]

Now that's a work of genius (no irony!). We can bikeshed about the name though... :-)

[timotheecour]

PTAL

[Araq]

.maybe is not the best name for this IMO, how about:

• nilToDef
• nocrash
• nildef
• notnil

[timotheecour]

how about:
nilwrap? (note: not nilWrap; it's easier to type all small case)
and then i replace all occurrences of maybe by nilwrap

also see note I added above regarding example showing behavior on procs

(notnil already has some kind of meaning; nocrash is a misnomer since it can still crash for other reasons (eg function calls that crash for unrelated reasons); nildef/nilToDef sounds like definition, not default)

if not nilwrap, I'd be ok with: nilsafe

[juancarlospaco]

notnil 👍

[disruptek]

defuse

[timotheecour]

notnil is not a good name, there's a different concept already called NotNil: see nimNotNil, tfNotNil etc; we shouldn't conflate these

[Araq]

I like nilwrap (don't forget to change the module's name to nilwraps and the changelog entry).

[timotheecour]

done PTAL

[Araq]

Wait a sec ... why nilwrap? Shouldn't this be wrapnil?

[disruptek]

denil

[timotheecour]

Wait a sec ... why nilwrap? Shouldn't this be wrapnil?

done; hopefully ends the bikeshed ;-)

[Araq]

auto return type is bad for documentation generator, speaking of which docgen (kochdocs) is not updated and this need a big doc comment saying this module is experimental and also relying on experimental Nim features.

Slightly offtopic, the rule we are following now is that new std modules start as experimental and are only in std so that we don't have a silly "deprecation" period should they turn out to be good enough...

[timotheecour]

PTAL

• auto return type is bad for documentation generator
  done (question: since nim doc runs nim, can't it infer it?)

• speaking of which docgen (kochdocs) is not updated
  done (IMO there should be a CI test (that runs as early as possible in CI) to make sure all modules are covered; instead of relying on such kind of knowledge

• and this need a big doc comment as being experimental and also relying on experimental Nim features.
  done

Slightly offtopic, the rule we are following now is that new std modules start as experimental and are only in std so that we don't have a silly "deprecation" period should they turn out to be good enough...

for future PR's how about the following fool-proof rule:
new module mymod requires a experimental flag: --experimental:std:mymod
then when we figure out the module is good enough we just make that flag default.
This avoids any shuffling around of files, and avoids any renames in user code (unlike what we have with lib/experimental/diff.nim and import experimental/diff)

[Araq]

question: since nim doc runs nim, can't it infer it?

In theory it could but for Nim's semi-broken generics it's even harder.

done (IMO there should be a CI test (that runs as early as possible in CI) to make sure all modules are covered; instead of relying on such kind of knowledge

kochdocs should use some globs. (I know you had such a PR and krux02 closed it, without my consent.)

new module mymod requires a experimental flag: --experimental:std:mymod
then when we figure out the module is good enough we just make that flag default.

I think that's not necessary and only more annoying. Instead we should document that std is mostly about namespacing modules and that both "standard" modules end up there and also experimental ones. The experimental ones will always have a comment saying that (and obviously they are candidates for the stdlib).

[timotheecour]

PTAL
I had to add a field validImpl to distinguish valid from invalid values: the wrapnil object becomes invalid (and stays invalid) as soon as a nil is found on the chain.
Without this, some tests in test suite would fail.
The test suite reflects the most useful semantics IMO.

[Araq]

Please bare with me but here are more remarks to be discussed, they don't imply actions on the PR itself:

• There is a weird asymetry between wrapnil and [] (deref to get out of the wrapnil context?). what if we use a wrap(x) instead. Needs the () though then.
• The primary use case is "safe access in testing for equality". So should this perhaps be a class of operators like ==?, <?, <=??

For example:

doAssert a.x2.x2.x3[1] ==? default(char)

Or maybe just

doAssert ?!a.x2.x2.x3[1] == default(char)

If you want to end the nil oblivious evaluation rules earlier, use ():

doAssert ?!(a.x2).x2.x3[1] == default(char)

I like ?! best because it sticks out and is unlikely to clash with any existing usages.

[timotheecour]

• I added deref for the uncommon case where a system.[] is needed: it's uncommon because field access don't require to first deref, but there is 1 case where it is needed: proc call that expects a deref'd input, see new test in ce296cb

• There is a weird asymetry between wrapnil and [] (deref to get out of the wrapnil context?)
  it's the exact same as in typeinfo: toAny / [].

I'm optimizing for the common case (usually you'll want to unwrap at end of chain), hence used a familiar (existing) operator
We can however add get to make it more flexible, as in options.get:

template `[]`*(a: Wrapnil): untyped = a.valueImpl
template get*(a: Wrapnil): untyped = a.valueImpl  # but also throws if not a.validImpl (as in `options.get`)
template get*(a: Wrapnil, b: defaultVal): untyped = a.valueImpl

The primary use case is "safe access in testing for equality"

# 1: safe echo / msgs / debug
echo (x1.wrapnil.x2[], ...)
# 2: safe testing: will assert with desired msg instead of segfault
doAssert x1.wrapnil.x2[] == val, additionalContext
# 3: safe delayed testing, using unittest.check: will allow testing to continue (keeping the err checkpoint) instead of segfault
check x1.wrapnil.x2[] == val
# 4: safe branching
if x1.wrapnil.x2[] == val: foo

what if we use a wrap(x) instead. Needs the () though then

not clear; how would u translate: x1.x2.wrapnil.x3.x4[].baz using wrap?
It suffers from same issues as $ which violates the natural left-to-right evaluation chains, making it awkward to use when what you're stringifying is not "the whole expression" (and would additionally require quoting).
Wheres with x1.x2.wrapnil.x3.x4[].baz it's clear where the wrapping/unwrapping happens

in any case I bet wrap can be written on top of primitives from this PR (via a macro), but I don't think it's simpler to use.

==?, <?, <=?

why are these needed? this works:

check x1.wrapnil.x2[] == val
check x1.wrapnil.x2[] > val

[Araq]

Ok, let me try again:

What downside does a unary operator have? It sticks out as "here is something fishy going on" and it's clear what it applies to without the need for deref.

Some examples from your tests, rewritten with ?! (which is what I'm proposing):

  doAssert ?!a.x2.x2.x1 == 0.0
  doAssert ?!a3.x2.x2.x5.len == 0
  doAssert ?!a.x2.x2.x3[1] == default(char)
  echo ?!a.b.c.d

[timotheecour]

compromise: I've un-exported wrapnil, isValid, unwrap for now to unblock this PR but kept them as private procs so I don't loose the work and this can be debated later;

As I tried to explain in #13023 (comment), ?. does not distinguish between

• invalid value (ie, had intermediate nils)
• default value that happens to be default (which is perfectly valid, eg an int that's 0)

This is the same rationale as in std/options which has an API to distinguish invalid from valid (even if valid-but-default):

import std/options

let x1 = some(0)
echo x1.isSome # true even if valid but default
echo x1.get(0)

let x2 = none(int)
echo x2.isSome # false
echo x2.get(0)

the example use is

let z = x1.wrapnil.x2.x3.level
if z.isValid:
  echo z.unwrap
else: # do smthg else, maybe throw with a useful context, or handle differently

[doongjohn]

# nim v1.2.6
# Windows 10 x64

type Person = ref object
  name: string
  age: int

var person: Person = nil

echo ?.person.name != "" # false
echo ?.person.name != "no name" # true
echo ?.person.age != 0 # false

echo ?.person.name != "no name" # true Is this intended?

[timotheecour]

Yes, why is it surprising? There is a nil in the chain, so it returns default(string) (string is the type of the expression)

[doongjohn]

Oh, I thought if it's null then the result will always be false.

[doongjohn]

What do you think about adding ??. for returning Option[T]?

# wrapnils.nim
# ...

# this returns Option[T] instead of default value.
macro `??.`*(a: untyped): untyped =
  result = replace(a)
  result = quote do:
    if `result`.validImpl:
      some(`result`.valueImpl)
    else:
      none(`result`.valueImpl.type)

# Option helper
# just to make things simpler :)

proc `==`*[T](a: Option[T], b: T): bool {.inline.} =
  if a.isNone: return false
  return a.get() == b

proc `!=`*[T](a: Option[T], b: T): bool {.inline.} =
  if a.isNone: return true
  return a.get() != b

type Person = ref object
  name: string
  age: int

var john = Person(name: "John", age: 20)
echo ??.john.name == "John" # true
echo ??.john.age != 20 # false
john = nil
echo ??.john.name == "" # false
echo ??.john.age != 0 # true

[timotheecour]

@doongjohn see #16931 which should solve your problem; note that == and != as you described are IMO not a good idea so I didn't do that.