how to support user-defined casts?

[mppf]

Spin-off from #5054. Related to #15838.

Currently, one can define support for a cast between two types by defining a _cast(type toType, value) function. However we would like to have a better user-facing way to do this.

Q1. If a user-defined record has an init= from another type, say proc R.init=(other: Q), then should the compiler be willing to invoke that to implement myQ: R ?

Q2. If a user-defined record has a proc = from another type, say proc =(ref lhs: R, const ref rhs: Q), then should the compiler be willing to invoke that to implement myQ: R ? (Note that if we generate init= from = or use = instead of init= when necessary - as issue #15838 proposes - this would fall out from Q1 in some cases).

Q3. Should there be a way to implement a cast only (i.e. without also = or initialization) to support cases where the assignment/initialization should not be allowed but where casts should be?

If the answer to Q3 is "yes", how would one write that?
Q3a. proc : (value: Q, type toType: R)
Q3b. proc cast(value: Q, type toType: R)
Q3c. proc Q.castToType(type toType: R) and proc R.init:(other: Q) to implement cast out of the user-defined type and cast into the user-defined type

[bradcray]

Q1: I tend to think "yes" since the two operations seem very similar to me: create something new of type R from something pre-existing of type Q.

Q2: It seems to me that a = should not support casts since that operator is about overwriting an existing instance of R and this is about creating a new R. If there were a compiler-generated init= that had the right signature, then I agree we're OK because we're back in case 1.

Q3: Without a good motivating case for this, I'd be disinclined to support it (or, at least, to not worry about supporting it until such a case comes up). Specifically, it doesn't seem like it would be a breaking change to implement it later (assuming that one could define an error overload of the cast-only operation that disabled the cast, say).

If/when we did need to support it, as long as operators are implemented as standalone functions, I think Q3a seems natural, though it's interesting that such casts may be more closely associated with the toType than with R. This makes it an interesting case w.r.t. the operator design issues that @lydia-duncan is wrestling with. For example, using a method syntax, proc R.:(type toType) seems like a natural syntactic fit, yet if the cast should be associated with toType, we'd presumably want to define it using a reverse-order operation style (?).

[mppf]

Thanks for your thoughts.

Re Q2 and Q3 - one reason we might want those features is if we want to support casts to types that don't support init= or to types that the author has no control over.

For example, suppose I am writing a record R. I might wish to support a cast from R to int but that's not possible to implement with init= as it stands. (Yes, we could consider allowing a "tertiary" int.init= but it is my current opinion that we shouldn't allow tertiary init= and so I lean against making a cast design that requires it). But if = is used for these (i.e. Q2 is "yes"), we are stuck with default-init and then assign (which would be fine for int specifically).

Default-init and then assign is not great for an array, though. So if I were writing a record MyArray and I wanted to support a cast to an array type, that's the case I'd expect to need proc cast or whatever form "yes" to Q3 takes.

Re. whether we could put some or all of this off - we could but I think we should try to make sure we have a consistent design. We already have _cast so it seems reasonable to replace that with a user-facing feature on its own. But it would also allow us to gain confidence in some other decisions that can be breaking (e.g. whatever we decide for mixed-type init= vs = errors (#15838) or how tertiary methods and operators interact with visibility (#16725) ).

[mppf]

I have another example motivating Q3 being "yes".

Consider the case brought up in issue #15806 :

record Point {
  var x: real;
  var y: real;
}

proc =(ref arr: [?d], ref r: Point) {
  arr[d.first] = r.x;
  arr[d.first+1] = r.y;
}

var arr: [0..1] real;
var p = new Point(10,20);

// arr[0] = 0;  // uncomment this and it works

arr = p;

writeln(arr);

The arr = p is a split init, which currently fails (with a poor error message). In issue #15838 we are proposing that this would be an error (since there is no array.init=(Point) ) - or that it would fall back on the = function (in which case it would default-initialize the array and then assign into it). Neither of those is great if we care about users implementing this pattern. The first prevents the pattern outright and the second is not ideal for performance due to the default-init and assign.

But, if the answer to Q3 is "yes", the user could be asked to provide a proc cast to go along with proc =. Then we could arrange for the split-init to run the proc cast as an alternative to array.init=(Point) which is not found. That avoids the user needing to create a "tertiary"init= on array.

In particular, in the case above, they could write

proc cast(value: Point, type toType: []) {
  var arr: [0..1] real = [value.x, value.y];
  return arr;
}

[lydia-duncan]

I think there's some typos in the original proposal? Either that or I'm misunderstanding.

Q1. If a user-defined record has an init= from another type, say proc R.init=(other: Q), then should the compiler be willing to invoke that to implement myR: Q ?

I think this should be myQ: R, right? Since the end result of R.init= is an instance of R? My instinct would be yes, but I think I'm getting confused by other aspects of the proposal.

I think Q3a seems natural, though it's interesting that such casts may be more closely associated with the toType than with R. This makes it an interesting case w.r.t. the operator design issues that @lydia-duncan is wrestling with. For example, using a method syntax, proc R.:(type toType) seems like a natural syntactic fit, yet if the cast should be associated with toType, we'd presumably want to define it using a reverse-order operation style (?).

I'm actually having a hard time visualizing the contents of the functions proposed by Q3, and I think it has to do with what Brad's saying here. I would expect a cast function focused on a specific type A to write to that type from the specified type B, rather than the other way around. In other words, I would expect the return value of these functions to be of type R, but calling the argument toType makes me think the return value is of type toType.

Maybe it's just the genericness that is throwing me off - I don't know that you could write something truly generic here, there's just too much variety in the scope of types that could be defined to be able to truly predict how to convert to any arbitrary type. That's still somewhat true in the opposite direction but at least there it is obvious to me how you could lean on interfaces to ensure you have some strategy for getting expected information out of the other type.

[mppf]

@lydia-duncan

I think there's some typos in the original proposal? Either that or I'm misunderstanding.

Q1. If a user-defined record has an init= from another type, say proc R.init=(other: Q), then should the compiler be willing to invoke that to implement myR: Q ?

I think this should be myQ: R, right? Since the end result of R.init= is an instance of R?

Yes it's an error in the proposal. Thanks for pointing it out. I've fixed it.

[mppf]

@lydia-duncan

I would expect the return value of these functions to be of type R, but calling the argument toType makes me think the return value is of type toType.

I think the issue description is misleading - these signatures should use Q to make sense with the previous text. I will fix.

I'm actually having a hard time visualizing the contents of the functions proposed by Q3, and I think it has to do with what Brad's saying here.

I will try showing each of them with an example. Suppose we want to implement cast for myInt : bigint and myBigint : int (where we are supposing the 2nd of these throws if the integer does not fit).

Q3a

// support myInt : bigint
proc : (value: int, type toType: bigint) {
  return new bigint(value);
}
// support myBigint : int
proc : (value: bigint, type toType: int) throws {
  return value.convertToIntOrThrow();
}

Q3b

(Just the same as the above but with cast as the function name instead of :.

// support myInt : bigint
proc cast(value: int, type toType: bigint) {
  return new bigint(value);
}
// support myBigint : int
proc cast(value: bigint, type toType: int) throws {
  return value.convertToIntOrThrow();
}

Q3c

// support myInt : bigint
proc bigint.init:(value: int) {
  return new bigint(value);
}
// support myBigint : int
proc bigint.castToType(type toType: int) throws {
  return this.convertToIntOrThrow();
}

[lydia-duncan]

Thanks, that clarifies it a lot!

[bradcray]

we could consider allowing a "tertiary" int.init= but it is my current opinion that we shouldn't allow tertiary init= and so I lean against making a cast design that requires it).

What's the cause of your disinclination to supporting tertiary init= as compared to casts? (given that both are ways to create a new int from an R, say where the author doesn't control ints?) Is it due to the way they're (currently) defined (i.e., standalone functions for casts vs. methods for init=)? Or is it because you think people will want to be able to write myR: int without also enabling var x: int = myR;? Or something else?

[mppf]

What's the cause of your disinclination to supporting tertiary init= as compared to casts?

There are two reasons (see below).

Is it due to the way they're (currently) defined (i.e., standalone functions for casts vs. methods for init=)?

Yes, the init= being a method has to do with it

Or is it because you think people will want to be able to write myR: int without also enabling var x: int = myR;?

I am not worried about that, no.

My reasoning:

Writing the tertiary initializer

First, we don't have initializers for int or similar types. I'm not convinced that enabling those is actually useful, because they would look different from other initializers. For example, supposed I am writing int.init=. What do I actually write?

proc int.init=(rhs: bigint) {
  var i = rhs.bigintToInt();
  // try 1
  this = i; // not legal under initializer rules
  // try 2
  this.init(i); // requires `proc int.init(from: int)` which we wouldn't normally add
  // try 3
  this.init=(i); // possible, but feels really strange
}

I also tried this with a real example:

module RModule {
  record R {
    var x: int;
  }
}

module QModule {
  use RModule;

  record Q {
    var x: int;
  }

  proc R.init=(rhs: Q) {
    var rr = new R(rhs.x);
    //this.init=(rr); syntax error
    //this = rr; error: cannot pass a record to a function before this.complete()
    //this.init(rr); error: unresolved call 'R.init(R)'
  }

  proc main() {
    var myQ = new Q(1);
    var myR: R = myQ;
    writeln(myR);
  }
}

The situation is similar for an array type. The author of this initializer can't know what all of the fields in an array are (they are implementation details and some might even be private) so can't be responsible for initializing them. So most of the initializer design is moot. I think they would only be able to write something like Try 3 above. E.g.

proc array.init=(rhs: Point) {
  this.init=( [ rhs.x, rhs.y ] );
}

I think it would be clearer to write it outside of an initializer:

proc cast(value: Point, type toType: []) {
  var arr:[0..1] real = [ rhs.x, rhs.y ];
  return arr;
}

I think that creating tertiary initializers creates new problems for us. For example, say I wanted to write.

Prohibiting tertiary initializers

Generally speaking, I think we should only use tertiary methods in limited cases. They come with some visibility challenges as we have discussed elsewhere. But besides that, I think that they are confusing (in terms of documentation and in terms of knowing what you have to do in order to be able to call them). At least, they are more confusing than primary or secondary methods!

Regarding initializers specifically - I think that these are "part of the type" and as such should be defined inside of the module defining the type. I am not sure if this needs to be a language rule but I would at least view it as a best practice.

On the other hand, it does not bother me if a module defining type B creates a function (or method) that defines how something of type A can be initialized from type B. While it is the reverse type relationship from init= (i.e. associated with type B rather than type A), it is still associated with one of the types, so I can still say something like "this functionality is part of the type B".

In contrast, we could imagine that proc A.init=(other: B) is associated with type B. But it is already intuitively associated with type A due to being a method on type A. I think this is unnecessarily confusing.

[bradcray]

I'm pushing back on your response, but not because I think you're wrong (necessarily), just to try and understand where your objections and reluctances stem from.

Yes, the init= being a method has to do with it

Does that suggest that if we switch to spelling operators and casts (and potentially rcasts) as methods, you'd have a similar objection? Or does expressing a cast from R to int as a reverse cast take away the concern since it's associated with R rather than int? Could init= overloads support a "reverse" form so that they would similarly not cause that concern for you?

Of your three approaches for writing a tertiary initializer on int, I think approaches 1 and 2 seem reasonable. Specifically, given that we currently support this as a way of assigning to the scalar's value for other tertiary methods, such as:

proc ref int.square {
  this *= this;
}

...approach 1 seems symmetric to me (though it also arguably raises questions with the existing approach for scalars, like "could a user write a type in which this similarly referred to the type's value wholesale in some way?" But note that that question exists even if we didn't enable approach 1 given the current tertiary method support on scalars).

Approach 2 also seems reasonable to me given that, over the years, we've discussed wanting to support new int(42) (and similarly for other built-in scalar value types) in order to support generic programming patterns like:

proc foo(type t) {
  return new t(42);
}

where today t must be a user-defined record or class type, but it could be attractive to permit it to be int or uint, say.

So while I understand that these concerns require extensions to existing features, they seem like modest ones that don't seem out-of-line with the current language design and goals to me.

I do understand from your response that creating something new and returning it (as in the current cast approach) can be more straightforward to write than an initializer-style approach.

I think that creating tertiary initializers creates new problems for us. For example, say I wanted to write.

Note that this sentence seems to lead nowhere.

In contrast, we could imagine that proc A.init=(other: B) is associated with type B. But it is already intuitively associated with type A due to being a method on type A. I think this is unnecessarily confusing.

I agree that this is odd, but it doesn't seem very much odder than supporting "reverse" operator methods on types if we were to go that way (this is why yesterday I was saying that it might be that solving the operator question first could potentially help given that while it seems related to this issue's questions, it doesn't seem dependent on them).

Generally speaking, I think we should only use tertiary methods in limited cases.

As I said when we were talking yesterday (but not on any of these issues), while this seems like an attractive design goal in general, it doesn't seem possible to avoid if you're bringing together two third-party libraries that don't know about each other and want to create new casts, operators, or implicit conversions between them. I'm also not convinced that it's all that much more confusing than defining any other operator that spans two types that aren't within your control if we get that story right.

[mppf]

@bradcray - thanks for your thoughts.

I'm pushing back on your response, but not because I think you're wrong (necessarily), just to try and understand where your objections and reluctances stem from.

Sounds good :)

Yes, the init= being a method has to do with it

Does that suggest that if we switch to spelling operators and casts (and potentially rcasts) as methods, you'd have a similar objection?

Yes

Or does expressing a cast from R to int as a reverse cast take away the concern since it's associated with R rather than int? Could init= overloads support a "reverse" form so that they would similarly not cause that concern for you?

IMO the question of how to associate operators with types has two options: 1) a feature to "associate these functions with this type" or 2) making them methods. I think that the 1st here adds some complexity in that people will have to figure out how to use the new feature. But I think the 2nd adds complexity because if they are methods (and we use the receiver type as the type they are associated with) then we are going to need the "reverse" forms for mixed-type binary operators. (So that either type in a mixed-type case can define the operator). If we have the reverse forms, then function resolution will have to be adjusted to resolve operators in a special way that combines the forward and reverse candidates into one list and then chooses between them somehow. This seems worse, in terms of complexity and potential for confusion, than the approach 1) of adding a feature to associate some free functions with a type.

Of your three approaches for writing a tertiary initializer on int, I think approaches 1 and 2 seem reasonable. Specifically, given that we currently support this as a way of assigning to the scalar's value for other tertiary methods, such as:

proc ref int.square {
  this *= this;
}

...approach 1 seems symmetric to me (though it also arguably raises questions with the existing approach for scalars, like "could a user write a type in which this similarly referred to the type's value wholesale in some way?" But note that that question exists even if we didn't enable approach 1 given the current tertiary method support on scalars).

I would be OK with doing Approach 1 for this if we need to.

Approach 2 also seems reasonable to me given that, over the years, we've discussed wanting to support new int(42) (and similarly for other built-in scalar value types) in order to support generic programming patterns like:

proc foo(type t) {
  return new t(42);
}

where today t must be a user-defined record or class type, but it could be attractive to permit it to be int or uint, say.

I really don't like this requirement. I don't think we should support things like new int(42) and I think your generic program would be better written as

proc foo(type t) {
  var x: t = 42;
  return x;
}

now that we have init=.

I think that creating tertiary initializers creates new problems for us. For example, say I wanted to write.

Note that this sentence seems to lead nowhere.

Aww now I can't delete it without confusing anyone else following :) I think it is just something I forgot to delete.

In contrast, we could imagine that proc A.init=(other: B) is associated with type B. But it is already intuitively associated with type A due to being a method on type A. I think this is unnecessarily confusing.

I agree that this is odd, but it doesn't seem very much odder than supporting "reverse" operator methods on types if we were to go that way (this is why yesterday I was saying that it might be that solving the operator question first could potentially help given that while it seems related to this issue's questions, it doesn't seem dependent on them).

Yeah, but I don't like the reverse operators either... so I guess that doesn't do much to convince me. But I agree it is reasonable to try to resolve the operator question soon.

Generally speaking, I think we should only use tertiary methods in limited cases.

As I said when we were talking yesterday (but not on any of these issues), while this seems like an attractive design goal in general, it doesn't seem possible to avoid if you're bringing together two third-party libraries that don't know about each other and want to create new casts, operators, or implicit conversions between them.

Right. I think that designs that use free functions and have a way to associate them with types (e.g. proc + continues not to be a method but could be associated with particular types) still allow a new library to "glue" together two other independently developed libraries (in which case the new functions would be associated with neither type). I think that's reasonable to allow for operators, =, and casts.

But, I am not convinced the language should allow it for implicit conversions. (And that is a different issue, anyway).

I'm also not convinced that it's all that much more confusing than defining any other operator that spans two types that aren't within your control if we get that story right.

I still think it will be more confusing to create tertiary methods than to create a free function/operator associated with neither type. Not sure what else to say about it, though.

[mppf]

In a separate discussion @vasslitvinov pointed out to me that if we keep cast as a function rather than making it some kind of initializer, then it has the potential to return ref or const ref in cases where that makes sense.

Vs. if it is an initializer, it must produce a value.

[bradcray]

then it has the potential to return ref or const ref in cases where that makes sense.

In what cases does it does make sense? I tend to think of casting (for value types, at least) as always creating a new value.

[bradcray]

If we have the reverse forms, then function resolution will have to be adjusted to resolve operators in a special way that combines the forward and reverse candidates into one list

I agree with this.

and then chooses between them somehow.

I'm not sure I agree with this. If the code doesn't contain ambiguous overloads, then I don't think the "chooses between them somehow" comes into play. If it does contain ambiguous overloads, then I think the compiler just complains about the ambiguity. I also think there's nothing about the method-based approach that makes ambiguous or redundant overloads more likely to be written than the standalone-function approach.

This seems worse, in terms of complexity and potential for confusion, than the approach 1) of adding a feature to associate some free functions with a type.

I agree that reverse operators take a bit more work to get one's mind around. But I think defining operators as methods has the strong advantages that it (a) makes operators less "special" and (b) it associates them with a type in the obvious way, so shouldn't be discarded while we're still discussing the options.

Leaving them as standalone functions, we've discussed having a way to associate them with types, and potentially extending that support to arbitrary standalone functions to make operators less special in this regard. But that feels pretty deeply problematic in terms of keeping namespaces and imports sane, in that it means that by importing a type, I potentially end up dragging in arbitrary functions that I didn't expect to just because someone tagged them as being part of the type (maybe incorrectly, maybe evil-ly). Methods seem to me like the clearer way to associate capabilities with a type.

We could potentially address the reverse operator confusion issue by taking a method-based approach more like Swift's "operators are static/type methods (yet ones that aren't called on a type name... weird)" approach (assuming I've understood it properly). Of these various weirdnesses ("reverse operators", "standalone functions that are not actually standalone", "static methods that aren't called on types") I think I find it the least off-putting.

and I think your generic program would be better written as

proc foo(type t) {
  var x: t = 42;
  return x;
}

now that we have init=.

This pattern will only work for t's that are value types, but not for classes (at least, without additional changes that I'm not sure we want to make).

I don't think we should support things like new int(42)

I'm curious why. This pattern seems really non-offensive to me given other value forms like new R(42), new bigint(42) in the language. Note that approach 2 isn't my favorite way to write this (approach 1 is), so I wouldn't introduce this simply to support this pattern. But I would be in favor of supporting new int(42) in any case.

Aww now I can't delete it without confusing anyone

Sure you can, you just delete my comment about it and your response and this one as well. :)

Yeah, but I don't like the reverse operators either

That's fine, but my point is "if the solution to defining operators turned out to be to support reverse operators and appeal to the Python programmers out there, then we've arguably got a path forward here as well."

still allow a new library to "glue" together two other independently developed libraries (in which case the new functions would be associated with neither type)

I'm not seeing how this approach would work: If I were an "import-only" style of user, how would I get access to these operators if they weren't associated with either type? (I've been assuming that in either of the "method-based" or "operators-must-be-associated-with-a-type" approaches, you'd get access to the operators by importing the type from the module, similar to tertiary methods).

[mppf]

We could potentially address the reverse operator confusion issue by taking a method-based approach more like Swift's "operators are static/type methods (yet ones that aren't called on a type name... weird)" approach (assuming I've understood it properly). Of these various weirdnesses ("reverse operators", "standalone functions that are not actually standalone", "static methods that aren't called on types") I think I find it the least off-putting.

I also have a pretty good feeling about the "operators are static/type methods" approach.

[mppf]

and I think your generic program would be better written as

proc foo(type t) {
  var x: t = 42;
  return x;
}

now that we have init=.

This pattern will only work for t's that are value types, but not for classes (at least, without additional changes that I'm not sure we want to make).

Good point! I think I would start to argue next that we don't necessarily need code that works the same on classes and records in this way, but...

I don't think we should support things like new int(42)

I'm curious why. This pattern seems really non-offensive to me given other value forms like new R(42), new bigint(42) in the language. Note that approach 2 isn't my favorite way to write this (approach 1 is), so I wouldn't introduce this simply to support this pattern. But I would be in favor of supporting new int(42) in any case.

It's just that I don't think that record authors should be writing both proc R.init=(other: int) and proc R.init(other: int) (which we do today in some standard modules). However we could define the language so that new R(1) will call R.init=(other: int) if the proc R.init(other: int) is not available. But that's almost certainly a matter for another issue.

[mppf]

This part is really about whether we can keep operators as standalone functions vs. making them methods (i.e. a side topic for this issue):

still allow a new library to "glue" together two other independently developed libraries (in which case the new functions would be associated with neither type)

I'm not seeing how this approach would work: If I were an "import-only" style of user, how would I get access to these operators if they weren't associated with either type? (I've been assuming that in either of the "method-based" or "operators-must-be-associated-with-a-type" approaches, you'd get access to the operators by importing the type from the module, similar to tertiary methods).

If we have operators that are standalone functions - I wouldn't expect that we also require them to be associated with a type. (We would allow them to be associated with a type). As a result, somebody doing imports could import the operators associated with neither type with something like import GlueModule.{+, =, :}.

[bradcray]

It's just that I don't think that record authors should be writing both proc R.init=(other: int) and proc R.init(other: int) (which we do today in some standard modules).

That's a reasonable point, and I like your suggestion about the presence of one supporting the other.

[mppf]

We have been developing a viewpoint that casts should be something one can request separately from = / init=. In other words that the answer to Q3 is "yes".

[mppf]

Given the operator keyword to introduce operators (from PR #17103), I'm currently expecting that user-defined casts will be written as operator : (value: FromType, type t: ToType)

Here is the current form of Q3c from #16732 (comment) :

// support myInt : bigint
operator : (value: int, type toType: bigint) {
  return new bigint(value);
}
// support myBigint : int
operator : (value: bigint, type toType: int) throws {
  return value.convertToIntOrThrow();
}

[mppf]

I don't think we should support things like new int(42)

I'm curious why. This pattern seems really non-offensive to me given other value forms like new R(42), new bigint(42) in the language. Note that approach 2 isn't my favorite way to write this (approach 1 is), so I wouldn't introduce this simply to support this pattern. But I would be in favor of supporting new int(42) in any case.

It's just that I don't think that record authors should be writing both proc R.init=(other: int) and proc R.init(other: int) (which we do today in some standard modules). However we could define the language so that new R(1) will call R.init=(other: int) if the proc R.init(other: int) is not available. But that's almost certainly a matter for another issue.

That's a reasonable point, and I like your suggestion about the presence of one supporting the other.

I've created issue #17199 for this question specifically.

[mppf]

PR #17146 implements the change settled on here, so closing this issue.

[mppf]

I made issue #17200 to ask about Q1 (where for now it is an error if the cast is not provided).

[mppf]

I've created issue #17225 to discuss tertiary initializers (and things like int.init=).