Add TrueTypeOf and ToFunctionType to phobos.sys.traits.

[jmdavis]

TrueTypeOf is completely new. It's a wrapper around typeof which bypasses typeof's behavior with @property and instead gives the actual type of the function. For other symbols, it's the same as typeof with the caveat that template alias parameters cannot accept general expressions. So, TrueTypeOf will work on symbols, but not stuff like &var. But it will work ion the symbol for a function or variable, thereby providing a replacement for typeof in situations where property functions need to be treated as the functions that they are.

The plan here is that the function-related traits in phobos.sys.traits will operate solely on types (except in situations where the actual symbol is required - e.g. to get the names of parameters), like most traits typically do. std.traits has a number of its function-related traits operate on symbols in part to work around the issue with property functions (as well as stuff like trying to treat variables with opCall as functions instead of requiring that the symbol for opCall itself be passed). TrueTypeOf will therefore allow code to get the actual type of the function and thus we won't need to have as many traits operate on symbols instead of types. That will also have the benefit of reducing the number of cases where traits operate on both types and other symbols, since that has a tendency to be error-prone and make the code harder to understand. It should also help with clarity when a trait doesn't try to handle everything itself.

Exactly how this will affect each trait will of course depend on the trait in question, but the idea is to simplify things and ultimately end up with traits which are easier to understand and less error-prone.

The other new symbol, ToFunctionType, is a template which converts function types, function pointer types, and delegate types to the corresponding function type. So, something like int function(string) or int delegate(string) would become int(string). This is primarily useful in implementing other function-related traits, but it also provides a way to get function types to use in is expressions for testing or providing examples.

In terms of functionality, ToFunctionType is a replacement for std.traits.FunctionTypeOf. FunctionTypeOf attempts to operate on anything that's "callable" (both types and symbols), which makes it a bit of a mess (and which actually cannot work in some cases due to templated types or functions not having been instantiated). And looking over where it's used in std.traits, it's usually used on types anyway.

So, ToFunctionType operates exclusively on types. The change in name is because FunctionTypeOf definitely sounds like a trait that's operating on a symbol to get its type (even if it also accepts types), whereas ToFunctionType sounds much more like it's converting the given type, which is what it's doing.

[dlang-bot]

Thanks for your pull request, @jmdavis!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub run digger -- build "master + phobos#10710"

[jmdavis]

BTW, the assertion with isFunctionPointer is commented out, because a PR for isFunctionPointer will be needed for that, but I've split it off to include in a later PR in order to reduce the size of the PRs.

[0xEAB]

While TrueTypeOf has nothing to do with that obviously, I’m not sure whether we want to have TrueType® in its name.

[jmdavis]

While TrueTypeOf has nothing to do with that obviously, I’m not sure whether we want to have TrueType® in its name.

As they're clearly completely unrelated, I see no issue.

[pbackus]

Is TrueTypeOf really necessary? Having library traits and language features that do similar-but-slightly-different things is a mistake I was hoping we'd correct in Phobos v3.

If it really needs to exist, its name should be something more descriptive, like TypeofSymbol.

[jmdavis]

Is TrueTypeOf really necessary?

Yes, it's necessary, because what typeof does with @property causes problems that a number of traits have to work around. Having TrueTypeOf makes it so that the problem can be dealt with in one place instead of needing special casing and extra logic in a bunch of different places. It also provides a place that helps document the problem.

Given that we're clearly not going to get rid of optional parens (as was originally the plan with @property), it would be better if we could fix it so that typeof didn't do anything special with @property, but it's been that way for over a decade, and changing it now would break who knows how much code, much of which is using the current behavior by accident. So, I would be very surprised if it ever got fixed. And even it does end up happening, we have to deal with the language as it is right now. And if it does actually happen, then TrueTypeOf can be changed to just give the result of typeof, and code that uses it will continue to work as it did before. If anything, it might help us reduce how much code would break if typeof were fixed, because code written with TrueTypeOf wouldn't be affected.

TrueTypeOf is the cleanest way that I could come up with to work around the problem without having it affect a bunch of different traits like it does in std.traits.

Having library traits and language features that do similar-but-slightly-different things is a mistake I was hoping we'd correct in Phobos v3.

If the language feature does the wrong thing, and we can't fix the language feature, the library needs to have an alternate solution. And when it comes to features related to type introspection, it's highly unlikely that we will ever be able to fix them, because it would break a lot of code in unpredictable ways if we tried - especially with how much D's type introspection is built on checking whether a particular piece of code compiles or not.

If it really needs to exist, its name should be something more descriptive, like TypeofSymbol.

Maybe? I would have thought that TrueTypeOf was descriptive enough, and it's shorter. In either case, you have to read the documentation to know what it does. The name just gives you the general idea and isn't enough on its own to tell you how it's different from typeof - and if anything, TrueTypeOf at least potentially implies that typeof isn't giving the true type, whereas TypeOfSymbal gives no indication that it does anything different from typeof.

[pbackus]

Yes, it's necessary, because what typeof does with @property causes problems that a number of traits have to work around.

Well, are the workarounds really necessary? My impression was that a lot of these workarounds are the result of generality creep, and that we could do away with them by being stricter about what kinds of inputs we accept. Since you're working on this directly, though, I'm willing to defer to your judgement here.

If the language feature does the wrong thing, and we can't fix the language feature, the library needs to have an alternate solution.

I'm fine with this as long as the alternate library solution has a clearly distinct name. E.g., having __traits(isScalar, T) and isScalarType!T do different things is a problem, because it uses the same vocabulary ("scalar type") to mean different things.

IMO, the name TrueTypeOf is not distinct enough from typeof to prevent this kind of confusion.

TrueTypeOf at least potentially implies that typeof isn't giving the true type, whereas TypeOfSymbol gives no indication that it does anything different from typeof.

I'll grant that TypeOfSymbol is also not great. Perhaps NonPropertyTypeOf or TypeOfIgnoringProperty would be more accurate? They're ugly names, but frankly the underlying concept is ugly too, and I would rather have an accurate name which reflects that ugliness than a pretty name that tries to hide it.

The name TrueTypeOf in particular is also bad because it implies that it is always preferable to use TrueTypeOf instead of typeof.

[jmdavis]

Yes, it's necessary, because what typeof does with @property causes problems that a number of traits have to work around.

Well, are the workarounds really necessary? My impression was that a lot of these workarounds are the result of generality creep, and that we could do away with them by being stricter about what kinds of inputs we accept. Since you're working on this directly, though, I'm willing to defer to your judgement here.

If any trait deals with property functions as symbols, and it needs to get the type of a symbol, then it has to deal with the fact that typeof lies for property functions. In the case of a getter property, you'll get the return type, and in the case of a setter property, you'll get a compilation error. What typeof does might make sense if we didn't have optional parens, but since we do have optional parens, it causes nothing but trouble outside of corner cases like lvalueOf and rvalueOf which explicitly take advantage of the behavior to do what they do (though it would be nice to find an alternate way to solve that problem).

From what I can see, the best way to clean this up for most traits is to make it so that they only operate on types and not on both types and symbols which aren't types. That way, what typeof does with @property doesn't matter for the trait at all. However, to make that work, we then need an alternative to typeof which does not treat @property functions as special. And in some cases, this means that a trait isn't even necessary outside of cases where you need a template predicate. For instance, isSomeFunction!foo could become is(TrueTypeOf!foo == return), whereas right now, you'd use isSomeFunction!foo instead of is(typeof(foo) == return) in order to avoid problems with property functions - or you wouldn't be aware of the problem and would potentially have bugs because of your use of typeof without checking whether the symbol was a property function. Or, since isSomeFunction accepts types as well, you might do isSomeFunction!(typeof(foo)) and also get the wrong result even though isSomeFunction!foo would have worked. By having TrueTypeOf, we're providing a way to just use is expressions instead when that makes sense in addition to allowing us to simplify some traits by making it so that they just take types.

TrueTypeOf also helps with many of the non-function traits which already take types and not general symbols. For instance, isPointer accepts only types, and thus you already need to use typeof to use it with symbols, and if a piece of code uses typeof on a property function that returns a pointer in order to pass it to isPointer, then the result will be true when it was almost certainly the case that what the programmer would have wanted would be for it to be false, because it was a property function which returned a pointer and not an actual pointer.

So, TrueTypeOf gives us a general tool to use instead of typeof in any situation where we need a type for a symbol, and it might be a property function, and we want it to be treated as a function if it is. It makes it easier to avoid bugs with regards to property functions with traits in general, not just those focused on functions.

If the language feature does the wrong thing, and we can't fix the language feature, the library needs to have an alternate solution.

I'm fine with this as long as the alternate library solution has a clearly distinct name. E.g., having __traits(isScalar, T) and isScalarType!T do different things is a problem, because it uses the same vocabulary ("scalar type") to mean different things.

IMO, the name TrueTypeOf is not distinct enough from typeof to prevent this kind of confusion.

Well, I disagree. If it were TypeOf, then sure, but I think that it's clear that it's something different, and if someone doesn't already know what it is, they can read the documentation to find out. I would not expect it to be common for someone to assume that they knew how it was different from typeof based on the name - or that they would think that it did exactly the same thing as typeof, since it's TrueTypeOf and not TypeOf. If anything, it seems to me that the name raises questions for someone who doesn't already know what it does, presumably leading them to look it up, rather than it being a name where they're likely to assume what it does based on the name alone.

TrueTypeOf at least potentially implies that typeof isn't giving the true type, whereas TypeOfSymbol gives no indication that it does anything different from typeof.

I'll grant that TypeOfSymbol is also not great. Perhaps NonPropertyTypeOf or TypeOfIgnoringProperty would be more accurate? They're ugly names, but frankly the underlying concept is ugly too, and I would rather have an accurate name which reflects that ugliness than a pretty name that tries to hide it.

I thought about names along those lines, but that's long and ugly for a trait that's likely to be used extensively. Honestly, even just writing the tests, I find TrueTypeOf to be annoyingly long, but it was the best name that I could think of, and calling it something like TypeOf would cause exactly the kind of confusion that you're worried about. TrueTypeOf was as short as I could get it while still being reasonably clear, whereas anything with property in the name gets really long.

The name TrueTypeOf in particular is also bad because it implies that it is always preferable to use TrueTypeOf instead of typeof.

Except that in the general case, if you're getting the type of a symbol rather than an expression, it is preferable to use TrueTypeOf. typeof makes sense when you know that the symbol can't be a property function, and it's what you need to use when checking the type of an expression rather than a symbol, but if you're checking the type of a symbol, there will be fewer bugs if you use TrueTypeOf instead of typeof. And if there is some confusion where someone wants decides to use TrueTypeOf on an expression instead of a symbol, they'll just get a compilation error. So, arguably, if TrueTypeOf compiles, it should be the preferred solution. Obviously if anyone is looking to reduce the number of templates used in a particular piece of code, and they're sure that the symbol won't be a property function, then it makes perfect sense to still use typeof, but I would argue that the default solution should become TrueTypeOf rather than typeof in order to prevent bugs.

I expect that the only reason that we don't get more bugs due to typeof's behavior with regards to @property is because it's often the case that the return type of a getter property doesn't match the trait being used any more than the type of the function would have. But in some cases, it does match when the type of the function wouldn't have - or it doesn't match when the type of the function would have. In the lucky cases, that results in a compilation error, so the programmer gets confused about what's going on and then figures out how to fix it, whereas in the unlucky cases, the code manages to compile but do the wrong thing. It can cause serious problems but does so just infrequently enough that folks tend to forget about it and not handle it properly.

TrueTypeOf gives us a way to consistently work around typeof's bad behavior - to the point that I'm honestly tempted to add it to std.traits as well and start telling folks to use it.

Honestly, if it weren't the case that fixing typeof would break a lot of existing code, I'd argue for simply fixing it - and maybe in a future edition, we'll be able to fix it in spite of how much code it would break (though given how type introspection works, I expect that such a change would be too risky at this point). And if we do ever fix typeof, then the code using TrueTypeOf will continue to work as-is (though it could then be changed to use typeof instead). So, if TrueTypeOf starts getting used enough, it might actually make it easier to fix typeof, because the change wouldn't break as much code.

[pbackus]

In my experience, when foo.bar could be interpreted as either a symbol or an expression, the type you are interested in is almost always the type of the expression, not the type of the symbol.

For example, in ElementType, when we check the type of r.front, we care about what type it is as an expression, not as a symbol. So we actually use a workaround to avoid getting the type of the symbol:

phobos/std/range/primitives.d

Lines 1377 to 1378 in b57f75d

	static if (is(typeof(R.init.front.init) T))
	alias ElementType = T;

A similar workaround is used in isForwardRange to check the type of r.save:

phobos/std/range/primitives.d

Lines 1025 to 1026 in b57f75d

	enum bool isForwardRange(R) = isInputRange!R
	&& is(typeof((R r) { return r.save; } (R.init)) == R);

And in hasToString in std.format:

phobos/std/format/internal/write.d

Lines 1896 to 1899 in b57f75d

	else static if (is(ReturnType!((T val) { return val.toString(); }) S) && isSomeString!S)
	{
	enum hasToString = HasToStringResult.hasSomeToString;
	}

So when you say things like this:

if a piece of code uses typeof on a property function that returns a pointer in order to pass it to isPointer, then the result will be true when it was almost certainly the case that what the programmer would have wanted would be for it to be false

...I think you are simply incorrect. In this case, the programmer almost certainly wants the result to be true—for both @property and non-@property functions. And they will probably have to apply a workaround like the ones above to make that happen.

[jmdavis]

In my experience, when foo.bar could be interpreted as either a symbol or an expression, the type you are interested in is almost always the type of the expression, not the type of the symbol.
...
...I think you are simply incorrect. In this case, the programmer almost certainly wants the result to be true—for both @property and non-@property functions. And they will probably have to apply a workaround like the ones above to make that happen.

Well, this factors into a related problem that I decided needed a trait when I was working on this but haven't sat down to actually work on yet, so I haven't yet thought about naming it. And maybe I should have sat down and worked through that before creating this PR, but I was almost done with this and several other traits and decided to wait until afterward, since the amount of code was piling up.

The problem relates to property functions in that optional parens cause problems with type introspection. It means that you could have something like obj.foo, and you're going to use that in code as obj.foo - e.g. auto value = obj.foo;. And if you do that, then typeof is going to give varying results depending on what foo actually is.

If @property had been completed, and optional parens had been abolished as had been the plan, then if auto value = obj.foo; were legal, typeof(obj.foo) would give you typeof(value) consistently. If instead, @property were removed from the language and we just had optional parens, then typeof(obj.foo) would give us the same as typeof(value) when foo was a variable, and it would give the type of the function when foo was a function. The fact that we have @property and optional parens means that typeof(obj.foo) sometimes gives typeof(value) for functions too, but not consistently, because it only happens if that function is marked with @property. And since many folks routinely call functions without parens, it's not at all reliable to assume that typeof(obj.foo) is a field or that it was even planned to be used as a field and marked with @property.

So, if you want to know the type of the symbol forobj.foo, @property gets in the way, making it so that you're not actually getting the type of the symbol. And if you want the type of the expression (presumably because you're looking to see what the type is when you use the symbol, not get information about the symbol itself), then @property shifts the problem so that you get the type of the expression in more cases, but you still get the type of the symbol in many cases.

Arguably, a part of the problem here is that typeof is designed to work on both symbols and expressions. For variables, there isn't really a difference, but for functions, there's a huge difference. And whether you want typeof(obj.foo) to give you the type of the symbol or the type of the expression depends on the context, but we have poor control over which you're getting. And if anything, right now, the control is in the wrong place, because it changes based on the function's attributes rather than on how the function is being used. And whether we have @property or optional parens, there are always going to be cases where you want the type of the symbol and cases where you want the type from using the symbol in an expression. So, maybe the big design flaw here was that typeof wasn't two separate constructs.

Right now, to consistently get the type of the symbol when used in an expression, you need to do something like declare a lambda which returns obj.foo, and then you get the return type - e.g. typeof(() { return obj.foo; }()). This is kind of error-prone, particularly since it's easy to miss the second set of parens. It's also not something that folks necessarily think of trying. If anything, they're probably more likely to just have tested the code with obj being a variable, and written typeof(obj.foo), resulting in problems later when someone uses a type where foo is a function but not a property function. And even if they tested it with a type where foo was a function, it was probably a @property function even though that's not at all required to use it as obj.foo.

So, we need a trait where you say that you want the type of obj.foo in an expression (in addition to a trait that consistently gets the type of the symbol, which TrueTypeOf does). And if the difference between those two traits is that in the one case, you want the type of the expression, and in the other, you want the type of the symbol, then maybe names like TypeOfExpr and TypeOfSymbol would make sense. And in that context, TrueTypeOf is indeed a poor name. I probably would have seen that if I had put off creating this PR until after I'd done the expression version first, but while it had occurred to me as something that needed doing, I hadn't yet sat down to really think through it. So, that was a mistake, and thank you for arguing the point.

As things stand, typeof is bound to be used incorrectly both when folks are looking to get the type of the symbol itself and when they are looking to get the type of the expression. So, if we had traits for both of those cases, we could tell folks to use the appropriate one based on what they were trying to do, leaving typeof to be used only in situations where you have an actual expression rather than just a symbol that you might want to use as an expression, or for when folks don't want the extra template instantiations and know to be careful with what they're doing.

So, I'll rename TrueTypeOf to TypeOfSymbol and then do TypeOfExpr in another - though Adam may complain about the abbreviation and insist that it be TypeOfExpression, which is way too long IMHO, in which case, I don't know what I'll name that one. TypeOfSymbol is already pushing it pretty far in the length department, particularly for something that's likely to be used frequently in longer expressions.

[jmdavis]

Actually, I'm just going to close this PR and do another one with TypeOfExpr alongside this, since having both affects both the documentation and the tests for TypeOfSymbol (at least if they're going to be quality). It'll make the PR larger than I'd like, but it'll be clearer.