Troubles with generics when they are too general

[waterlink]

Code I am trying to write: http://carc.in/#/r/l61 - this is super generic code that tries to avoid knowing types beforehand. With current generic limitations it does not work: hence use a more specific type.

So I tried to use virtual types a bit to fix this problem and encountered this particular issue: http://carc.in/#/r/l69 - It segfaults at runtime. Though it is not reproducible locally, unfortunately :( Local output is as expected:

result.unwrap as Array(String) = ["hello", "world"]
result.unwrap as Bool = true

Though adding two more register calls for the same method and object:

r.register(Message.new("split", s, [". "]), Result.new(["abc", "def"]))
r.register(Message.new("split", s, ["hello"]), Result.new(["world"]))

Makes it crash both in http://carc.in/#/r/l6b and on my local machine:

crystal-playground|⇒ ./crash 
[1]    32013 segmentation fault (core dumped)  ./crash

And finally after introducing downcast method into both virtual type and its implementation:

module GenericMessage
  # .. abstract definitions here ..
  abstract def downcast
end

class Message(T, A)
  # .. class body here ..
  def downcast
    self
  end
end

# Each usage of GenericMessage now looks like this:
generic_message :: GenericMessage
generic_message.downcast.hash   # => <hash of message>

# The same goes for `GenericResult` and `Result(T)`.

And it finally worked correctly: http://carc.in/#/r/l6d

So my main question is what may cause the problem in the first place? And my other question is: Why introducing this #downcast method that will be always implemented as return self helps to solve it?

[waterlink]

I call it #downcast but do not be confused it might be very well mis-named.. Maybe it is an #upcast or something else.

[waterlink]

Oh, just read the difference between upcasting and downcasting and what I do there is really downcast, but without actual knowledge about "downcast to which type".. So it is like auto-downcast supported by type-inference..

[waterlink]

Another point that bothers me: I feel like I am trying to trick compiler into believing me that I know what I am doing, just let me store anything in this Hash.

It bothers me, because it could be a potential bug of compiler that I am able to do these things at all, and in that case they will be fixed in future version(s) and my code will be broken. Potentially without a chance of fixing, because I am just doing "the wrong thing".

[asterite]

Some answers :-)

After the next step you'll be able to have a generic object as a generic type argument. If dealing with such object, T will be Object.

For example:

class Generic(T)
  getter value
  getter size

  def initialize(@value : T)
    @size = 0
  end
end

a = [] of Generic
a << Generic.new(1)
a << Generic.new("hello")
typeof(a[0]) # Generic
typeof(a[0].size) # Int32 (known because of `@size = 0` which means it's an Int32)
typeof(a[0].value) # Object (known because of `@value : T` and T can be anything)

So you could then do:

a[0].value as Int32 # OK
a[0].value as String #=> cast exception

Of course, if you have a concrete instantiation the compiler will know the type of T:

typeof(Generic.new(1).value) #=> Int32

I think with this change you won't have to fight the compiler anymore and no tricks will be needed. The current crashes are just compiler bugs, but instead of fixing them now we probably won't have to fix them in the next compiler because the semantic will be different that the current one.

[waterlink]

Sounds good. That answers my questions on this topic. Thanks!

[asterite]

@waterlink I think one thing that we'll loose with the new apporach is doing a wrapper class to hold "any type that's used to create this wrapper, thus forming a union of used types". For example this works right now:

class Wrapper
  def initialize(@value)
  end

  def size
    @value.size
  end
end

Wrapper.new("foo").size # => 3
Wrapper.new([1, 2]).size # => 2

The new compiler won't be able to infer the type of @value so you'll have to write @value :: Object or @value :: TypesThatHaveSize and include that module in the types you want to wrap. Or, if we introduce the concept of interfaces, just declare what you need from the objects and the compiler could detect which classes implement it, without making that explicit.

I think the Wrapper above is used to workaround the "can't use generics in type arguments" limitation. I guess it has other uses, mostly wrapping all types that quack in some way, but with interfaces we might make that a bit more explicit and maybe clearer for the reader, while still retaining that duck typing ability.

[waterlink]

I see, what about generic type like this:

class Wrapper(T)
  @value :: T
  def initialize(@value : T) end
  def size; @value.size end
end

Wrapper.new("foo").size     # => 3
Wrapper.new({1, "x"}).size  # => 2

Would that still work?

[asterite]

Yes.

But I forgot another thing, right now you can do:

class Wrapper
  def initialize(@value)
  end

  def size
    @value.size
  end
end

ws = [] of Wrapper
ws << Wrapper.new("foo")
ws << Wrapper.new([1, 2])

# This works, because @value is inferred to be `String | Array(Int32)`
puts ws.map &.size #=> [3, 2]

In the new compiler the above won't work, @value has no type annotation. And if I annotate @value :: Object it won't work, because not all objects have a size method. So you could try to do this:

class Wrapper(T)
  @value :: T

  def initialize(@value : T)
  end

  def value
    @value
  end

  def size
    @value.size
  end
end

ws = [] of Wrapper
ws << Wrapper.new("foo")
ws << Wrapper.new([1, 2])

# But it won't work
puts ws.map &.size # Error: undefined method `size` for Object

That's because:

typeof(ws[0]) # Wrapper
typeof(ws[0].value) # Object (not String | Array(Int32), an instance variable type can't depend on the usage of a class for modular compilation)

So we can make:

module HasSize
end

class String
  include HasSize
end

class Array
  include HasSize
end

class Wrapper
  @value :: HasSize
end

And now it will work because all HasSize types have a size method. But it's very ugly and annoying because I need to include these dummy modules in every type that wants to be used by Wrapper.

That's why I'm maybe thinking interfaces can solve this problem very easily. You define an interface:

interface HasSize
  def size
end

The compiler checks which objects have a size method so HasSize is the same as a union of all those types. It can also infer the type of the size method: just the union of the types of all the size methods in that union.

The downside of interface is that you have to list the methods you need.

The only thing I don't know is how quickly the compiler can do that check. It will have to do it for all interfaces and all objects.

Anyway, these are just ideas for alternatives to the semantic loses will have.

[waterlink]

Yeah, I think it is quite complicated to figure out beforehand.

And I would say 👍 to the idea of implicit interface.

[waterlink]

If it will be really too slow to figure out implicit interface types, why not make it explicit in this case:

interface HasSize
  def size
end

implements HasSize String
implements HasSize Array
# .. and so on ..

This is a bit more typing, but I think that is fine. And use can always upon creating new class mark it as implementor of certain interfaces.

[asterite]

@waterlink In that case module is just fine and there's no need for interface

[waterlink]

Do you mean module with abstract defs ?

[asterite]

@waterlink Yes, exactly. And even without abstract def it will work because the compiler can know the types that include it before having to start typing methods.

[waterlink]

Ahh, and so it will know all the common methods, right? Like size in this case.

Wouldn't the error message be confusing, if one of the types does not implement size ?

[waterlink]

I would say it is still nice to see interface defined with method signatures. For example when you are writing library and expose such interface to the user of library - the user knows which methods to implement on his object to make usage of this library features.

[waterlink]

BTW, interface + implements one already can kind of mimic with macros: http://carc.in/#/r/ld0

macro interface(name, &block)
module {{name.id}}
  macro idef(spec)
    abstract def \{{spec}}
  end
  {{block.body}}
end
end

macro implements(iface, ty, name)
{{ty.id}} {{name.id}}
include {{iface}}
{{:end.id}}
end

---

EDIT: updated carc.in link

[waterlink]

But still, if it is possible to make fast for compiler to figure out, I would prefer implicit interfaces :)

[waterlink]

Another possibility is to make generics implicitly define interfaces:

class Wrapper(T)
  @value :: T
  def initialize(@value : T) end

  def size
    @value.size
    # compiler, after this point, restricts T to be `respond_to(:size)`, 
    # or generates an implicit interface with one method with zero
    # arguments and makes T restricted to it.
  end
end

# so that works like a charm:
a = [] of Wrapper
a << Wrapper.new([1, 2, 3])
a << Wrapper.new("hello")
a.map &.size # => [3, 5]

[waterlink]

Maybe we can take some inspiration from Rust and apply interface restrictions on generic parameters:

class Wrapper(T : HasSize & HasStuff)
  @value :: T
  def initialize(@value : T) end
  delegate size, @value
  delegate stuff, @value
end

[omninonsense]

👍 for the second to last approach

[sdogruyol]

@waterlink +1 for Rust's Trait like interface