Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reflect: NamedOf #16522

Open
crawshaw opened this issue Jul 28, 2016 · 16 comments
Open

reflect: NamedOf #16522

crawshaw opened this issue Jul 28, 2016 · 16 comments

Comments

@crawshaw
Copy link
Member

@crawshaw crawshaw commented Jul 28, 2016

Consider adding a function to the reflect package for creating a new named type with a method set:

func NamedOf(t Type, name string, methods []Method) Type

(Broken out from #4146.)

@crawshaw crawshaw added this to the Go1.8 milestone Jul 28, 2016
@crawshaw crawshaw self-assigned this Jul 28, 2016
@crawshaw
Copy link
Member Author

@crawshaw crawshaw commented Aug 2, 2016

This turns out to require more invasive changes than I originally realized.

To be useful, it needs to be possible to add methods to types created at run time. That means taking some equivalent to func(receiver reflect.Value, args []reflect.Value) (results []reflect.Value) as the user's implementation of a method.

But methods are called directly from pointers to the text segment (stored in the itab). The unboxed receiver value is on the stack for the method body to use, but no type is available. So there's no way for the reflect package to take the receiver value and box it up into a reflect.Value when the method is called.

Assuming we want to avoid code generation, some change to dynamic method calling would have to be made. The smallest is probably dropping the underlying type and method number onto the stack. A more significant change would be making the function pointers in the itab indirect like func() pointers (https://golang.org/s/go11func). A third option would be a note on the itab saying these are func()-style pointers. Any of these actions incurs some cost on dynamic method calls.

Still pondering.

Loading

@glycerine
Copy link

@glycerine glycerine commented Dec 19, 2016

The unboxed receiver value is on the stack for the method body to use, but no type is available.

I thought the precise GC work back in Go1.4 meant all types on the stack are known to the runtime. Is that no longer true?

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 19, 2016

We only have pointer/nonpointer bits for each word on the stack. We do not have full type information.

Loading

@jimmyfrasche
Copy link
Member

@jimmyfrasche jimmyfrasche commented Aug 21, 2017

A discussion in the original thread starting at #4146 (comment) eventually honed in on really wanting something like described in this issue.

Referencing here because there is some discussion about how NamedOf/NewTypeOf would ideally behave to satisfy the goal of extending known interfaces without hiding unknown optional methods.

Loading

@nerdatmath
Copy link
Contributor

@nerdatmath nerdatmath commented Dec 27, 2017

Since NamedOf() knows the new type T that it is creating, couldn't it build a closure for each method, and store a pointer to the runtime._type for T in each closure? I'm assuming we can build a reflect.Value given an unboxed value and runtime._type.

Since there is no subtyping, we don't have to worry about the receiver's actual type being different from T (or *T), right?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Dec 27, 2017

@nerdatmath The language doesn't permit methods to be closures. So, for simplicity, the same mechanism is used to pass the receiver value as is used to pass the pointer to the closure. For a method, we pass the receiver, and for a closure, we pass the closure pointer. We don't have a way to pass both.

Loading

@nerdatmath
Copy link
Contributor

@nerdatmath nerdatmath commented Dec 28, 2017

OK another shot. Suppose T is a concrete type, s is of type T, x is of type A, and m is a method of T accepting a single parameter of type A. From the programmer's perspective, calling s.m(x) is essentially the same as calling T.m(s, x). Does that correspondence extend down to the calling convention? When calling T.m the compiler knows the concrete type of s (and knows that T.m knows it too), so it doesn't need to pass the type info.

If so, we already have a way to create functions of type func(T, A): runtime.FuncOf. Could this or a similar function create method pointers that could be put in itabs? Please forgive me if I'm being hopelessly naive.

Loading

@edaniels
Copy link
Contributor

@edaniels edaniels commented Dec 28, 2017

@nerdatmath, I think the issue that you'll run into that @ianlancetaylor is referring to is that once you do create the struct/type of T, the code that compiler will generate for s.m(x) will behave such that if you try to interact with s as the expected receivever, you will encounter a nil pointer exception do to the nature of creating a closure and the calling conventions described in https://golang.org/s/go11func. The word passed along the stack in the call to the function will be the closure data pointer, not the receiver (s).

I believe the long standing calling conventions would need to modified or extended in order to support this. There's probably a safe way to extend the compiler to support a new calling convention. I went down a very deep rabbit hole trying to implement NamedOf using StructOf as a reference and hit this roadblock.

Loading

@glycerine
Copy link

@glycerine glycerine commented Dec 28, 2017

But methods are called directly from pointers to the text segment (stored in the itab). The unboxed receiver value is on the stack for the method body to use, but no type is available.

I thought the concrete type was pointed at by the other half of the interface value (https://research.swtch.com/interfaces). If one is calling through the interface value, would the type not be available? What am I missing?

Loading

@nerdatmath
Copy link
Contributor

@nerdatmath nerdatmath commented Dec 28, 2017

I dropped the idea of creating a closure. What does reflect.MakeFunc create? Not a closure but an actual function like TopLevel in https://golang.org/s/go11func, right? From that document:

Direct call of method. In order to use the same generated code for both an indirect call of a func value and for a direct call, the code generated for a method (both value and pointer receivers) is chosen to have the same calling convention as a top-level function with the receiver as a leading argument.

So essentially, creating a method should be the same as creating a top-level function, which is done by reflect.MakeFunc. My apologies for referring to it as runtime.FuncOf earlier.

Loading

@nerdatmath
Copy link
Contributor

@nerdatmath nerdatmath commented Dec 28, 2017

OK so I guess reflect.MakeFunc actually creates something like a func literal / closure. So I think I see the difficulty.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Dec 28, 2017

@glycerine The interface value has the type, yes, but in the current calling convention we don't pass the entire interface value in a method call. We only pass the value pointer inside the interface value. We don't need to pass the type, since by definition a method knows which type it has been compiled for.

Loading

@glycerine
Copy link

@glycerine glycerine commented Dec 28, 2017

Thanks Ian.

Related discussion (for others catching up) from Carl Chatfield's proposal a couple years back, calling convention adaptation to provide the entire interface value.

https://groups.google.com/d/msg/golang-dev/coyYwxU3dfM/dZzOleaeFu4J

Loading

@cosmos72
Copy link

@cosmos72 cosmos72 commented May 13, 2018

In my opinion, reflect.NamedOf would be useful even without support for adding methods to the newly created type - just like reflect.StructOf is useful even without (full) support for wrapper methods of embedded fields.

In any case, I think that reflect.NamedOf() should have, as proposed, a function parameter to pass the list of methods and, until support for adding methods is implemented, it could simply panic if the methods list is non-empty.

I also have an idea on how to support adding methods to types created with reflect.NamedOf without changing the calling convention for methods, but it's quite tricky - better if I write a prototype first.

P.S. it is worth noting that the signature func NamedOf(t Type, name string, methods []Method) Type proposed above would not be able to create recursive types: they need a multi-step procedure, analogous to what go/types.Named has:

func NewNamed(obj *TypeName, underlying Type, methods []*Func) *Named
// then create an underlying type that contains the result of NewNamed
// and pass it to SetUnderlying:
func (t *Named) SetUnderlying(underlying Type)

Loading

@corebreaker
Copy link

@corebreaker corebreaker commented Oct 30, 2018

In my opinion, reflect.NamedOf would be useful even without support for adding methods to the newly created type - just like reflect.StructOf is useful even without (full) support for wrapper methods of embedded fields.

In any case, I think that reflect.NamedOf() should have, as proposed, a function parameter to pass the list of methods and, until support for adding methods is implemented, it could simply panic if the methods list is non-empty.

I also have an idea on how to support adding methods to types created with reflect.NamedOf without changing the calling convention for methods, but it's quite tricky - better if I write a prototype first.

P.S. it is worth noting that the signature func NamedOf(t Type, name string, methods []Method) Type proposed above would not be able to create recursive types: they need a multi-step procedure, analogous to what go/types.Named has:

func NewNamed(obj *TypeName, underlying Type, methods []*Func) *Named
// then create an underlying type that contains the result of NewNamed
// and pass it to SetUnderlying:
func (t *Named) SetUnderlying(underlying Type)

Yes you are right, and it will introduce to the idea of Proxy Object.

Loading

navytux added a commit to navytux/go123 that referenced this issue Dec 26, 2019
…or contexts

In many cases IO needs to be able to be canceled. For example in WCFS
filesystem I need to cancel handling sysread(/head/watch) when FUSE
INTERRUPT request comes in [1,2,3]. The READ handler for /head/watch
inside WCFS is interally implemented via io.Pipe which does not provide
read/write cancellattion besides "destructive" close.

Standard Go answer for cancellations is via contexts. So as a first step
let's add corresponding interfaces - xio.Reader, xio.Writer etc - that
are io analogs that add support for contexts.

For compatibility with legacy code that work with only io.X (e.g.  only
with io.Reader), in spirit of [4], add BindCtx which binds xio.X
instance with context and converts it into io.X. Add WithCtx -
corresponding inverse operation that converts io.X back into xio.X and
for general io.X adds cancellation handling on a best-effort basis.

[1] https://lab.nexedi.com/kirr/wendelin.core/commit/b17aeb8c
[2] https://lab.nexedi.com/kirr/wendelin.core/commit/f05271b1
[3] https://lab.nexedi.com/kirr/wendelin.core/commit/5ba816da
[4] golang/go#20280
[5] golang/go#16522
LLKennedy added a commit to LLKennedy/mercury that referenced this issue Feb 4, 2020
@TheCount
Copy link

@TheCount TheCount commented Jun 16, 2020

I have a vague idea which I wanted to run past you before I spend time digging deeper.

So methods have no access to a closure pointer (assuming we don't want to change calling conventions), we don't want to incur extra costs for dynamic method calls unrelated to reflect, and we don't want dynamic code generation.

The idea is that the method code of a NamedOf generated method is just the equivalent of

TEXT Forwarder
   CALL DispatchMethod

DispatchMethod then inspects the stack to identify the Forwarder's virtual address and uses that to look up the actual user function (as an interface{}, so we also have its type) and (un-)wrap receiver, args and results (plus some stack manipulation to return directly to the original caller).

Obviously, this requires multiple copies of Forwarder so that we have a unique virtual address to look up the function. How do we do this without runtime code generation? We can pre-seed a virtual memory page with copies of Forwarder at compile time. For example, if the page size is 4096 bytes and Forwarder requires 16 bytes, we can make 256 individual methods from that one page.

What happens if we run out of free slots? We create a new virtual memory mapping to the same physical memory (or possibly pre-seed the binary with "enough" of such mappings at compile time). So the same physical code gets new virtual addresses and can serve more methods. No generation of new code necessary.

In total, the cost would be:

  • Extra page with Forwarder copies in the binary image.
  • Lookup and dispatch facilities in the reflect package.
  • Calls to NamedOf-generated methods would be slower than regular method calls, but unrelated method or function calls would not be affected at all.
  • One address mapping for each ~PAGE_SIZE/ForwarderSize new methods.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet