proposal: reflect: package reflection

[glacials]

Proposal

I'd like to propose basic package reflection in reflect to support discovery of types, functions, and variables in a package.

Similar to how value reflection allows discovery of methods and fields given a struct value; package reflection should allow discovery of functions, variables, and types given a package.

Problem

This closes a gap that exists today where code using reflect must be "pre-loaded" with the types it should know about, making every new type, function, or variable that "wants" to be reflected first register itself with the reflector. This is not up to the standard of how reflection of structs works, where reflecting code can iterate over every struct field and method.

Example Usage

One possibility is to retrieve a package from an "anchor" type:

import "reflect"
import "time"

t := reflect.TypeOf(time.Time{})
p := t.Package() // Returns a reflect.Package

Another, which uses a modified syntax, allows direct retrieval:

import "reflect"
import "time"

p = reflect.PackageOf(time) // Returns a reflect.Package

Example Interface

A reflect.Package might respond to similar calls as a reflect.Value:

type Package interface {
	// Equal reports true if p is equal to q.
	// For two invalid packages, Equal will report true.
	// Otherwise, Equal will report true if p and q
	// refer to the same package.
	Equal(q Package) bool
	// Function returns a function value corresponding to p's i'th function.
	// Function panics if i is out of range.
	Function(i int) Value
	// FunctionByName returns a function value corresponding to the function
	// of p with the given name.
	// It returns the zero Value if no function was found.
	FunctionByName(name string) Value
	// NumFunction returns the number of functions in the package p.
	NumFunction() int
	// NumType returns the number of types in the package p.
	NumType() int
	// NumVar returns the number of fields in the package p.
	NumVar() int
	// PkgPath returns a defined package's path, that is, the import path
	// that uniquely identifies the package, such as "encoding/base64".
	Path() string
	// String returns a string representation of the package.
	// The string representation may use shortened package names
	// (e.g., base64 instead of "encoding/base64") and is not
	// guaranteed to be unique among types.
	// To test for package identity, compare the Packages directly.
	String() string
	// Type returns a type corresponding to p's i'th type.
	// Type panics if i is out of range.
	Type(i int) Type
	// TypeByName returns a type value corresponding to the type
	// of p with the given name.
	// It returns the zero value if no type was found.
	TypeByName(name string) Type
	// Var returns the i'th variable of the package p.
	// It panics if i is out of range.
	Var(i int) Value
	// VarByName returns the package variable with the given name.
	// It returns the zero Value if no variable was found.
	VarByName(name string) Value
}

[ianlancetaylor]

This makes it impossible for the linker to discard unused functions and variables, which it does today. I don't think this is realistically feasible.

[mvdan]

To further drive @ianlancetaylor's point: this would make most Go binaries significantly larger, considerably hurting #6853.

[glacials]

Is this different than what happens for unused struct methods and fields?

[thediveo]

The "unused struct fields" aren't removed because they are typically needed in order to interface with syscalls, C libraries, et cetera. Such fields might be used for padding or could be left-overs from syscall API changes over time. In any case, I don't think that you can compare unused struct fields with unused functions. If your proposal would be implemented as is, then my Wireshark plugin would be multiple times its current size, because all of a sudden there would be "tons" of Azure and Kubernetes API client code included, even if not used at all.

Maybe you might would like to be able to discover only the functions and types that actually end up in a binary, because they are used? Maybe more akin to this thwd's answer to How to discover all package types at runtime?

[glacials]

I see, thanks for explaining. For the problem I'm trying to solve, which is something akin to first- and second-party plugin management, the packages being excluded from the binary would be a show-stopper.

So it sounds like this is a nonstarter, although I'm curious to hear if there are practical alternatives outside of having the importing package maintain a list of explicit references to names it needs, even when all further usage is through metaprogramming.

Options I've looked at are plugin which creates too much fragility for us, and github.com/hashicorp/go-plugin which seems complex for our needs.

[Splizard]

This makes it impossible for the linker to discard unused functions and variables, which it does today. I don't think this is realistically feasible.

Only when the package is used for reflection, if the package is not used in this way, unused functions and variables can be discarded as usual. Similar to runtime reflection data, which isn't stored if the value is never stored within an interface value.

[ianlancetaylor]

If one small package somewhere uses this new facility, then the linker can't discard anything from any package anywhere. That's a hazard for any large Go project.

[Splizard]

If one small package somewhere uses this new facility, then the linker can't discard anything from any package anywhere. That's a hazard for any large Go project.

If you can call Package() on arbitrary reflect.Type values, sure. Easy adjustment to the proposal is to disallow this and restrict package reflection to explicit package selection calls ie. reflect.PackageOf(time).

[earthboundkid]

Re: plug-ins, this talk was interesting and lists several ways of doing it: https://youtube.com/watch?v=pRT36VqpljA

[Jorropo]

Currently trying to pass a package identifier as anything gives use of package time not in selector compile-time error.
PackageOf would be an adhoc rule of this which I would find confusing.

As far as I can think about only unsafe and builtin are allowed that luxury of special language rules right now, both since they happen to be created before the existence of generics and unsafe because having it return constants is just too useful and you should know what you are doing anyway if using it.
IMO in this case the gains does not outweigh the cost of an other std exception to the language rules.

[ianlancetaylor]

I see that you suggested reflect.Package(time) in the original post, but I don't understand why a program would want to get a list of functions in a known package. What is the use case for that?

[earthboundkid]

ISTM, it would probably be better as reflect.Package("time") because apart from the technical issues with bare package names, then you can do pkgName := discoverPlugins(); pkg := reflect.Package(pkgName).

[Jorropo]

@carlmjohnson This would be opening the door to package names being passed at runtime. Preventing almost all code pruning.
What if someone pass a package that is not used at all by the binary, should the runtime download sources from proxy.golang.org, build it in memory, and load the result JIT like ?

You could make the signature always require a const string but that is not a thing the language offers and would be yet an other exception.

[thediveo]

@Jorropo please don't overshoot here, this comes across as trying a little bit too hard to push the original request over the cliff. Instead, let's try to see the chances here.

From my own experience with plugin management of statically build-in plugins I can imagine a benefit of being able to discover the actually linked-in modules, and then their exported types (again, only the actually included ones).

However, I'm unclear as to how to ensure that the plugin functions to be exported actually get linked in, as just underscore importing their containing packages isn't sufficient? Does this mean that this idea of package inspection would fall flat because the desired pruning would cause the p,ugin functions to not even being linked into the final binary?