Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
515 lines (407 sloc) 25 KB

Chapter 12. Reflection

Go provides a mechanism to do the following on variables without knowing their types at compile time:

  • Update variables and inspect their values at run time
  • Call their methods
  • Apply the operations intrinsic to their representation

This mechanism is called reflection. Reflection also lets us treat types themselves as first-class values.

This chapter covers Go's reflection features on how they increase the expressiveness of the language, and in particular how they are crucial to the implementation of two important APIs:

Reflection is also essential to the template mechanism provided by the text/template and html/template packages as seen in Section 4.6. However, reflection is complex to reason about and not for casual use, so although these packages are implemented using reflection, they do not expose reflection in their own APIs.

Why Reflection?

Sometimes we need to write a function capable of dealing uniformly with values of types, which have one of the following traits:

  • They don't satisfy a common interface,
  • They don't have a known representation,
  • They don't exist at the time we design the function,
  • All three of above.

A familiar example is the formatting logic within fmt.Fprintf, which can usefully print an arbitrary value of any type, even a user-defined one. Let's try to implement a function like it using what we know already. For simplicity, our function will accept one argument and will return the result as a string like fmt.Sprint does, so we'll call it Sprint.

We start with a type switch that tests whether the argument defines a String method and call it if so. We then add switch cases that test the values' dynamic type against each of the basic types: string, int, bool, etc., and perform the appropriate formatting operation in each case.

func Sprint(x interface{}) string {
	type stringer interface {
		String() string
	}
	switch x := x.(type) {
	case stringer:
		return x.String()
	case string:
		return x
	case int:
		return strconv.Itoa(x)
	// ...similar cases for int16, uint32, and so on...
	case bool:
	if x {
		return "true"
		}
		return "false"
	default:
		// array, chan, func, map, pointer, slice, struct
		return "???"
	}
}

But how do we deal with other types?

  • Types like []float64, map[string][]string: we can add more cases, but the number of such types is infinite.
  • Named types like url.Values: even if the type switch had a case for its underlying type map[string][]string, it wouldn't match url.Values because the two types are not identical, and the type switch cannot include a case for each type like url.Values because that would require this library to depend upon its clients.

Without a way to inspect the representation of values of unknown types, we quickly get stuck. What we need is reflection.

reflect.Type and reflect.Value

Reflection is provided by the reflect package. It defines two important types, Type and Value. A Type represents a Go type. It is an interface with many methods for discriminating among types and inspecting their components, like the fields of a struct or the parameters of a function. The sole implementation of reflect.Type is the type descriptor (Section 7.5), the same entity that identifies the dynamic type of an interface value.

The reflect.TypeOf function accepts any interface{} and returns its dynamic type as a reflect.Type:

t := reflect.TypeOf(3)  // a reflect.Type
fmt.Println(t.String()) // "int"
fmt.Println(t)          // "int"

The TypeOf(3) call above assigns the value 3 to the interface{} parameter. Recall from Section 7.5 that an assignment from a concrete value to an interface type performs an implicit interface conversion, which creates an interface value consisting of two components:

  • Its dynamic type is the operand's type (int)
  • Its dynamic value is the operand's value (3)

Because reflect.TypeOf returns an interface value's dynamic type, it always returns a concrete type. For example, the code below prints "*os.File", not "io.Writer". Later, we will see that reflect.Type is also capable of representing interface types.

var w io.Writer = os.Stdout
fmt.Println(reflect.TypeOf(w)) // "*os.File"

Notice that reflect.Type satisfies fmt.Stringer. Because printing the dynamic type of an interface value is useful for debugging and logging, fmt.Printf provides a shorthand, %T, that uses reflect.TypeOf internally:

fmt.Printf("%T\n", 3) // "int"

The other important type in the reflect package is Value. A reflect.Value can hold a value of any type. The reflect.ValueOf function accepts any interface{} and returns a reflect.Value containing the interface's dynamic value. As with reflect.TypeOf, the results of reflect.ValueOf are always concrete, but a reflect.Value can also hold interface values.

v := reflect.ValueOf(3) // a reflect.Value
fmt.Println(v)          // "3"
fmt.Printf("%v\n", v)   // "3"
fmt.Println(v.String()) // NOTE: "<int Value>"

Like reflect.Type, reflect.Value also satisfies fmt.Stringer, but unless the Value holds a string, the result of the String method reveals only the type. Instead, use the fmt package's %v verb, which treats reflect.Values specially.

Calling the Type method on a Value returns its type as a reflect.Type:

t := v.Type()            // a reflect.Type
fmt.Println(t.String())  // "int"

The inverse operation to reflect.ValueOf is the reflect.Value.Interface method. It returns an interface{} holding the same concrete value as the reflect.Value:

v := reflect.ValueOf(3) // a reflect.Value
x := v.Interface()      // an interface{}
i := x.(int)            // an int
fmt.Printf("%d\n", i)   // "3"

A reflect.Value and an interface{} (empty interface) can both hold arbitrary values. What is the difference between them?

  • An empty interface hides the representation and intrinsic operations of the value it holds and exposes none of its methods, so unless we know its dynamic type and use a type assertion to peer inside it (as we did above), there is little we can do to the value within.
  • In contrast, a Value has many methods for inspecting its contents, regardless of its type.

Let's use the methods of a Value for our second attempt at a general formatting function called format.Any.

Instead of a type switch, we use reflect.Value's Kind method to discriminate the cases. Although there are infinitely many types, there are only a finite number of kinds of type:

  • The basic types Bool, String, and all the numbers
  • The aggregate types Array and Struct
  • The reference types Chan, Func, Ptr, Slice, and Map
  • Interface types
  • Invalid, meaning no value at all (The zero value of a reflect.Value has kind Invalid.)

gopl.io/ch12/format/format.go

package format

import (
	"reflect"
	"strconv"
)

// Any formats any value as a string.
func Any(value interface{}) string {
	return formatAtom(reflect.ValueOf(value))
}

// formatAtom formats a value without inspecting its internal structure.
func formatAtom(v reflect.Value) string {
	switch v.Kind() {
	case reflect.Invalid:
		return "invalid"
	case reflect.Int, reflect.Int8, reflect.Int16,
		reflect.Int32, reflect.Int64:
		return strconv.FormatInt(v.Int(), 10)
	case reflect.Uint, reflect.Uint8, reflect.Uint16,
		reflect.Uint32, reflect.Uint64, reflect.Uintptr:
		return strconv.FormatUint(v.Uint(), 10)
	// ...floating-point and complex cases omitted for brevity...
	case reflect.Bool:
		return strconv.FormatBool(v.Bool())
	case reflect.String:
		return strconv.Quote(v.String())
	case reflect.Chan, reflect.Func, reflect.Ptr, reflect.Slice, reflect.Map:
		return v.Type().String() + " 0x" +
			strconv.FormatUint(uint64(v.Pointer()), 16)
	default: // reflect.Array, reflect.Struct, reflect.Interface
		return v.Type().String() + " value"
	}
}

This function treats each value as an indivisible thing with no internal structure (hence formatAtom). For aggregate types (structs and arrays) and interfaces it prints only the type of the value, and for reference types (channels, functions, pointers, slices, and maps), it prints the type and the reference address in hexadecimal. This is less than ideal but still a major improvement. Since Kind is concerned only with the underlying representation, format.Any also works for named types. For example:

var x int64 = 1
var d time.Duration = 1 * time.Nanosecond
fmt.Println(format.Any(x))                  // "1"
fmt.Println(format.Any(d))                  // "1"
fmt.Println(format.Any([]int64{x}))         // "[]int64 0x8202b87b0"
fmt.Println(format.Any([]time.Duration{d})) // "[]time.Duration 0x8202b87e0"

Display, a Recursive Value Printer

This section is about how to improve the display of composite types. Rather than try to copy fmt.Sprint exactly, we'll build a debugging utility function called Display that, given an arbitrarily complex value x, prints the complete structure of that value, labeling each element with the path by which it was found.

e, _ := eval.Parse("sqrt(A / pi)")
Display("e", e)

In the call above, the argument to Display is a syntax tree from the expression evaluator in Section 7.9. eval.Parse is from gopl.io/ch7/eval/parse.go.

The output of Display is shown below:

Display e (eval.call):
e.fn = "sqrt"
e.args[0].type = eval.binary
e.args[0].value.op = 47
e.args[0].value.x.type = eval.Var
e.args[0].value.x.value = "A"
e.args[0].value.y.type = eval.Var
e.args[0].value.y.value = "pi"

Where possible, you should avoid exposing reflection in the API of a package. We'll define an unexported function display to do the real work of the recursion, and export Display, a simple wrapper around it that accepts an interface{} parameter:

gopl.io/ch12/display/display.go

func Display(name string, x interface{}) {
	fmt.Printf("Display %s (%T):\n", name, x)
	display(name, reflect.ValueOf(x))
}

display uses the formatAtom function defined earlier to print elementary values, e.g. basic types, functions, and channels, but uses the methods of reflect.Value to recursively display each component of a more complex type. As the recursion descends, the path string, which initially describes the starting value (for instance, "e"), will be augmented to indicate how we reached the current value (for instance, "e.args[0].value").

func display(path string, v reflect.Value) {
	switch v.Kind() {
	case reflect.Invalid:
		fmt.Printf("%s = invalid\n", path)
	case reflect.Slice, reflect.Array:
		for i := 0; i < v.Len(); i++ {
			display(fmt.Sprintf("%s[%d]", path, i), v.Index(i))
		}
	case reflect.Struct:
		for i := 0; i < v.NumField(); i++ {
			fieldPath := fmt.Sprintf("%s.%s", path, v.Type().Field(i).Name)
			display(fieldPath, v.Field(i))
		}
	case reflect.Map:
		for _, key := range v.MapKeys() {
			display(fmt.Sprintf("%s[%s]", path,
				formatAtom(key)), v.MapIndex(key))
		}
	case reflect.Ptr:
		if v.IsNil() {
			fmt.Printf("%s = nil\n", path)
		} else {
			display(fmt.Sprintf("(*%s)", path), v.Elem())
		}
	case reflect.Interface:
		if v.IsNil() {
			fmt.Printf("%s = nil\n", path)
		} else {
			fmt.Printf("%s.type = %s\n", path, v.Elem().Type())
			display(path+".value", v.Elem())
		}
	default: // basic types, channels, funcs
		fmt.Printf("%s = %s\n", path, formatAtom(v))
	}
}

In the above code:

  • Slices and arrays: The logic is the same for both. The Len method returns the number of elements of a slice or array value, and Index(i) retrieves the element at index i, also as a reflect.Value; it panics if i is out of bounds. These are analogous to the built-in len(a) and a[i] operations on sequences. The display function recursively invokes itself on each element of the sequence, appending the subscript notation "[i]" to the path.

    • Although reflect.Value has many methods, only a few are safe to call on any given value.
  • Structs: The NumField method reports the number of fields in the struct, and Field(i) returns the value of the i-th field as a reflect.Value. The list of fields includes ones promoted from anonymous fields. To append the field selector notation ".f" to the path, we must obtain the reflect.Type of the struct and access the name of its i-th field.

  • Maps: The MapKeys method returns a slice of reflect.Values, one per map key. As usual when iterating over a map, the order is undefined. MapIndex(key) returns the value corresponding to key. We append the subscript notation "[key]" to the path. (We're cutting a corner here. The type of a map key isn't restricted to the types formatAtom handles best; arrays, structs, and interfaces can also be valid map keys.)

  • Pointers: The Elem method returns the variable pointed to by a pointer as a reflect.Value. This operation would be safe even if the pointer value is nil, in which case the result would have kind Invalid, but we use IsNil to detect nil pointers explicitly so we can print a more appropriate message. We prefix the path with a "*" and parenthesize it to avoid ambiguity.

  • Interfaces: IsNil is used to test whether the interface is nil, and if not, we retrieve its dynamic value using v.Elem() and print its type and value.

Let's use Display on some example types. The Movie type below is a slight variation on the one in Section 4.5:

type Movie struct {
	Title, Subtitle string
	Year            int
	Color           bool
	Actor           map[string]string
	Oscars          []string
	Sequel          *string
}

Let's declare a value of this type and see what Display does with it:

strangelove := Movie{
	Title:    "Dr. Strangelove",
	Subtitle: "How I Learned to Stop Worrying and Love the Bomb",
	Year:     1964,
	Color:    false,
	Actor: map[string]string{
		"Dr. Strangelove":            "Peter Sellers",
		"Grp. Capt. Lionel Mandrake": "Peter Sellers",
		"Pres. Merkin Muffley":       "Peter Sellers",
		"Gen. Buck Turgidson":        "George C. Scott",
		"Brig. Gen. Jack D. Ripper":  "Sterling Hayden",
		`Maj. T.J. "King" Kong`:      "Slim Pickens",
	},
	Oscars: []string{
		"Best Actor (Nomin.)",
		"Best Adapted Screenplay (Nomin.)",
		"Best Director (Nomin.)",
		"Best Picture (Nomin.)",
	},
}

The call Display("strangelove", strangelove) prints:

Display strangelove (display.Movie):
strangelove.Title = "Dr. Strangelove"
strangelove.Subtitle = "How I Learned to Stop Worrying and Love the Bomb"
strangelove.Year = 1964
strangelove.Color = false
strangelove.Actor["Gen. Buck Turgidson"] = "George C. Scott"
strangelove.Actor["Brig. Gen. Jack D. Ripper"] = "Sterling Hayden"
strangelove.Actor["Maj. T.J. \"King\" Kong"] = "Slim Pickens"
strangelove.Actor["Dr. Strangelove"] = "Peter Sellers"
strangelove.Actor["Grp. Capt. Lionel Mandrake"] = "Peter Sellers"
strangelove.Actor["Pres. Merkin Muffley"] = "Peter Sellers"
strangelove.Oscars[0] = "Best Actor (Nomin.)"
strangelove.Oscars[1] = "Best Adapted Screenplay (Nomin.)"
strangelove.Oscars[2] = "Best Director (Nomin.)"
strangelove.Oscars[3] = "Best Picture (Nomin.)"
strangelove.Sequel = nil

We can use Display to display the internals of library types, such as *os.File:

Display("os.Stderr", os.Stderr)
// Output:
// Display os.Stderr (*os.File):
// (*(*os.Stderr).file).fd = 2
// (*(*os.Stderr).file).name = "/dev/stderr"
// (*(*os.Stderr).file).nepipe = 0

Notice that even unexported fields are visible to reflection. Beware that the particular output of this example may vary across platforms and may change over time as libraries evolve. (Those fields are private for a reason.)

We can even apply Display to a reflect.Value and watch it traverse the internal representation of the type descriptor for *os.File. The output of the call Display("rV", reflect.ValueOf(os.Stderr)) is shown below (though your output may vary:

Display rV (reflect.Value):
(*rV.typ).size = 8
(*rV.typ).hash = 871609668
(*rV.typ).align = 8
(*rV.typ).fieldAlign = 8
(*rV.typ).kind = 22
(*(*rV.typ).string) = "*os.File"
(*(*(*rV.typ).uncommonType).methods[0].name) = "Chdir"
(*(*(*(*rV.typ).uncommonType).methods[0].mtyp).string) = "func() error"
(*(*(*(*rV.typ).uncommonType).methods[0].typ).string) = "func(*os.File) error"
...

Observe the difference between the following two examples:

var i interface{} = 3
Display("i", i)
// Output:
// Display i (int):
// i = 3

Display("&i", &i)
// Output:
// Display &i (*interface {}):
// (*&i).type = int
// (*&i).value = 3
  • In the first example, Display calls reflect.ValueOf(i), which returns a value of kind Int. As we mentioned in Section 12.2, reflect.ValueOf always returns a Value of a concrete type since it extracts the contents of an interface value.
  • In the second example, Display calls reflect.ValueOf(&i), which returns a pointer to i, of kind Ptr. The switch case for Ptr calls Elem on this value, which returns a Value representing the variable i itself, of kind Interface. A Value obtained indirectly like this one may represent any value at all, including interfaces. The display function calls itself recursively and this time, it prints separate components for the interface's dynamic type and value.

As currently implemented, Display will never terminate if it encounters a cycle in the object graph, such as this linked list that eats its own tail:

// a struct that points to itself
type Cycle struct{ Value int; Tail *Cycle }
var c Cycle
c = Cycle{42, &c}
Display("c", c)

Display prints this ever-growing expansion:

Display c (display.Cycle):
c.Value = 42
(*c.Tail).Value = 42
(*(*c.Tail).Tail).Value = 42
(*(*(*c.Tail).Tail).Tail).Value = 42
...ad infinitum...

Many Go programs contain at least some cyclic data. Making Display robust against such cycles is tricky, requiring additional bookkeeping to record the set of references that have been followed so far (which is also costly). A general solution requires unsafe language features, as we will see in Section 13.3.

Cycles pose less of a problem for fmt.Sprint because it rarely tries to print the complete structure. For example, when it encounters a pointer, it breaks the recursion by printing the pointer's numeric value. It can get stuck trying to print a slice or map that contains itself as an element, but such rare cases do not warrant the considerable extra trouble of handling cycles.

Example: Encoding S-Expressions

(skipped) [p338-341]

Setting Variables with reflect.Value

In previous sections, reflection has only interpreted values in our program. The point of this section is to change them.

Recall that some Go expressions like x, x.f[1], and *p denote variables, but others like x + 1 and f(2) do not. A variable is an addressable storage location that contains a value, and its value may be updated through that address.

A similar distinction applies to reflect.Values. Some are addressable; others are not. Consider the following declarations:

x := 2                     // value   type   variable?
a := reflect.ValueOf(2)    // 2       int    no
b := reflect.ValueOf(x)    // 2       int    no
c := reflect.ValueOf(&x)   // &x      *int   no
d := c.Elem()              // 2       int    yes (x)
  • The value within a is not addressable. It is merely a copy of the integer 2. The same is true of b.
  • The value within c is also non-addressable, being a copy of the pointer value &x.
  • In fact, no reflect.Value returned by reflect.ValueOf(x) is addressable.
  • But d, derived from c by dereferencing the pointer within it, refers to a variable and is thus addressable. We can use this approach, calling reflect.ValueOf(&x).Elem(), to obtain an addressable Value for any variable x.

We can ask a reflect.Value whether it is addressable through its CanAddr method:

fmt.Println(a.CanAddr()) // "false"
fmt.Println(b.CanAddr()) // "false"
fmt.Println(c.CanAddr()) // "false"
fmt.Println(d.CanAddr()) // "true"

We obtain an addressable reflect.Value whenever we indirect through a pointer, even if we started from a non-addressable Value. All the usual rules for addressability have analogs for reflection. For example, since the slice indexing expression e[i] implicitly follows a pointer, it is addressable even if the expression e is not. By analogy, reflect.ValueOf(e).Index(i) refers to a variable, and is thus addressable even if reflect.ValueOf(e) is not.

To recover the variable from an addressable reflect.Value requires three steps:

  1. We call Addr(), which returns a Value holding a pointer to the variable.
  2. We call Interface() on this Value, which returns an interface{} value containing the pointer.
  3. If we know the type of the variable, we can use a type assertion to retrieve the contents of the interface as an ordinary pointer. We can then update the variable through the pointer:
x := 2
d := reflect.ValueOf(&x).Elem()   // d refers to the variable x
px := d.Addr().Interface().(*int) // px := &x
*px = 3                           // x = 3
fmt.Println(x)                    // "3"

Alternatively, we can update the variable referred to by an addressable reflect.Value directly without using a pointer, by calling the reflect.Value.Set method:

d.Set(reflect.ValueOf(4))
fmt.Println(x) // "4"

The same assignability checks, which are ordinarily performed by the compiler, are done at run time by the Set methods. In the above code, the variable and the value both have type int, but if the variable had been an int64, the program would panic, so it's crucial to make sure the value is assignable to the type of the variable:

d.Set(reflect.ValueOf(int64(5))) // panic: int64 is not assignable to int

Calling Set on a non-addressable reflect.Value panics too:

x := 2
b := reflect.ValueOf(x)
b.Set(reflect.ValueOf(3)) // panic: Set using unaddressable value

There are variants of Set specialized for certain groups of basic types: SetInt, SetUint, SetString, SetFloat, and so on:

d := reflect.ValueOf(&x).Elem()
d.SetInt(3)
fmt.Println(x) // "3"

These methods are more forgiving. For example, SetInt will succeed so long as the variable's type is some kind of signed integer, or even a named type whose underlying type is a signed integer, and if the value is too large it will be quietly truncated to fit. But tread carefully: calling SetInt on a reflect.Value that refers to an interface{} variable will panic, even though Set would succeed.

x := 1
rx := reflect.ValueOf(&x).Elem()
rx.SetInt(2)                     // OK, x = 2
rx.Set(reflect.ValueOf(3))       // OK, x = 3
rx.SetString("hello")            // panic: string is not assignable to int
rx.Set(reflect.ValueOf("hello")) // panic: string is not assignable to int
var y interface{}
ry := reflect.ValueOf(&y).Elem()
ry.SetInt(2)                     // panic: SetInt called on interface Value
ry.Set(reflect.ValueOf(3))       // OK, y = int(3)
ry.SetString("hello")            // panic: SetString called on interface Value
ry.Set(reflect.ValueOf("hello")) // OK, y = "hello"

When we applied Display to os.Stdout, we found that reflection can read the values of unexported struct fields that are inaccessible according to the usual rules of the language, like the fd int field of an os.File struct on a Unix-like platform. However, reflection cannot update such values:

stdout := reflect.ValueOf(os.Stdout).Elem() // *os.Stdout, an os.File var
fmt.Println(stdout.Type())                  // "os.File"
fd := stdout.FieldByName("fd")
fmt.Println(fd.Int()) // "1"
fd.SetInt(2)          // panic: unexported field

An addressable reflect.Value records whether it was obtained by traversing an unexported struct field and, if so, disallows modification. Consequently, CanAddr is not usually the right check to use before setting a variable. The related method CanSet reports whether a reflect.Value is addressable and settable:

fmt.Println(fd.CanAddr(), fd.CanSet()) // "true false"