Free ephemeral resources used for scanning

[jcharum]

When we scan results, we get a slice of *openerAtReaders, one for each result task. We read from each reader sequentially. When we are done with a given reader, we retain some of the resources used when reading from it, most notably gob decode buffers. As we scan, we accumulate these defunct buffers, and our memory footprint grows.

This happens for two reasons:

1. We pop off the slice of readers to iterate, i.e. q = q[1:]. However, we do not clear the backing array reference to the reader.
2. When we close the reader, we don't clear the sliceioReader, which in turn holds the gob decoder.

Fixing either would eliminate the specific scan leak. Fix both, as I think it's the correct behavior.

[jschellenberger]

LGTM. Did you try running? Should I try running?

[jcharum]

LGTM. Did you try running? Should I try running?

I diagnosed and tested this using a test program.

You can patch this if you have urgent needs and don't want to wait for this to land internally.

[prasadgopal]

is it possible to write a test to make sure we don't undo this fix at some point? (if it is not too difficult)

[jcharum]

I did not come up with a good way of testing, as it is really an internal resource usage detail. Two possibilities:

1. Write a test that accesses internals. This does not meet my personal judgment for utility to fragility ratio.
2. Write a test that creates, uses, and closes many readers, forces a GC, and checks memory usage. This seems possibly fragile and I again question the utility to fragility ratio.

This goes for both the multiReader and openerAtReader changes. I'm open to debate and suggestion though.

[josh-newman]

I'm ok with landing this and improving testing as we think of better alternatives (as measured by utility vs. fragility).

Just to toss in another idea, we could write a test that scans some notably large objects, then uses a recursive, reflective object size estimator [1] to walk the scanner's references and make sure total size drops after close.

[1] For GRAILers: @jschellenberger made a prototype of something like this: D35649.