[SPARK-2405][SQL] Reusue same byte buffers when creating new instance of InMemoryRelation

[marmbrus]

Reuse byte buffers when creating unique attributes for multiple instances of an InMemoryRelation in a single query plan.

[aarondav]

This loses the synchronization guarantees we had before -- without synchronized or volatile, there are no cross-thread guarantees. Is this OK?

(By the way, how is newInstance() used?)

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[marmbrus]

@aarondav, newInstance() is used before we perform resolution to ensure that that all expression ids in a plan are unique. Consider the case where you self join an InMemoryRelation with itself: we need to know which side of the join a given attribute is coming from, so we produce unique instances of the relation before resolving attributes.

I thought about the possible concurrency issues, but they will only arise in edge cases (simultaneous self-join queries on a table that is cached, but not yet materialized?), and will only result in double caching, not correctness issues... so this patch is strictly better than what we had before I think.

That said I guess we could fix it with a SyncVar probably... I'll have to think about it some more.

[aarondav]

This type of concurrency problem can be a correctness issue, albeit with low probability. For instance, we could at any point read a not-fully-instantiated "_cachedColumnBuffers", which would not be null but still be in an invalid state, leading to a very tricky exception. This is a problem even without the byte buffer reuse -- use of the same InMemoryRelation in two different threads could now cause this issue.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16412/

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16420/

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16421/

[rxin]

Merging in master & branch-1.0.