[SPARK-10612][SQL] Add prepare to LocalNode. #8761
Conversation
|
Test build #42475 has finished for PR 8761 at commit
|
| @@ -27,6 +27,10 @@ case class ConvertToSafeNode(conf: SQLConf, child: LocalNode) extends UnaryLocal | |||
|
|
|||
| private[this] var convertToSafe: Projection = _ | |||
|
|
|||
| override def prepare(): Unit = { | |||
| child.prepare() | |||
There was a problem hiding this comment.
Why not implement this one in UnaryLocalNode? Then we don't need to duplicate them in all subclasses of ``UnaryLocalNode`.
There was a problem hiding this comment.
Actually then we could just implement it in LocalNode itself since it can traverse the children.
|
OK updated - I put everything in LocalNode. I was initially thinking it might be better to be more explicit - but it seems vast majority of the operators don't need to do anything. |
|
Test build #42493 has finished for PR 8761 at commit
|
|
retest this please |
|
Actually this patch is essentially a no-op so I'm just going to merge it. This is going to master. |
|
Test build #42512 has finished for PR 8761 at commit
|
The idea is that we should separate the function call that does memory reservation (i.e. prepare) from the function call that consumes the input (e.g. open()), so all operators can be a chance to reserve memory before they are all consumed.